Interpreting nonsignificant results:A quantitative investigation based on 500 Chinese psychological research

Abstract

Nonsignificant results are common in psychological research and can be easily misinterpreted as evidence for accepting null hypothesis. This misinterpretation may lead to false statistical inferences in empirical research. However, how prevalent this misinterpretation exists in Chinese published psychological studies is unknown. To answer this question, we randomly selected 500 empirical research papers published between 2017 and 2018 in Acta Psychological Sinica, Journal of Psychological Science, Chinese Journal of Clinical Psychology, Psychological Development and Education, Psychological and Behavioral Studies, screened articles in which the abstracts contained any sentences that indicated nonsignificant results (we call these sentences “negative statements” hereafter). We then read those articles and extracted negative- statements-related statistics and their interpretations, and evaluated the correctness of each interpretation. Finally, we calculated Bayes factors based on the available t values in these nonsignificant results. The protocol was pre-registered at OSF (https://osf.io/czx6f). We found that (1) out of 500 empirical research, 36% of their abstracts (n = 180) contained negative statements; (2) in those 180 articles, we extracted 236 nonsignificant results and corresponding interpretations, and found that 41% of these interpretations was incorrect, (3) Bayes factor analysis revealed that only 5.1% (n = 2) of available nonsignificant t-values (n = 39) can provide strong evidence in favor of null hypothesis (BF01 > 10). We compared the results with Aczel et al. (2018) and discussed the potential reasons that caused the misinterpretation. These data suggest that Chinese psychology researchers need to improve their understanding of nonsignificant results and statistical inference.