The status quo, challenges, and recommendations of pre-registration in psychological science

Abstract

Reproducibility serves as a critical criterion for determining the scientific validity of research findings (Platt, 1964; Schmidt, 2009). However, empirical investigations in psychological science revealed a low rate of reproducibility (Open Science Collaboration, 2015), which has been referred to as the “replication crisis”. Over the past decade, researchers have reflected on this issue and found that many research practices may inflate the false positive rate of published studies and compromise the reproducibility. These practices include p-Hacking, HARKing, small sample size, publication bias, and lack of data sharing and replication studies (Munafò et al., 2017). In response to the replication crisis and the need to enhance the credibility of psychological science, researchers introduced a series of methodological innovations. Among these, pre-registration and registered reports (RRs) have attracted a lot of attention. There are three types of pre-registration 1) pre-registration without peer review; 2) peer-reviewed registered reports (RRs); and 3) registered replication reports (RRR) that replicate previous studies. Of these, pre-registration, in its narrow sense, refers to the process wherein researchers register a study before initiating data collection. In contrast, RRs and RRR require researchers to submit their registration to a specific journal or platform for peer review. This evaluation examines the protocol and decides whether to accept it in principle before data collection begins. To facilitate the adaptation of pre-registration, researchers have developed templates tailored to specific research sub-fields (e.g., studies with neuroimaging techniques or cognitive modeling). This article summarizes 18 pre-registration templates and 11 registration platforms for different research purposes. A typical pre-registration template includes basic study information, design plan, sampling plan, variable settings, and analysis plan. Usually, registration platforms or journals also provide their own recommended templates. As for RRs, the review process is different from traditional research articles and consists of two stages (Chambers, 2019; Chambers et al., 2014). In the first stage, authors are required to submit their protocol, usually following a certain template and including some pilot data and scripts for data analysis. After a peer-review process, the journal decides whether to accept the stage 1 protocol in principle. If accepted, the journal is committed to publishing the final paper, provided the authors adhere to the approved protocol while conducting the study. In the second stage, after the authors have completed the study according to the original protocol, the authors submit the full manuscript. The journal would invite the previous reviewers to review the full manuscript again. Upon successful completion of the peer review process, the manuscript would be accepted and published by the journal regardless of the significance of the results (Chambers, 2013). Recently, the Peers-Community-In Registered Report (PCI-RR) emerged as a new development of RRs and authors can choose a journal after stage 2 recommended by the platform PCI-RR. Since pre-registration (or RRs) significantly differs from traditional research practices, it has been hotly debated since its initial introduction to the field (Pham & Oh, 2021; Szollosi et al., 2020). Unfortunately, much of the criticism arises from misconceptions about pre-registration. Some misunderstand its working, assuming that no changes are permitted once the pre-registration is made public, while others misconstrue its objectives, erroneously arguing that pre-registration does not prevent fraudulent practices. Some critics also pointed out pragmatic challenges associated with pre-registration, noting, for instance, that it tends to be more time-consuming compared to conventional research methodologies (Allen & Mehler, 2019; Spitzer & Mueller, 2021). The effect of pre-registration has also been questioned. However, some recent meta-researchs have provided answers to these questions. Empirical findings revealed that RRs reduce the proportion of positive results and alleviate publication bias. Also, RRs outperform non-pre-registration on almost every metric, e.g., novelty, innovation, and rigor (Soderberg et al., 2021). Yet, pre-registration without peer review did not show the same pattern as RRs, which may be due to lack of comparison between the registry and the final manuscript (Macnamara & Burgoyne, 2023; Syed, 2023). To encourage broader adoption of pre-registration and RRs, all stakeholders should work together to change the research culture. For example, individual researchers can plan their next project as a registered report, or they can integrate preregistered replicated experiments into the courses they are teaching so that the next generation of researchers would be familiar with the workflow of registered reports. On the other hand, academic institutes could develop policies to prioritize quality over quantity of publications and incorporate transparency as a criterion for quality, which will encourage researchers to adopt pre-registration and RRs. Publishers can also contribute by promoting pre-registration and RRs, offering RRs as a new article type, providing templates for preparing RRs, and implementing a streamlined review process for RRs.