Registered Reports Author Guidelines

Please carefully read these Registered Reports Author Guidelines before submitting a Registered Report to EJP. Registered Reports are manuscripts that contain a study proposal submitted prior to data collection (see https://cos.io/rr). If you want to submit a paper for which data has already been collected (with or without preregistered hypotheses) then the paper should be submitted as a regular empirical paper in accordance with our general Author Guidelines.

We also recommend to read the 2017 Editorial, 2018 Editorial and 2019 Editorial for further details and discussion.

Please also consider EJP's general Author Guidelines for more basic information regarding evaluation criteria and formal aspects of manuscript.

Introduction

In these Registered Reports Author Guidelines we only summarize information that specifically applies to Registered Reports. For general information regarding EJPs evaluation criteria, mandatory transparency standards, streamlined review options, our blind review policy, and formal details regarding manuscript preparation, submission, and production please also read the general Author Guidelines. It is also recommended to read the 2017 Editorial, 2018 Editorial and 2019 Editorial for further details and discussion.

Introduction

Registered Reports are a form of empirical article in which the methods and proposed analyses are pre-registered and reviewed prior to research being conducted. The cornerstone of this article format is that a substantial part of the manuscript will be assessed prior to data collection. Initial submissions will include a description of the theoretical and empirical background, the main research questions and hypotheses, and a detailed Method section that includes the planned study procedure and measures, plans for sampling and analyses, and pilot data (if applicable).

As explained in detail elsewhere (e.g., Chambers, 2013; Munafò et al., 2017; Nosek & Lakens, 2014; also see https://cos.io/rr), RRs offer an elegant and straightforward way to increase the representativeness, trustworthiness, and robustness of our field’s findings. They also offer a number of advantages for authors, including in-depth conceptual and methodological feedback before the start of data assessment (i.e., feedback that can indeed be considered without having to start anew) as well as a much faster and result-independent security regarding the publication of one’s research.

Initial submissions of Registered Reports to EJP will be triaged by the Editor-in-chief, Associate Editors and members of the Editorial board, respectively. Those that pass triage will then be sent for in-depth peer review (Stage 1). Following review, the article will then be either rejected, revised, or accepted in principle for publication. Following in principle acceptance (IPA), the authors will then proceed to conduct the study, adhering exactly to the peer-reviewed procedures. When the study is complete the authors will submit their finalized and full manuscript for re-review (Stage 2) and will upload their raw data, digital study materials/code, and laboratory log to a free and publicly accessible file-sharing service. Pending quality checks and a sensible interpretation of the findings, the manuscript will be published regardless of the results.

Stage 1: Initial manuscript submission and review

Stage 1 submissions should include the manuscript (details below) and a brief cover letter. Authors are welcome to request pre-submission advice on the suitability of a study as a Registered Report by contacting the Editor of EJP (mitja.back@wwu.de). However, please note that the editorial board will not agree to send manuscripts for in-depth review until a complete Stage 1 submission has been considered.

 The Stage 1 cover letter must include (submissions not including this information will be desk-rejected):

• A brief scientific case for consideration in the case of novel studies. Authors who want to propose a replication study are encouraged to make a case for the scientific value of this replication (e.g., with regards to impact to the field, importance for increased precision of effect size,..). High-value replication studies are welcome and will be treated with equal priority to novel studies.
• A statement confirming that all necessary support (e.g. funding, facilities) and approvals (e.g. ethics) are in place for the proposed research. Note that manuscripts will be considered only for studies that are able to commence immediately. Authors who wish to submit a protocol prior to funding or ethical approval should discuss their proposal with the editorial board prior to submission.
• An anticipated timeline for completing the study if the initial submission is accepted.
• A statement whether the authors are or are not opting for Open Peer Review, whereby the review history is published alongside the paper if accepted.
• A statement confirming that the authors agree to share their anonymized raw data, digital study materials (including, for example, instructions, stimuli, variables, experiment code, coding and rating systems) and their analysis code for all published results.
• A statement confirming that, following Stage 1 in principle acceptance, the authors agree to register their approved protocol on the Open Science Framework (https://osf.io/rr) or other recognized repository, either publicly or under private embargo until submission of the Stage 2 manuscript.
• A statement confirming that if the authors withdraw their paper following in principle acceptance, they agree to the European Journal of Personality publishing a short summary of the pre-registered study on its Registered Reports OSF page under a headline Withdrawn Registrations.

Manuscript preparation guidelines – Stage 1

Initial Stage 1 submissions should include the following sections:

• Introduction
  
  
  • A review of the relevant theoretical and empirical literature that motivates the research question and a full delineation and description of the hypotheses. Please note that following IPA, the Introduction section cannot be altered apart from correction of typographic errors and altering of tense from future to past (see below).
    
    
• Methods
  
  
  • Full description of proposed sample characteristics, including criteria for subject inclusion and exclusion, and detailed description of procedures for defining outliers. Procedures for objectively defining exclusion criteria due to technical errors or for any other reasons must be documented, including details of how and under what conditions subjects would be replaced.
    
    
  • A description of study procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information. These procedures must be adhered to exactly in the subsequent execution of the study or any Stage 2 manuscript will be summarily rejected. Please note that reviewers at Stage 1 will be asked to specifically consider whether the stated study procedures contain sufficient detail to prevent undisclosed procedural flexibility.
    
    
  • Proposed analysis pipeline, including all preprocessing steps from raw data onwards, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Any covariates or regressors must be stated. Proposed analyses involving covariates must be reported with and without the covariate(s) included. Where analysis decisions are contingent on the outcome of prior analyses, these contingencies must be specified and adhered to. Only pre-planned analyses can be reported in the main Results section of Stage 2 submissions. However, unplanned post hoc analyses will be admissible in a separate section of the Results (see below).
    
    
  • Interpretative plan, including specification of which outcomes will be interpreted as support or disconfirmation of the proposed hypotheses, for each of the proposed analyses. In each case, authors should include a statement of what result would be taken as consistent with the prediction, what result would be taken as disconfirmation, and what result (if any) would be taken as inconclusive.
    
    
  • Studies involving frequentist inference must include a sampling plan such as statistical power analysis or appropriate alternative. Where effect sizes from previous literature are used to inform sampling plans, authors should account for publication bias, which leads to overestimation of true effect sizes. Power analysis, when undertaken, must be based on the lowest available or meaningful estimate of the effect size, achieving an a priori power (1 - β) of 0.9 or higher for all proposed hypothesis tests. In the case of highly uncertain effect sizes, a variable sample size and interim data analysis will be permissible but with inspection points stated in advance, appropriate Type I error correction employed, and a final stopping rule for data collection outlined.
    
    
  • For studies involving analyses with Bayes Factors, the hypotheses must be specified so that a Bayes factor can be calculated. Authors should indicate the relationship of the psychological theory to the statistical hypotheses, what distributions will be used to represent the hypotheses and how its parameters will be specified. The parameters need not be stated in advance, but where unstated, authors must indicate how the parameters will be later determined. For inference by Bayes factors, authors should discuss a target strength of evidence that is likely to be useful to readers (e.g., that a Bayes factor of 10 will be suitably convincing for the effect in question). If the stopping rule is dependent on the Bayes factor, authors should indicate a maximum feasible sample size after which sampling will stop, regardless of the Bayes factor.
    
    
  • Full descriptions must be provided of any outcome-neutral criteria that are required for successful testing of the stated hypotheses. Such ‘reality checks’ might include the absence of floor or ceiling effects, or positive controls. Please note that reviewers will be asked to judge whether the manuscript includes sufficient specification of reality checks.
    
    
  • Timeline for completion of the study and proposed resubmission date if registration review is successful. Extensions to this deadline can be negotiated with the action editor.
    
    
  • Any description of prospective methods or analysis plans should be written in future tense.
    
    
• Pilot Data
  
  
  • Can be included to establish reality checks, effect size estimations, feasibility, or proof of principle. Any pilot experiments will be published with the final version of the manuscript and will be clearly distinguished from data obtained for the main experiment(s).
    
    
• Secondary Registrations
  
  
  • The journal welcomes submissions proposing secondary analyses of existing data sets, provided authors can supply sufficient evidence (e.g. self-certification; letter from independent gatekeeper) to confirm that they have had no prior access to the data in question nor to summary reports of the data through descriptive or inferential statistics or narrative descriptions of the data, in talks, papers, or personal communication with others) . For advice on the eligibility of specific scenarios, authors are welcome to contact the Editor of EJP, Mitja Back (mitja.back@wwu.de).

Stage 1 submissions that are judged by the editorial board to be of sufficient quality and rigor will be sent for peer review. In considering papers at the registration stage, reviewers will be asked to assess (see Appendix below for the more detailed reviewer scoresheets):

1. The theoretical and/or practical relevance of the research question.
2. The logic, rationale, and plausibility of the proposed hypotheses
3. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis)
4. Whether the clarity and degree of methodological detail would be sufficient to exactly replicate the proposed study procedures and analysis pipeline
5. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls) for ensuring that the results obtained are able to test the stated hypotheses

Following Stage 1 peer review, manuscripts will be either rejected outright, offered the opportunity to be revised, or accepted. Manuscripts that pass peer review will be issued an in principle acceptance (IPA), indicating that the article will be published pending successful completion of the study according to the exact methods and analytic procedures outlined, as well as a defensible and evidence-bound interpretation of the results.

Please note that any deviation from the stated study procedures, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript. In cases where the pre-registered protocol is altered after IPA due to unforeseen circumstances (e.g. change of equipment or unanticipated technical error), the authors must consult the editorial board immediately for advice, and prior to the completion of data collection. Minor changes to the protocol may be permitted according to editorial discretion. In such cases, IPA would be preserved and the deviation reported in the Stage 2 submission. If the authors wish to alter the study procedures more substantially following IPA but still wish to publish their article as a Registered Report then the manuscript must be withdrawn and resubmitted as a new Stage 1 submission. Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in a final manuscript (see below).

Stage 2: Full manuscript review

Once the study is complete, authors prepare and resubmit their manuscript for full review, with the following additions:

• Cover letter.The Stage 2 cover letter must confirm that:
  • the manuscript includes a link in the method section to the public archive containing anonymized study data, digital materials, and statistical code.
  • the manuscript contains a link in the method to the approved Stage 1 protocol on the Open Science Framework or other recognised repository.
  • for primary Registered Reports, no data for any pre-registered study (other than pilot data included at Stage 1) was collected prior to the date of IPA.
  • for secondary Registered Reports, authors should confirm that no data (other than pilot data included at Stage 1) was subjected to the pre-registered analyses prior to IPA, and that authors had no prior access to the data in question nor to summary reports of the data through descriptive or inferential statistics or narrative descriptions of the data, in talks, papers, or personal communication with others.
    
    
• Submission of anonymized raw data, digital study materials, and laboratory log
  • Anonymized raw data and digital study materials must be made freely available in a public repository with a link provided within the Stage 2 manuscript. Authors are free to use any repository that renders data and materials freely and publicly accessible and provides a digital object identifier (DOI) to ensure that the data remain persistent, unique and citable.
  • Data files should be appropriately time stamped to show that data were collected after IPA and not before. Other than pre-registered and approved pilot data, no data acquired prior to the date of IPA is admissible in the Stage 2 submission. Raw data must be accompanied by guidance notes, where required, to assist other scientists in replicating the analysis pipeline. Authors are required to upload any relevant analysis scripts and other experimental materials that would assist in replication (e.g. stimuli & presentation code).
  • Any supplementary figures, tables, or other text (such as supplementary methods) can either be included as standard supplementary information that accompanies the paper, or they can be archived together with the data. Please note that the raw data itself should be archived (see above) rather than submitted to the journal as supplementary material.
  • A basic laboratory log must also be provided outlining the range of dates during which data collection took place. This log should be uploaded to the same public archive as the data and materials.
  • The Stage 2 manuscript must also contain a link to the registered protocol (deposited following IPA) on the Open Science Framework or other recognized repository.
    
    
• Background, Rationale and Methods
  • Apart from minor stylistic revisions, the Introduction cannot be altered from the approved Stage 1 submission, and the stated hypotheses cannot be amended or appended. At Stage 2, any description of the rationale or proposed methodology that was written in future tense within the Stage 1 manuscript should be changed to past tense. Any textual changes to the Introduction or Methods must be clearly marked in the Stage 2 submission. Depending on the timeframe of data collection, new relevant literature may have appeared between Stage 1 and Stage 2. Any such literature should be covered in the Discussion.
    
    
• Results & Discussion
  • These will be similar to standard original research reports but with added requirements. The outcome of all registered analyses must be reported in the manuscript, except in rare instances where a registered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and editor must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases the analysis would still be mentioned in the Methods but omitted with justification from the Results.
  • It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between IPA and full review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, appropriately caveated, and reported in a separate section of the Results titled “Post hoc analyses”. Authors should be careful not to base their conclusions entirely on the outcome of statistically significant post hoc
  • Authors will be required to report exact p values, effect sizes, and 95% confidence intervals for all inferential tests using the Neyman-Pearson approach.

The resubmission will ideally be considered by the same reviewers as in the registration stage, but could also be assessed by fresh reviewers. In considering papers at Stage 2, reviewers will be asked to decide (see Appendix below for the more detailed reviewer scoresheets):

1. Whether the data are able to test the authors’ proposed hypotheses by passing the approved outcome-neutral criteria (such as absence of floor and ceiling effects)
2. Whether the Introduction, rationale and stated hypotheses are the same as the approved Stage 1 submission (required)
3. Whether the authors adhered precisely to the registered study procedures
4. Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative
5. Whether the authors’ conclusions are justified given the data

Crucially, reviewers will be informed that editorial decisions will not be based on the perceived importance, novelty, or conclusiveness of the results. Thus while reviewers are free to enter such comments on the record, they will not form a valid basis for editorial decisions.

Manuscript withdrawal and Withdrawn Registrations

It is possible that authors with IPA may wish to withdraw their manuscripts following or during data collection. Possible reasons could include technical error or an inability to complete the study due to other unforeseen circumstances. In all such cases, manuscripts can of course be withdrawn. However, the journal will publicly record each case in a section on an EJP Registered Reports Open Science Framework page (https://osf.io/b3num/) called Withdrawn Registrations. This section will include the authors, proposed title, the abstract from the approved Stage 1 submission, and brief reason(s) for the failure to complete the study. Partial withdrawals are not possible; i.e. authors cannot publish part of a registered study by selectively withdrawing one of the planned experiments. Such cases must lead to withdrawal of the entire paper.

References

Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49, 609–610.

Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., …Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature and Human Behavior, 1, 0021.

Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137–141.

Appendix – Stage 1 and Stage 2 reviewer scoresheets

Stage 1 – reviewer ratings

Overall Contribution

• Importance: Does the paper deal with a key question of personality research, relevant to several research fields within personality psychology and beyond?
• Novelty: Does the paper address novel questions and provides novel insight? Does it explore important but overlooked phenomena, a creative approach to a topic, new or seldom used designs and methods, or understudied samples?

Theoretical background

• Literature review: Do the authors provide a comprehensive and well-integrated overview of previous work relevant to the theoretical rationale and methodological approach?
• Conceptual reasoning: Do the authors provide a thought-through and well-outlined theoretical reasoning and delineation of hypotheses? Are constructs and research problems well-defined and distinguished from each other?

Methodology

• Clarity and detail: Does the paper include all necessary information regarding sampling, procedures and measures, data preparation and aggregation, and analyses to exactly replicate the proposed study procedures and analysis pipeline?
• Robustness: Does the design have sufficiently high statistical power? Does it include cross-cultural, cross-laboratory and/or cross-sample validations of the results?
• Representativeness: Does the design and measures allow for a good representation of the phenomena of interest? Were participants, as well as stimuli, or situational context features representative samples of the universe of relevant participants, stimuli etc.?
• Statistical analyses: Are the planned statistical analyses appropriate and up-to-date? Do the authors include sufficient alternative/supplementary analyses to back-up the robustness of the findings?
• Multiple testing: Do the authors sufficiently address issues of multiple testing?
• Outcome-neutral conditions: Have the authors considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls) for ensuring that the results obtained are able to test the stated hypotheses?

Stage 2 – reviewer ratings

Match of methodology with preregistration

• Outcome-neutral data criteria: Are the data able to test the authors’ proposed hypotheses by passing the approved outcome-neutral criteria (such as absence of floor and ceiling effects)?
• Unaltered introduction and hypotheses: Are the Introduction, rationale and stated hypotheses the same as the approved Stage 1 submission?
• Execution of study procedures: Did the authors adhere precisely to the registered study procedures including data analysis?
• Post hoc analyses: Are any unregistered post hoc analyses added by the authors clearly indicated as such, justified, methodologically sound, and informative?

Results

• Reporting standards: Are all necessary descriptive information reported (including means, standard deviations, and reliabilities for all measures, as well as zero-order correlations between all measures)? Do the authors report effect sizes, confidence (or credible) intervals, and exact p-values?
• Careful language: Does the description and interpretation of results reflect the fact that results cannot be interpreted as ultimate truth (e.g., past tense, non-causal language)?

Discussion

• Careful inferences: Is there a good correspondence between data and results and the inferences drawn? Are the authors’ conclusions justified given the data? Is the writing cautious regarding causality and finality? Are results and effect sizes discussed in an appropriate and context-sensitive way?
• Theoretical discussion: Do the authors provide a thought-through discussion of the conceptual implications of their work? Does the discussion reflect a careful thinking about mechanisms and causality in how the phenomena are linked? Is there a meaningful integration into previous work and competing theories?
• Limitations section: Is the limitations section thorough? Do the authors show awareness to a restricted statistical power, potential alternative interpretations, and potential methodological confounds? Is there a careful discussion of generalizability? Do the authors provide thoughtful and stimulating guidance regarding potential solutions to these limitations in future research?

Quality of writing / presentation

• Clarity and coherence: Is there a well-organized and consistent structure? Does the manuscript have clear and meaningful subsections and –headings? Is the reasoning and labeling consistent throughout the manuscript?

Formal standards: Is the writing correct and concise (spelling, grammar, and style)? Does the manuscript follow APA standards (incl. references, tables, figures, and notes).