Registered Reports

Language Learning is one of very few journals in the language sciences to accept manuscript submissions under the Registered Report category. The following information provides an overview of Registered Reports for interested authors and for applicants to the Language Learning Early Career Research Grant program. The Grant program now includes Registered Reports as a priority category for these competitive awards (see the announcements in the September issue). For more specific details about submitting a Registered Report, please see our Submission Guidelines.

Registered Reports. Registered Reports are a form of empirical article in which a substantial part of the manuscript (including the literature review, methods, and proposed analyses) is reviewed and then pre-registered prior to the main data being collected. This format is designed to reduce bias and other questionable research practices, particularly in deductive science, while also allowing researchers the flexibility to conduct subsequent unregistered analyses and to report serendipitous findings. As well as being appropriate for hypothesis-driven research, the format can also be suitable for other approaches, such as observation, case study, or ethnography, wherever some amount of the methods can be determined in advance of data collection. 

The cornerstone of the Registered Reports format is that part of the manuscript will be assessed prior to data collection so that high quality submissions are given in-principle acceptance before the actual study is carried out.

Initial submissions will include the following aspects of the proposed study:

• literature review that clearly and fully justifies the research question(s) and study design and methods, 

• hypotheses (for hypothesis-driven research),

• design and procedures, 

• full data elicitation materials,

• coding and scoring procedures, analysis pipelines, a statistical power analysis (where appropriate), and 

• pilot data (where applicable). 

Authors are strongly encouraged to plan the study with a view to eventually making the materials, methods and procedures, raw data, coding, and analysis procedures available via a publicly accessible and sustained file-sharing service such as IRIS and/or the OSF. Making data publicly available usually requires researchers to seek IRB (ethical review) approval for this and to make participants aware that their (usually anonymized) data will be made public and used by others in the future. This is in line with Language Learning’s policy to encourage open science practices and methodological transparency, and to promote and facilitate replicable research.

Review procedures. The review process is illustrated in the flow-chart below. Initial submissions (manuscript and all accompanying materials) will be evaluated by the Editors for scientific significance, rigor, and relevance to the scope of the journal. Those that meet the necessary criteria will then be sent for in-depth peer review (Stage 1). Following review, the submission will then either be rejected, returned to authors for minor or major revisions, or accepted in principle.

Once accepted, authors pre-register their Stage 1 manuscript and all accompanying materials on a public and sustainable file-sharing service, via, for example, the dedicated Stage 1 RR registration portal on the OSF. This pre-registration can be embargoed, if the authors wish, so that the manuscript and materials do not become publicly available until (no later than) acceptance of the Stage 2 manuscript.

If authors wish to carry out an open multi-site study (such as a multi-site replication), they can make their Stage 1 manuscript, materials, and procedures publicly available (not embargoed), in order to seek potential multi-site collaborators; authors would have signalled this intention as part of the Stage 1 submission.

At this point, authors are granted In-Principle Acceptance (IPA). The authors are then committed to publishing the completed study in Language Learning and, provided they adhere to their pre-registered study and materials (which will be evaluated during the Stage 2 review process), their manuscript will be accepted for publication with Language Learning.

When undertaking the study, authors should strictly adhere to the peer-reviewed and approved procedures as documented in the pre-registration. If deviations to the approved procedures occur, authors should contact the Editors and describe these in detail. The Editors may consult with the Stage 1 reviewers and possibly one or more new reviewer(s). Where deemed necessary, authors can be offered the opportunity to withdraw the manuscript from the Registered Report route and pursue the standard publication route. In this case, the completed study could be submitted for consideration by Language Learning following its regular review procedures, which would include one or more rounds of peer review.

Documentation from the accepted Stage 1 submission will be stored on Scholar One in the corresponding author’s account associated with the submission, while they conduct the study. It will be made available to reviewers when the Stage 2 manuscript is submitted. 

When the study is complete, authors should submit their finalized manuscript for a Stage 2 review. The manuscript (and all the Stage 1 pre-registered documents) will be sent to one or more of the original reviewer(s) and possibly one (or, under unusual circumstances, more) new reviewer(s). Stage 2 review will include checks for fidelity to the pre-registered methods, data and analysis checks, and an evaluation of the interpretation of the findings. This second stage review process is normally a lot faster and smoother than the review procedure for standard article types, because the number and nature of issues that can be raised by reviewers at this stage is severely constrained. Evidence of the date of data collection will also be required, such as time-stamped data collection (e.g., via upload to a data storage facility) or submission of the raw data (e.g., transcriptions of audio files, audio recordings) to Language Learning as soon as possible after they have been collected.

If the above criteria are satisfied, the manuscript will be published regardless of the actual results.

On acceptance, authors will be expected to make available their materials, methods, procedures, raw data, coding, and analysis procedures in a publicly accessible and sustained file-sharing service such as IRIS and/or the OSF. As with all manuscripts submitted to Language Learning, authors will be recognized for their Open Science practices (for data, materials, and pre-registration as described in our Author Guidelines).