Nonstandard Errors

A. Project Design

Before starting the #fincap experiment, we filed a pre-analysis plan (PAP) with the Open Science Foundation (OSF; https://osf.io/h82aj/). The original version of this paper contains the results of the analysis outlined in the PAP. This original version remains available as Tinbergen Institute Discussion Paper TI 2021-102/IV. Subsequent feedback from various presentations and from reviewers at the The Journal of Finance have led to the results presented here. Relative to the PAP, we now use robust methods to cope with unanticipated extreme outliers, we account for multiple testing, and we add a multiverse analysis to deepen insights. Appendix B reconciles the current results with those in the original version.

In a nutshell, the #fincap experiment is about multiple RTs independently testing the same hypotheses on the same sample. We refer to these hypotheses as RT hypotheses and to the sample as the RT sample. This labelling distinguishes the hypotheses that RTs test from the hypotheses that we test based on the results generated by RTs and PEs (Section 3).12

The RT sample is a plain-vanilla trade sample for the EuroStoxx 50 index futures with the addition of a principal-agent flag.13 For each side to a trade (i.e., buy and sell), we therefore know whether the exchange members traded for their own account or for a client. The sample runs from 2002 through 2018 and contains 720 million trade records. These index futures are among the world's most actively traded index derivatives. They give investors exposure to Europe, or, more precisely, to a basket of euro-area blue-chip equities. With the exception of over-the-counter activity, all trading is done through an electronic limit-order book (see, for example, Parlour and Seppi (2008), for details on limit-order book markets).

Note that, by design, there is considerable variation across RT hypotheses in the level of abstraction. RT-H1, for example, corresponds to the relatively abstract notion of market efficiency. In contrast, RT-H3 corresponds to the share of client volume in total volume, which should be relatively straightforward to calculate because, in the RT sample, each buy and sell trade is flagged agent (client) or principal (proprietary).

RTs are asked to test these RT hypotheses by estimating an average yearly change for a self-proposed measure.14 They are further asked to report SEs for these estimates. We compute the ratio of the two, which we refer to as the implied t-value, or t-value for short.

RTs write a short academic paper in which they present and discuss their findings. These papers are evaluated by PEs who were recruited outside the set of researchers who registered as RTs. RT papers were randomly and evenly assigned to PEs in such a way that each paper is evaluated twice, and each PE evaluates nine or 10 papers. PEs score the papers by providing an overall rating as well as a rating per RT hypothesis. They do so in a single-blind process: PEs see the names of RTs, but not vice versa.15 The reason for single-blind instead of double-blind is to incentivize RTs to exercise maximum effort.

PEs are asked to motivate their scores in a feedback form where they are encouraged to add constructive feedback. RTs receive this feedback unabridged, and are allowed to update their results based on it. Importantly, the design of #fincap was common knowledge to all because it had been available on a dedicated website before registration opened (see footnote 4).

B. Hypotheses

Before running the experiment, we translated the project's four overarching questions into a set of preregistered hypotheses. These hypotheses all center on the dispersion in estimates across RTs. Our main measure is the IQR, which we refer to as NSE. All hypotheses are stated as null hypotheses and tests are two-sided.

C. Statistical Framework

Conceptually, the distribution of $$X$$ could be obtained by bootstrapping. Such a procedure is infeasible, however, because it requires that researchers rerun their analysis for every new draw of the sample. Instead, we use MHT results to make the testing procedure feasible.

Before turning to MHT, we pause for a moment and take stock of what is available to us. The #fincap sample consists of estimates $$x_j$$, along with their SE $$s_j$$. This is useful, but misses information on the covariance among all possible pairs of estimates across researchers.

To account for multiple testing, we rely on well-developed statistical theory. If one aims to test at a level of 5% for a family of $$N$$ tests, then individual tests should be performed with a $$(5/N)\%$$ critical value, if the test statistics are mutually independent (Bonferroni (1936), Šidák (1967), Harvey, Liu, and Zhu (2016)).23

In summary, we propose an NSE test where the null hypothesis is that there is no dispersion in RFEs. We use a Bonferroni adjustment of significance levels to account for multiple testing. The test is conservative, because Bonferroni assumes independence. As pointed out in footnote 20, estimates are likely to correlate across researchers, in which case, the effective number of tests is likely to be smaller than the actual number of tests. In the implementation, we add a trivial extension in which correlations between estimates are calibrated based on our multiverse analysis (Section 11). We close the section by discussing an alternative test and pointing out a caveat.

Alternative test. Note that a natural alternative to the proposed test is to simply test whether IQR is statistically different from zero. We did not use this shortcut because our focus is on whether there is any dispersion at all in estimates across researchers. Although we use IQR to express dispersion in a single number, the underlying interest is whether the distribution in estimates is nondegenerate.

Caveat. We point out a potential caveat. The procedure to obtain a conservative test on RFEs implicitly assumes that SEs reported by researchers are consistent estimators of the true SEs. This may not be true if (some) researchers report nonrobust SEs. Nonrobust SEs tend to be smaller because they ignore commonalities. To the extent that this holds, NSE tests tend to turn significant more often. NSEs themselves, however, remain consistent estimators.24

A. Summary Statistics

Table I summarizes our stage 1 sample by means of three sets of statistics, organized in three panels.25 Panel A summarizes the characteristics of the #fincap community. It consists of 164 RTs and 34 PEs. Maximum RT size is two members, which is the size of 79% of RTs.

The statistics testify to the high quality of the #fincap community. The percent of RTs that have at least one publication in a top finance or economics journal is 31% (see footnote 6 for the list of journals). For PEs, this figure is 85%. The percent of RTs who have at least one member who is tenured at the associate or full professor level is 52%. For PEs, this figure is 88%. Feedback seems to come from more established scholars, which likely mirrors reality.

RT members and PEs cover the global academic finance community reasonably well (see Figure IA.II in the Internet Appendix). RT members reside in 34 countries, with most residing in the United States (51 out of 293). PEs reside in 13 countries with, again, most residing in the United States (13 out of 34). The strong skew toward the United States is not surprising given that the more senior, well-published finance scholars are predominantly affiliated with U.S. universities.

Most RTs and PEs appear to have the appropriate background for testing the RT hypotheses on the RT sample. Their average self-reported scores on having experience in the field of empirical finance is 8.1 for RTs and 8.4 for PEs on a scale from 0 (low) to 10. For experience with market liquidity, these average scores are 6.9 for RTs and 7.8 for PEs. There is considerable variation around these averages as the SDs range from 1.7 to 2.4. When it comes to working with samples as large as the RT sample (720 million trade records) most RTs and PEs seem up to it as 65% of RTs have worked with samples at least as large and 88% of PEs. In sum, we believe that the group of #fincap participants is reasonably representative of the academic community in empirical finance/liquidity.

Panel B of Table I shows that the average quality of the RT analysis is solid, but the dispersion is large. The average reproducibility score is 64.5 on a scale from 0 (low) to 100 (see footnote 19). This is high when benchmarked against other studies on reproducibility (Colliard, Hurlin, and Pérignon (2021)). The accompanying SD is 43.7, which shows that there is large variation across RTs: Most code reproduces either close to perfectly or barely at all. The paper quality scores provided by PEs show a similar pattern, albeit with less dispersion. The average score across RTs is 6.2 on a scale from 0 (low) to 10, with an SD of 2.0.

Panel C provides descriptive statistics on the distribution of results across RTs, both by RT hypothesis and by type of result (estimate, SE, and t-value). Since our focus is on dispersion in estimates across RTs, we relegate discussion of RT medians to Appendix C. More specifically, Appendix C discusses the RT hypotheses in depth and summarizes what RTs as a group seem to find, with a focus on the across-RT median instead of the across-RT IQR (i.e., the NSE).

Perhaps the most salient feature of the evidence in Panel C is that there is substantial variation across RTs for both all RT hypotheses and all types of results. Panel A in Figure 1 illustrates this result for estimates. For RT-H1 on market efficiency, for example, the median estimate across RTs is −1.1% with an IQR of 6.7%. Even for RT-H3, which is a seemingly straightforward calculation of market share, the dispersion is sizable: The IQR is 1.2% with a median of −3.3%. The variation in IQR across RT hypotheses suggests that the more abstract a hypothesis is, the larger the IQR is. Finally, the figure illustrates that there are extreme outliers for all RT hypotheses, which motivates our analysis in terms of robust statistics.

NSE test results. Is the dispersion in estimates statistically significant? Table II presents the NSE test results. The null of no dispersion in RFE is rejected for all RT hypotheses at the 0.5% (family) significance level. The conservative Bonferroni adjustment in Panel A yields at least 11 estimates that are individually significantly different from the median (RT-H6), and at most 38 significant differences (RT-H3). There are significant estimates both above and below the median for all RT hypotheses.

If, instead of assuming zero correlation across test statistics as in Bonferroni, one calibrates them based on bootstrapping from the multiverse analysis (Section 11), results change as shown in Panel B. The implied “effective” number of tests is much lower than the 164 tests used in Bonferroni—it ranges from 21 (RT-H3) to 86 (RT-H6). The factor by which significance levels are adjusted is therefore up to almost seven times smaller than what Bonferroni suggests (i.e., 164/24 = 6.8). The result is that more differences, indeed, become significant. The increases are moderate though, with at most two more differences becoming significant.

In sum, the statistics presented thus far show that there is substantial dispersion across RTs, in terms of not only their estimates, but also their team quality, reproducibility score, and in PE rating. In the next subsection, we use this dispersion to test the first three hypotheses, that is, we ask: is there more dispersion in estimates for lower quality teams, for results that are harder to reproduce, or for lower quality papers?

B. Hypothesis Tests

The results on the three sets of hypotheses are discussed in the following three subsections. SEs in the quantile regressions account for correlation in residuals by adding RT hypothesis fixed effects and by clustering per RT across all stages.

B.1. Covariates for Stage 1 Dispersion (H1–H3)

The first set of hypotheses relates NSEs to various quality variables. One of these is team quality, which we measure using the first principal component (PC1) of five standardized quality proxies (see H1 in Section 3). PC1 explains 38.3% of total variance. Moreover, it loads positively on all quality proxies—it loads strongest on publications and weakest on big data, but, importantly, it loads positively on all of them. Table IA.IV in the Internet Appendix provides detailed results on the PCA.

Table III summarizes results of the stage 1 quantile regressions, where the dependent variables are the 10^th, 25^th, 50^th, 75^th, and 90^th percentiles of the distribution in estimates across RTs. Figure 2 depicts the results by showing how a one-SD increase in each covariate affects IQR (i.e., NSE) and IDR. Taken together, these results allow us to test the first three hypotheses that relate quality variables to dispersion in estimates.

Table III. Stage 1 Quantile Regressions

This table presents results of quantile regressions that characterize how the distribution of stage 1 estimates covaries with various quality metrics. These metrics are team quality, reproducibility score, and (demeaned) peer rating. The three quality variables have been standardized and, subsequently, multiplied by the IQR for each RT hypothesis. The coefficients therefore measure the result of a one-standard-deviation change, expressed in terms of IQR units. * and ** correspond to significance at the 5% and 0.5% level, respectively.
	Q(0.10)	Q(0.25)	Q(0.50)	Q(0.75)	Q(0.90)
Team quality (standardized/scaled)	0.597**	0.004	0.002	0.032**	−0.325**
	(0.030)	(0.014)	(0.007)	(0.012)	(0.030)
Reproducibility score (standardized/scaled)	0.473**	0.109**	−0.001	−0.142**	−0.555**
	(0.033)	(0.014)	(0.007)	(0.011)	(0.028)
Average rating (standardized/scaled)	0.766**	0.230**	−0.001	−0.097**	−0.626**
	(0.034)	(0.014)	(0.007)	(0.011)	(0.028)
Dummy RT-H1 Efficiency	−29.592**	−6.099**	−1.132**	0.939**	9.057**
	(0.813)	(0.340)	(0.166)	(0.269)	(0.708)
Dummy RT-H2 RSpread	−15.933**	−3.930**	−0.017	3.674**	22.451**
	(0.849)	(0.342)	(0.166)	(0.268)	(0.705)
Dummy RT-H3 Client Volume	−5.629**	−3.789**	−3.319**	−2.386**	0.221
	(0.836)	(0.339)	(0.166)	(0.268)	(0.721)
Dummy RT-H4 Client RSpread	−12.089**	−2.437**	0.162	4.161**	19.619**
	(0.837)	(0.340)	(0.166)	(0.266)	(0.704)
Dummy RT-H5 Client MOrders	−2.479**	−0.744*	−0.001	0.297	1.625*
	(0.837)	(0.339)	(0.166)	(0.268)	(0.721)
Dummy RT-H6 GTR	−194.457**	−21.385**	0.022	5.137**	65.203**
	(0.806)	(0.337)	(0.167)	(0.268)	(0.679)
#Observations	984	984	984	984	984

First, we find that higher team quality coincides with somewhat larger IQR but with smaller IDR. The effect of team quality on the 25^th percentile is not significant except for the 75^th percentile, for which it is significantly positive. The economic magnitude is small, however, as can be seen in Figure 2. A one-SD increase in quality raises IQR by only $$(0.032-0.004)\times 7.2 = 0.2$$ percentage points (pps), where 7.2 is the average IQR across hypotheses (see Panel C of Table I). This increase of 0.2 pps implies a relative increase of 2.8%.26 In contrast, a one-SD increase in team quality reduces IDR by 6.7 pps (−11.9%, since average IDR is 56.3). This result is a significant increase in the first decile and a significant reduction in the ninth decile. These findings suggest that higher quality teams are less likely to report extreme estimates.

If one replaces team quality by the five quality variables on which it is based, a more nuanced picture emerges (Table IA.V in the Internet Appendix). Among the statistically significant and sizable relationships, we find that a one-SD increase in academic seniority (i.e., an associate/full professor in the team) reduces IQR by 1.4 pps (−19.4%) and a one-SD increase in team size reduces IQR by 0.9 pps (−12.5%). A one-SD increase in top publications, however, increases IQR by 1.9 pps (+26.4%). These three variables are positively correlated, which explains why we find that the (aggregate) team variable has a relatively small effect on IQR. For IDR, the effects are of the same sign but larger in magnitude, at −19.4, −7.0, and +6.1 pps respectively (−34.4%, −12.4%, and +10.8%). Note that now the negative effects really dominate, which explains why IDR covaries negatively with team quality. In sum, these findings suggest that well-published scholars seem to disagree more, but this effect is offset by the presence of a senior scholar or a second team member.27

Second, all percentiles covary significantly with reproducibility except for the median. The 10^th and 25^th percentiles covary positively and the 75^th and the 90^th percentiles covary negatively. Figure 2 shows that these changes are sizable. A one-SD increase in reproducibility reduces IQR by 1.8 pps (−25.0%) and IDR by 7.5 pps (−13.3%). In sum, better reproducibility lowers overall dispersion.

Third, the results for paper quality mirror those of reproducibility, albeit are a bit stronger in magnitude. The 10^th and 25^th percentiles covary significantly positively, and the 75^th and 90^th percentiles covary significantly negatively. A one-SD increase in paper quality reduces IQR by 2.4 pps (−33.3%) and IDR by 13.6 pps (−17.9%). Higher rated papers exhibit less dispersion in estimates.

In summary, the evidence on the first three hypotheses is such that the null of no covariation is rejected for all three. Generally, higher quality is associated with less dispersion in estimates.

B.2. Convergence across Stages? (H4)

The analysis of first-stage results shows that dispersion in estimates is sizable and statistically significant. We next ask whether peer feedback creates convergence. In particular, we examine whether dispersion in estimates declines in the three subsequent stages in which teams get feedback from peers. This is the focus of the fourth hypothesis.

Table IV presents results of quantile regressions to explain the dispersion in estimates in all four stages (thus far, only stage 1 has been analyzed). To account for heterogeneity in dispersion across RT hypotheses, the explanatory variables are stage dummies that are multiplied by stage 1 (estimate) IQR separately for each hypothesis RT hypothesis. The coefficients therefore measure a stage effect, expressed in IQR units.

Table IV. All-Stages Quantile Regressions

This table presents results of quantile regressions that characterize how the distribution of estimates varies across all stages of the #fincap project. The stage dummies have been multiplied by the (stage 1) IQR for each RT hypothesis. The coefficients therefore measure the effect in terms of IQR units. Standard errors account for correlation in residuals by adding RT hypothesis fixed effects and by clustering by RT across all stages. * and ** correspond to significance at the 5% and 0.5% level, respectively.
	Q(0.10)	Q(0.25)	Q(0.50)	Q(0.75)	Q(0.90)
Dummy Stage 2 - Dummy Stage 1	2.44*	0.07	−0.00	−0.06	−0.73
	(1.18)	(0.14)	(0.01)	(0.06)	(0.64)
Dummy Stage 3 - Dummy Stage 2	0.94*	0.15	0.00	−0.09	−0.73
	(0.41)	(0.09)	(0.01)	(0.05)	(0.40)
Dummy Stage 4 - Dummy Stage 3	0.21*	0.06*	0.00	−0.04	−0.25*
	(0.09)	(0.03)	(0.01)	(0.03)	(0.11)
Dummy Stage 4 - Dummy Stage 1	3.59**	0.28*	−0.00	−0.19**	−1.71**
	(1.23)	(0.14)	(0.01)	(0.05)	(0.50)
RT-hypotheses dummies	Yes	Yes	Yes	Yes	Yes
#Observations	3,936	3,936	3,936	3,936	3,936

The evidence rejects the null hypothesis of no convergence across all stages. All changes across consecutive stages are positive for the 10^th and 25^th percentiles, and negative for the 75^th and 90^th percentiles. The majority, however, are insignificant. In contrast, the total change across stages is significant for all of the percentiles considered at the 5% level and for all but one at the 0.5% level. Taken together, these results show that there is significant convergence from the first to the last stage, but a decomposition across the various stages lacks significance.

Figure 3 illustrates that the convergence is sizable. Panel A shows that the total decline in IQR is 3.4 pps (−47.2%). The decline appears evenly distributed across the stages, although this decomposition is mostly insignificant. Panel B shows that the total decline in IDR is even larger, at 38.4 pps (−68.2%). More than half of the effect appears to occur from the first to the second stage, where RTs receive anonymized feedback from two PEs. However, this result is only weakly significant, since only the increase in the first decile is weakly significant (i.e., at the 5% level, not at the 0.5% level).

B.3. Are RT Beliefs on Dispersion in Estimates Accurate? (H5)

The fifth and final hypothesis focuses on whether RTs are aware of the dispersion in estimates across teams. Beliefs have been solicited in an incentivized way. All teams were asked to predict SDs in estimates across teams.28 We randomly selected 20% of all RTs and paid each of them $300 if one of their predictions (randomly drawn) was within 50% of the realized SD. Details on the reward scheme are in the instruction sheet they received before reporting their beliefs (see Section III.I in the Internet Appendix). The hypothesis pertains to stage 1 estimates, because beliefs are solicited for this stage only.

As H5 is stated in terms of the average belief being correct, testing it requires a test on the equality of means: the mean belief about SDs in estimates across teams, and the SDs of these estimates in the population. We measure the relative distance between beliefs and realizations using the test statistic $$D$$,

$$$$\begin{align} D = \frac{1}{6 n}\sum _{i,j} {\left(\frac{\mathit {BeliefOnSD}_{ij} - \mathit {RealizationOfSD}_j}{\mathit {RealizationOfSD}_j} \right)}, \end{align}$$$$(6)

where $$\mathit {BeliefOnSD}_{ij}$$ is the belief of team $$i$$ on the SD in estimates across teams for RT hypothesis $$j$$, and $$\mathit {RealizationOfSD}_j$$ is the realized SD for this RT hypothesis in the raw sample.29 The distribution of $$D$$ under the null of equal means is obtained by bootstrapping. For details on the bootstrap procedure, we refer the reader to Appendix E.

Figure 4 plots the distribution of beliefs on SDs, along with realized SDs depicted by red dots. It illustrates that the vast majority of teams underestimate dispersion in estimates. The IQR denoted by the boxes is consistently below the red dot, which implies that at least 75% of the teams underestimate the dispersion.

One might think that teams simply overlook the extreme values that make realized SDs explode. This does not appear to be the case, however, because even if one trims the estimates by removing the top and bottom 2.5%, the IQR box stays below these “trimmed” realized SDs, depicted by orange dots. The only exception is RT-H3, for which the orange dot is just within the top of the box.

The formal test results are in Table IA.VI of the Internet Appendix. Pooling across all RT hypotheses, the test statistic shows that the predicted SD is 71.7% below the realized SD. This underestimation is significant at the 0.5% level. Similar results hold for all RT hypotheses individually except RT-H3, for which the underestimation is insignificant. Its value was also lowest, with only 9.0% underestimation. RT-H3 is an hypothesis on market shares that is arguably relatively straightforward to test. In summary, the vast majority of tests show significant underestimation and we therefore firmly reject the null that beliefs on the dispersion in estimates are accurate.

C. Digging Deeper: A Multiverse Analysis

NSEs in #fincap are significant and sizable. Why? Can we somehow identify which forks on the analysis path cause most of the dispersion? More specifically, can we rank key forks on the path according to the degree of refraction they cause in the light the sample sheds on the research question at hand? We employ a multiverse analysis to address these questions.

Steegen et al. (2016) coined the term multiverse analysis to emphasize that data construction involves multiple decisions. The sample that enters the analysis, therefore, is a function of the set of reasonable choices, with the sample becoming a (p. 702) “many worlds or multiverse of data sets.” A particular result of an analysis then becomes a distribution of results (because samples vary). We generalize this approach by adding decision forks for the part of the analysis that follows the sample construction (e.g., the choice of econometric model).

The strength of a multiverse analysis is that it reveals how sensitive an estimate is to a particular fork on the analysis path. It does so by studying how much the estimates refract when varying across all reasonable alternatives at the fork. For example, let there be $$N$$ reasonable analysis paths. Now suppose there are $$k\le N$$ reasonable alternatives at the $$j$$^th fork. Then, the $$N$$ estimates associated with the $$N$$ paths are sorted into $$k$$ sets, depending on the alternative selected at the fork. The degree to which the results differ across the $$k$$ sets determines how sensitive results are to the $$j$$^th fork. We measure the degree to which $$k$$ distributions differ by a $$k$$-sample Anderson-Darling (AD) test. Appendix F discusses the AD test in detail, including why it fits our application particularly well. AD is a standard option in the Boba software that we use (Liu et al. (2021)).30

To make the multiverse feasible, we identify key forks on the analysis path and, for each fork, we ask RTs to select the alternative they picked among a set of predefined alternatives. This is done by means of a questionnaire that all participants filled out after the experiment. The choice of forks and the alternatives at each fork is informed by the short papers RTs wrote for #fincap. The discretization of the decision space enables us to project the large space of realized analysis paths onto a manageable space of “representative” paths. Table V provides an overview of all forks for the six RT hypotheses. It lists the alternatives at each fork, along with the fraction of RTs that picked them (depicted in Figure IA.V of the Internet Appendix).

For each fork, we also asked RTs to rate the fit between the alternative they picked from the set and what they actually did in #fincap. Their average rating ranges between 4.0 for RT-H6 and 4.4 for RT-H3 on a scale from 1 “Far from what we did” to 5 “Very close to what we did” (see Figure IA.IV in the Internet Appendix). We therefore believe that the multiverse analysis is representative of the #fincap analysis itself.

A multiverse analysis is powerful but resource intensive. The table illustrates that the analysis becomes very large very quickly. For RT-H6, for example, the nine forks generate $$2\times 2\times 3\times 4\times 3\times 4\times 2\times 3\times 2 = 6,912$$ possible paths. Not all possible paths are equally reasonable, however, and the #fincap data help us select the most reasonable ones. The result is a weighted multiverse, with untraveled paths get tiny zero weight. The other paths get weights proportional to the number of teams that picked the path. The vast majority of paths, however, was picked by only one team, so the size of the multiverse is slightly less than 164 (the actual number varies across RT hypotheses).

The analysis is done for the original sample as well as for 1,000 bootstrapped samples. These additional samples are needed to estimate the correlations in test statistics across paths. These correlations are used to adjust significance levels when accounting for MHT. This is used in assessing whether NSEs are statistically significant, and whether individual estimates are statistically significant (see Panel B in Table II and Table IA.VII, where the latter is in the Internet Appendix, respectively). Each RT hypothesis therefore requires processing the 720 million trade records almost 164,000 times.31

Results. Figure 5 illustrates that the multiverse is able to generate dispersion in estimates that is on par with the dispersion in reported estimates. The box plots for reported estimates are drawn in gray, overlaid by the multiverse box plots in color. The large dispersion in multiverse estimates is remarkable, since they are based on a few decisions only.32

Figure 6 further highlights some idiosyncratic sensitivities. For RT-H1, for example, the second-most sensitive fork is the frequencies that are picked to assess the deviation from a random walk. Further analysis reveals that when comparing high frequencies, such as one-second to one-minute returns, almost all analyses exhibit a decline in market efficiency. However, when comparing low frequencies, such as daily to monthly returns, about half of the analyses show an increase in market efficiency whereas the other half show a decline.

Another example is the retain-negative-sign fork, which is the most sensitive one for RT-H6. The decision that each team had to make is whether a negative number that becomes more negative yields a positive percentage change or a negative percentage change. The first case emphasizes that a (negative) number becomes magnified, whereas the second emphasizes a negative trend (i.e., “retain a negative sign”). We find that 21% of the teams picked the first option and 79% picked the second one. It is not surprising that mapping an estimate from the positive to the negative domain causes strong refraction in estimates. This is an example of how a decision that each team might have thought was a trivial one (in the sense that there is only one option) can generate NSE.

D. Alternative Explanations

Having presented our results, it is useful to discuss alternative explanations. In Section 6 above, we argue that the group of #fincap participants is representative of the academic community in empirical finance/liquidity. One might nonetheless wonder to what extent the sizable NSEs are due to the presence of inexperienced researchers testing unsuitable hypotheses with little effort. We believe this is unlikely to be the case for the following three reasons.

Experience. Aware of this potential pitfall, we selectively approached researchers (for RTs and PEs) whom we knew were sufficiently experienced in the field. When signing up, they ticked a box indicating that they understood that participating in #fincap requires research experience in empirical finance/liquidity and in the analysis of large data sets. Ticking the box further means that they acknowledge that one of the team members held a PhD in finance or economics. After ticking the box, researchers had to specify in an open text box why they believe they meet these requirements. We parsed the content of this box to ensure that the team qualifies before accepting them into #fincap (see Figure A in the Internet Appendix for the sign-up sheet).

Hypotheses. We proceeded with care when designing the RT hypotheses. Early versions were shared with senior scholars, and their feedback helped us fine-tune the RT hypotheses. We therefore feel confident that the RT hypotheses are suitable and well-motivated hypotheses to test with the RT sample (see Figure E in the Internet Appendix for the RT instruction sheet, which shows how RT hypotheses were presented to RTs).

Related to the suitability question, one might wonder whether vagueness of an RT hypothesis might be a viable alternative explanation for sizable NSEs. To address this concern, we included a very precise RT hypothesis: RT-H3 on client volume share. The results for RT-H3 show that NSEs can be sizable, even for relatively precise hypotheses. It is true, however, that NSEs tend to be lower for the more precise RT hypotheses.

Effort. We incentivized RTs to exert effort by providing them with the following information (before they signed up): the deadlines of the various stages so that they could plan for them; their nonanonymized paper would be evaluated by senior peer reviewers; the top-five (anonymized) papers would be announced to all others33; and only those who complete all stages become coauthors. In addition to these incentives, we believe that most scientists are propelled by an intrinsic motivation to do good research.

Looking back, we have several reasons to believe that researchers did indeed exert serious effort. First, only four out of 168 RTs failed to complete all stages; 123 out of 168 teams (73.2%) handed in their stage 1 report at least a day early, and none of the teams seriously breached any deadline. Second, the average reproducibility score was 64.5 on a scale from 0 (low) to 100, which is high in comparison to what has been reported in other reproducibility studies (Colliard, Hurlin, and Pérignon (2021)). Finally, the average paper quality was 6.2 on a scale from zero (low) to 10. As for PEs, we also believe they exerted serious effort, because all who signed up as a PE completed their reviews on time.