The Reproducibility of Experimental Economics

There is increasing concern about reproducibility in science, i.e. to what extent statistically significant published research findings are true positives or false positives. Factors contributing to a lack of reproducibility include low statistical power, the testing of hypotheses with low prior probability of being true, and publication bias. As an interdisciplinary team spanning the globe we launch a program for assessing and improving the reproducibility in the social sciences, specifically in experimental economics. To achieve that we use a three-pronged program: The first part is systematic replication of published studies, where we will introduce a replication program of experimental economics studies published in leading economics journals and leading general science journals. The second is using prediction markets to quantify the reproducibility of published research findings. The prediction markets allow us to estimate probabilities for the tested hypotheses being true at different stages of scientific discovery. The third tool we use are surveys/questionnaires before and after the prediction markets which will allow us to assess which priors traders held and how they were updated during the prediction market. Based on the surveys and prediction markets the probability that the tested hypothesis is true can be derived before and after the replication. Our first study, reporting on the replication of 18 studies in two leading economics journals, was accepted for publication in Science (Camerer et al. 2016). There we report replication rates of 61% to 78% (depending on specification), which are the highest rates of any discipline we are aware of. This also reflects the expectation of researchers in the field, as the survey and prediction markets we conducted before the replications delivered expected replication rates of 71% and 75%, respectively.

In the context of this project, replication studies were carried out and new scientific territory was broken with a large and innovative crowd analysis project in the field of neurosciences. The most innovative sub-project NARPS (https://www.narps.info/) is a crowd analysis project in the field of neuroscience. Together with colleagues from the USA, Sweden and Israel, brain scans were carried out on 108 people while they were going through an experimental set-up. We made the resulting raw data available to 70 research teams around the world (who volunteered for analysis), who then independently evaluated the data and submitted their results on nine research questions to us. What we were interested in was the consistency (or inconsistency) of the responses from the responding teams, since in the very new field of neuroscience (where brain scans play a decisive role) it is not clear how clear (and thus replicable) results are. A key finding is that the results are very inconsistent; because not two of the 70 analysis teams had the same analysis path (certainly not the same results). At the same time, however, the analysis teams were "overconfident", i.e. expected three times more significant results than they ultimately gave. We also combined this data analysis with a survey and with forecast markets, which all in all produced a very exciting and comprehensive research project. We also looked at the risk perception of financial professionals around the world in this project, comparing the predictions of financial experts and laypeople on US and UK stock returns, and the replicability of scientific results in the social sciences.