Improving Reproducibility of Experiments in Parallel Computing

It is common practice in Computer Science, in particular in the field of parallel computing, to base scientific contributions on observations and measurements of specific properties of a system under consideration. In fact, experimental works are equally important as are advances in theory. Experiments are especially important in parallel computing since parallel hardware is changing rapidly and no general applicable model of a machine or application exists. However, authors often pay little attention to the description of experimental details, although we believe that the description to reproduce findings should be a major part of every research article. As a consequence, most computational results are hardly and indeed often impossible to reproduce. This lack of reproducibility entails many disadvantages for researchers in Computer Science. Most importantly, extending work of others or comparing new approaches to published methods is almost impossible. Additionally, reviewing articles becomes harder as it is often unclear whether published findings have significant impact or not. Reviewers rarely have the chance to examine the source code or to verify the experimental design or data analysis. In the extreme case, articles describing experimental work have no scientific value when experiments are not reproducible. This project aims at improving the reproducibility of experiments in parallel computing. First, we will critically look at the current state of reproducibility of experiments shown in articles in the field of parallel computing as part of a survey. We will attempt to reproduce experimental results presented on a broad range of articles, ranging from higher-ranked journals to lower-ranked conferences. We will classify all attempts by whether and how far we can reproduce them. Second, with the survey at hand, we will name the requirements for obtaining reproducibility of parallel experiments. We will formalize these requirements and develop a description language for defining the experimental steps to ensure reproducibility. Third, we will develop small portable tools that help researchers to obtain reproducible experiments, e.g., taking snapshots of source code and parameters for each experimental run. As last goal we will apply the formalism and tools developed within this project on our own research topic. Thus, we will attempt to publish an article that allows others to reproduce our experimental findings.

Scientific advances in many research disciplines, such as Computer Science, are driven by the experimental validation of hypotheses, where experimental results are used either to support or to refute a scientific hypothesis. A problem arises if independent researchers are unable to reproduce the work of others. In such cases, a validation or a refutation of published results is impossible.We have addressed the problem of reproducibility of experimental results in the field of parallel computing. Specifically, our research is concerned with data communication operations on larger, parallel machines, such as computer clusters or supercomputers. Data communication in this context is most often done using the Message Passing Interface (MPI), which provides a standardized set of communication operations. Hence, the goal of this project was to analyze and possibly improve the state of reproducibility of experimental results of research works targeting MPI.We defined a minimal set of criteria that scientific papers must fulfill (e.g., providing a link to the source code or describing hardware details), such that other scientists may have a chance to reproduce the findings claimed. We analyzed the state of reproducibility in our domain by compiling a survey, in which we measured the reproducibility potential of papers by applying our defined reproducibility criteria. The outcome was rather disappointing, as most results published in these papers (that we had examined) could not be reproduced.Therefore, a second goal of our project was to improve this situation. In particular, we examined the problem of reproducibly measuring the time to complete several MPI communication operations. Due to a comprehensive and thorough experimental evaluation, we were able to detect experimental factors that every published, experimental description (for this particular case of MPI communication operations) must necessarily contain, such that results can be reproduced and fairly compared. These factors are, for example, the compiler flags, the operating frequency of each processor, or the caching strategy used. Our analysis also helped us to devise a novel experimentation procedure for timing MPI communication operations. This new procedure allows scientists to significantly reduce the variance between experimental runs, which leads to a far better reproducibility.