Measuring Reproducibility in Computer Systems Research

Measuring Reproducibility in Computer Systems Research

A research group in Arizona decided to reproduce as many published claims in computer science as possible. They built a website to track each program until they could fully build it and test it, and you can see the results at this link.


Reproducibility is a cornerstone of the scientific process: only if my colleagues can reproduce my work should they trust its veracity. Excepting special cases, in applied Computer Science reproducing published work should be as simple as going to the authors’ website, downloading their code and data, typing “make” and seeing if the results correspond to the published ones.

To investigate the extent to which Computer Science researchers are willing to share their code and data, and the extend to which this code will actually build with reasonable effort, we performed a study of 613 papers in eight ACM conferences (ASPLOS’12, CCS’12, OOPSLA’12, OSDI’12, PLDI’12, SIGMOD’12, SOSP’11, VLDB’12) and five journals (TACO’9, TISSEC’15, TOCS’30, TODS’37, TOPLAS’34).

The authors also wrote a comprehensive report on their findings. The entire report is worth reading, but we found ‘Anecdote 2’ at the end of the paper particularly interesting. It shows how far they are willing to go to recover materials related to the published claims. We are giving you steps 1 (first email request), 4 (Formal Request for Email Trail) and 5 (Request for the NSF grant application). The paper has lot more details on very interesting responses from the professors and student (tracked through LinkedIn !) .

Step 1.

A.2.1 First emails

On October 14, 2012 we had sent this email to the authors:


I’m Christian Collberg from the University of Arizona. I was

wondering if you have a copy of {SYSTEM-R} that we could

try out? We have a new {SYSTEM} and I’d like to see how

well your system handles our {TECHNIQUES}.

Thank you!


Three of the four email addresses extracted from the author list in the paper failed, the fourth, to {PROFESSOR}, appeared to work. We received no response.

In February 19, 2013, we tried again:

Hi again!

I’m Christian Collberg from the University of Arizona. I wrote

to you earlier to see if I could have a copy of hSYSTEM-Ri so

that we could try it out, but never received a reply. We have a

new {SYSTEM} and I’d like to see how well your system handles


Step 4.

A.2.4 Formal Request for Email Trail

We next sent a request for the emails between the authors, in order to trace the whereabouts of the source code of {SYSTEM-R}:

Pursuant to {OPEN RECORDS STATUTE}, the Open Records Act,

I request copies of all electronic mail between





regarding the article


published in


and the development of the


system described therein.

In accordance with {OPEN RECORDS STATUTE} I request that the

electronic databases to which the following email addresses

are attached be searched:


I also request that any private (non-university) email accounts containing work-related emails be searched.

I am willing to pay applicable fees. These records are sought in furtherance of scholarly research, and I am employed by an Educational,

Non-commercial Scientific Institution. Therefore, I ask that fees,

other than duplication fees, be waived.



and finally stage 5 !!

A.2.5 Request for the NSF grant application

We made a formal request to the NSF for the two applications of the grants that supported the research. In one, the Principal Investigator ({PROFESSOR} in the discussion above) writes:

We wil also make our data and software available to the research community when appropriate.

Needless to say, we never received any communication from {PROFESSOR}, in particular no explanation as to why releasing the code might not have been \appropriate.”

Written by M. //