Reproducibility

View project on GitHub

What?

This repository contains the result of a study on the reproducibility of networking papers. The goals is to be presented in an editorial to be published in SIGCOMM CCR.

Why ?

It was not common for networking researchers to reproduce results shown in the literature because of many reasons :

  1. not enough details in papers on scenario;
  2. no access to the source code;
  3. diffuclty to reproduce the same conditions;
  4. ...
Recently networking researchers have started to alleviate the problem by describing in further details their experimentations scenarios, making their code and script available,... But even networking researchers make some effort, the result is not always reproducible because of data privacy, some patent, too much work in order to make a public release, lose script (after some year),...

Ideally, any researcher should be able to reproduce results shown by her colleague, not only to verify the results published in the paper but also to easily evaluate and debug the protocol on other scenarios with different scales, compare it with other approaches and possibly propose enhancements. TODO: complete

Methodology

  1. For a given conference, we read each paper and identify all useful information in order to reproduce it.
  2. If there is some forgotten information, like typically a URL to a code source, a measurement scrit or simply a plot script.
  3. When we receive a response, we check the given information an update the dataset according this informations.
  4. At the end, we plan to check more in depth if the paper is reproduce. It's means that we really try to reproduce the result.

Data set

Actually, our study take into account for conferences and two workshop :

  • Workshop HotNets 2014 and 2013;
  • Conference SIGCOMM 2014 and 2013;
  • Conference coNEXT 2014 and 2013.

In order to show the reproducibility of networking papers, we create a .csv file which contains some informations. The dataset file. can be found on our github repository. TODO: explain which informations and why

Reproduce our result

TODO: explain

Authors and Contributors

This study is perform in cooperation of professor Bonaventure Olivier (@obonaventure) and Houtain Nicolas (@nhoutain).

Support or Contact

You can join us by sending a mail to olivier.bonaventure@uclouvain.be and nicolas.houtain@student.uclouvain.be

If there is some mistake with the data set file, don't hesitate to contributing to our open source github by create a issue or sending a pull request You can find here more information about How contribute