1. Learning Reproducibility with a Yearly Networking Contest
Marco Canini (KAUST) and Jon Crowcroft (University of Cambridge)

2. What it takes to do reproducible research?
It is actually hard work for both authors and reproducers
Produce solid artifacts
Create, package, share reproducible experiments
It’s about the ecosystem
Mindset – motivations, incentives, expectations (“golden standards”)
Skillset – the art of reproducible research: skills, methods, technologies

3. Position
Goal: increase pervasiveness of reproducible networking research
Proposal: yearly international contest with SIGCOMM branding
Targeted at plurality of students, especially at early stage career
Enable students to learn how to perform reproducible networking research
Desired outcome is two-fold:
Instill in students an appreciation for reproducible research
Provide a learning experience of what reproducibility entails

4. Non Goals Substitute or conflict with other processes
like reproducibility evaluation committees, special-issue journals, etc.
Establish a “best of reproductions” competition
Add yet another checkbox for reproducible research
Focus on advancing the state of the art on a big problem
i.e., it’s not a DARPA Grand Challenge for networking
We view our proposal as complementary
be a cog in the wheel for making research reproducibility more pervasive

5. What you’ll find in the paper
A short review of existing contests in networking and why they do not apply to our proposal
Various contests are industry focused or have incompatible goals
We connect how a contest would aid with reproducibility
Supporting examples from Stanford, MIT and UCLouvain
Describe a possible format of the contest
Some other implications (longer term)

6. What you’ll NOT find in the paper
It is not a complete blue print or spec to follow
We have not trialed it yet

7. What you’ll NOT find in the paper
It is not a complete blue print or spec to follow
We have not trialed it yet
Your feedback!

8. So, why a contest?
Learn something of the skills, methods and technologies
Participants: gain understanding of what’s required to reproduce an experiment and make it reproducible
Work with a sim/emulator, set up an environment, do traffic generation, etc.
Package and share solution with organizers

9. So, why a contest?
Learn something of the skills, methods and technologies
Participants: gain understanding of what’s required to reproduce an experiment and make it reproducible
Work with a sim/emulator, set up an environment, do traffic generation, etc.
Package and share solution with organizers
Contest brings excitement and challenges to stimulate students
Makes learning reproducibility a by-product of participating

10. So, why a contest?
Learn something of the skills, methods and technologies
Participants: gain understanding of what’s required to reproduce an experiment and make it reproducible
Work with a sim/emulator, set up an environment, do traffic generation, etc.
Package and share solution with organizers
Enrich testbeds, test harnesses, datasets, and platforms
Organizers: gain experiences with evaluation of submissions and share these experiences with the community (reviewing committees)
Advance our “best practices” for research reproducibility
Support the sociotechnical system, including archives, testbeds, budget, and staff

11. On the contest format Reproduce experiments from published papers
+ independent verification of results
+ gain understanding of difficulties and recommendations of what to improve
- unclear how to rank participants (success might be high)
Well-defined problem with quantitative goal
+ score submissions objectively
- reproducing research is not an integral part

12. Our proposal Objective is not to reproduce research per se
but a first phase creates a baseline by building upon an existing approach and reproducing a certain experiment from a paper
in a second phase focus is to find a better solution
Contest design has to promote clean and well-structured solutions, packaged and shared with organizers
Disincentive “hack it together” mentality
Make reproduction integral part but take it a step further

13. Some details Students compete in team
Not a hackathon; should last a few weeks
Possible run down for research group R during year Y
Acquainted with skeleton code and small exercises
Reproduce an experiment and validate on supplied workload
Start to improve upon baseline solution; submit and watch the scoreboard
Fine tune solutions; submit and get ranked on a final (secret) workload
Final prize and award at SIGCOMM
Give more visibility to reproducibility in CVs
Y+=1; R = select()

14. Summary and open questions
Make the case that a contest promotes better research reproducibility ecosystem
Learn the skillset and mindset
Aid the reproduction evaluation committees with tooling and training
Success: an increase of reproducible research by contest participants
How does all this sound?
How to define tractable and interesting problems for the contest?
Any other ideas and contributors?
Would you encourage your students to participate?