Learning Reproducibility with a Yearly Networking Contest Marco Canini (KAUST) and Jon Crowcroft (University of Cambridge)

What it takes to do reproducible research? It is actually hard work for both authors and reproducers Produce solid artifacts Create, package, share reproducible experiments It’s about the ecosystem Mindset – motivations, incentives, expectations (“golden standards”) Skillset – the art of reproducible research: skills, methods, technologies

Position Goal: increase pervasiveness of reproducible networking research Proposal: yearly international contest with SIGCOMM branding Targeted at plurality of students, especially at early stage career Enable students to learn how to perform reproducible networking research Desired outcome is two-fold: Instill in students an appreciation for reproducible research Provide a learning experience of what reproducibility entails

Non Goals Substitute or conflict with other processes like reproducibility evaluation committees, special-issue journals, etc. Establish a “best of reproductions” competition Add yet another checkbox for reproducible research Focus on advancing the state of the art on a big problem i.e., it’s not a DARPA Grand Challenge for networking We view our proposal as complementary be a cog in the wheel for making research reproducibility more pervasive

What you’ll find in the paper A short review of existing contests in networking and why they do not apply to our proposal Various contests are industry focused or have incompatible goals We connect how a contest would aid with reproducibility Supporting examples from Stanford, MIT and UCLouvain Describe a possible format of the contest Some other implications (longer term)

What you’ll NOT find in the paper It is not a complete blue print or spec to follow We have not trialed it yet

What you’ll NOT find in the paper It is not a complete blue print or spec to follow We have not trialed it yet Your feedback!

So, why a contest? Learn something of the skills, methods and technologies Participants: gain understanding of what’s required to reproduce an experiment and make it reproducible Work with a sim/emulator, set up an environment, do traffic generation, etc. Package and share solution with organizers

So, why a contest? Learn something of the skills, methods and technologies Participants: gain understanding of what’s required to reproduce an experiment and make it reproducible Work with a sim/emulator, set up an environment, do traffic generation, etc. Package and share solution with organizers Contest brings excitement and challenges to stimulate students Makes learning reproducibility a by-product of participating

So, why a contest? Learn something of the skills, methods and technologies Participants: gain understanding of what’s required to reproduce an experiment and make it reproducible Work with a sim/emulator, set up an environment, do traffic generation, etc. Package and share solution with organizers Enrich testbeds, test harnesses, datasets, and platforms Organizers: gain experiences with evaluation of submissions and share these experiences with the community (reviewing committees) Advance our “best practices” for research reproducibility Support the sociotechnical system, including archives, testbeds, budget, and staff

On the contest format Reproduce experiments from published papers + independent verification of results + gain understanding of difficulties and recommendations of what to improve - unclear how to rank participants (success might be high) Well-defined problem with quantitative goal + score submissions objectively - reproducing research is not an integral part

Our proposal Objective is not to reproduce research per se but a first phase creates a baseline by building upon an existing approach and reproducing a certain experiment from a paper in a second phase focus is to find a better solution Contest design has to promote clean and well-structured solutions, packaged and shared with organizers Disincentive “hack it together” mentality Make reproduction integral part but take it a step further

Some details Students compete in team Not a hackathon; should last a few weeks Possible run down for research group R during year Y Acquainted with skeleton code and small exercises Reproduce an experiment and validate on supplied workload Start to improve upon baseline solution; submit and watch the scoreboard Fine tune solutions; submit and get ranked on a final (secret) workload Final prize and award at SIGCOMM Give more visibility to reproducibility in CVs Y+=1; R = select()

Summary and open questions Make the case that a contest promotes better research reproducibility ecosystem Learn the skillset and mindset Aid the reproduction evaluation committees with tooling and training Success: an increase of reproducible research by contest participants How does all this sound? How to define tractable and interesting problems for the contest? Any other ideas and contributors? Would you encourage your students to participate?