Sponsored by the National Science Foundation Systematic Experimentation Sarah Edwards GENI Project Office

Sponsored by the National Science Foundation 2 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 Agenda General concepts Resource reservation Installing software

Sponsored by the National Science Foundation 3 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 How do I build repeatable experiments of non-trivial size?

Sponsored by the National Science Foundation 4 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 Systematic Experimentation on a Testbed Combine best practices and methodology from: Science Scientific Method Experiment Design Repeatability Software Engineering Programming Version Control Debugging System Administration Using Node Types to Scale Up Configuration Management Today, focus on concepts from system administration that will help us be better scientists

Sponsored by the National Science Foundation 5 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 Why experiment systematically? Scalability and Repeatability A well designed experiment will be repeatable and systematically scalable Valid Develop a solid understanding of and validate your experimental setup Rigorous Perform a scientifically rigorous (and publishable) study Debugging Easier to debug experiments. Easier to get help debugging. Especially for complex experiments.

Sponsored by the National Science Foundation 6 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 Our Advice for Novice Experimenters Creating Repeatable Computer Science and Networking Experiments on Shared, Public Testbeds by S. Edwards, X. Liu, N. Riga In Operating Systems Review, Jan ‘15 Concise advice for novices on designing experiments to run on a testbed: 1.Formulate a clear plan 2.Automate the execution/analysis of your experiment using best practices 3.Build scalable experiments a.k.a. Start small. Then scale up Plus a case study created by a PhD student. Link to ACM Library For students designing their own experiments

Sponsored by the National Science Foundation 7 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 #1 Formulate a clear plan What vs How Figure out what you are doing then figure out how Draw a picture Statistics and procedure Try out tools and techniques

Sponsored by the National Science Foundation 8 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 #2 Automate Using Best Practices Examples: Software/image Configuration Number of nodes/links Geographic distribution of nodes/links Log all artifacts –RSpec –image –install script –custom software –Measurements –etc Use version control to store your artifacts More on this later… Always know the last working configuration Only change one thing at a time + save what you do

Sponsored by the National Science Foundation 9 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 #3: Start Small. Then Scale Up. Start by building smallest possible topology by hand. Automate as needed. Test and measure as you go. clientserver host OVS switch host router worker master worker node More next…

Sponsored by the National Science Foundation Resource Reservation

Sponsored by the National Science Foundation 11 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 Tools to generate a scaled RSpec (from painful to easy) Copy paste existing RSpec in a text editor CON Tedious AND very error prone Manually create each node using an Rspec Editor (Jacks, jFed) CON Tedious AND somewhat error prone Write a shell script to generate the RSpec –Brecht and Thierry’s 150 node topology PRO Large topologies CON Single purpose Duplicate and Auto-IP buttons in Jacks PRO General purpose CON Medium topologies We’ll use today!!! scaleup tool PRO Large topologies AND General purpose

Sponsored by the National Science Foundation 12 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015

Sponsored by the National Science Foundation Configuration Management with Ansible

Sponsored by the National Science Foundation 14 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 What do experimenters need? Ensure experiment is in a known configuration Easily repeat existing experiment setups (for multiple runs, changing parameters, etc) Reproduce configuration across space and time Easy to build many nodes that look the same Useful for scaling up the size of topologies (space) Useful for rebuilding a topology repeatedly (time)

Sponsored by the National Science Foundation 15 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 CM Systems are Idempotent CM systems are usually idempotent Repeated runs generates the same state Contrast to a shell script which encodes how to transition between two known states The state is usually easy to describe. iperf should be (or not be) installed File foo.html should exist (or not) Describing all possible transitions from every possible previous state is hard. If iperf is not already installed, then install it, else do nothing Known State A Known State B Shell script A -> B Unknown State A Known State B ? CM Idempotent

Sponsored by the National Science Foundation 16 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 Ansible Setup Experimenter ssh  Public key is stored on each remote machine  Private key is stored on local machine  Ansible client installed GENI Slice  Inventory file describes substrate

Sponsored by the National Science Foundation 17 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 Using Ansible Ad Hoc mode Execute individual commands on one or more nodes Playbooks Enforce configurations written in YAML

Sponsored by the National Science Foundation 18 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 Example YAML files # server.yml hosts: server* sudo: True roles: - webserver - nmap # roles/apache/tasks/main.yml name: install apache2 apt: name=apache2 update_cache=yes

Sponsored by the National Science Foundation 19 Systematic Experimentation – Regional Workshop at NEIU – September 18, 2015 In Conclusion … Hyperbole & A Half &