Basic Configuration & Deployment South Carolina State University SPAN Workshop Matt Zekauskas, matt@internet2.edu 2017-05-19 This document is a result of work by the perfSONAR Project (http://www.perfsonar.net) and is licensed under CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/).

Host Considerations http://docs.perfsonar.net/install_hardware.html Dedicated perfSONAR hardware is best Server class is a good choice Desktop/Laptop/Mini (Mac, Shuttle) can be problematic, but work in a diagnostic capacity Other applications running may perturb results (and measurement could hurt essential services) Running Latency and Throughput on the Same Server If you can, devote 2 interfaces – the toolkit will support this. If you can’t, note that Throughput tests can cause increased latency and loss (latency tests on a throughput host are still useful however) © 2017, perfsonar.net September 22, 2018September 22, 2018

Host Considerations http://docs.perfsonar.net/install_hardware.html 1Gbps vs 10Gbps testers There are a number of problem that only show up at speeds above 1Gbps – both are still super useful Virtual Machines do not always work well as perfSONAR hosts (use specific) Clock sync issues are a bit of a factor throughput is reduced significantly for 10G hosts VM technology and motherboard technology has come a long way, YMMV NAGIOS/SNMP/1G Throughput are good choices for a VM, OWAMP/10G Throughput are not © 2017, perfsonar.net September 22, 2018September 22, 2018

Toolkit Installation A CentOS netinstall is the quickest way to start http://docs.perfsonar.net/install_centos_netinstall.html Others explained here http://docs.perfsonar.net/install_getting.html © 2017, perfsonar.net September 22, 2018September 22, 2018

Toolkit Configuration The best source of information is here: http://docs.perfsonar.net http://docs.perfsonar.net/manage_choose.html There are two use cases for configuration: Installation is done via the use of an entire distribution via ISO, or use of RPM packages Diagnostic: Configure host to allow tests from others, don’t test on your own (e.g. beacon) Permanent: Participate in a mesh, or configure tests as an island © 2017, perfsonar.net September 22, 2018September 22, 2018

Toolkit Configuration - Web © 2017, perfsonar.net September 22, 2018September 22, 2018

Toolkit Configuration - Web We can do most other things via the web interface once we have a user that can auth against it (do-it-yourselfers can still hand edit config – we just won’t deal with that here) Some things to check – for all see items under “perfSONAR Toolkit” on http://docs.perfsonar.net/ : Administrative info http://docs.perfsonar.net/manage_admin_info.html NTP (time keeping) http://docs.perfsonar.net/manage_ntp.html Turning on/off services http://docs.perfsonar.net/manage_services.html Configuring some tests (directly) http://docs.perfsonar.net/manage_regular_tests.html © 2017, perfsonar.net September 22, 2018September 22, 2018

NTP Scheduling, and one-way latency (OWAMP) depend on good knowledge of local time Note that it may take a day to fully stabilize the clock Pick 4 – 5 Close servers for NTP We have a fast way to do this, or you can manually select Can also add your own servers if you don’t like ours © 2017, perfsonar.net September 22, 2018September 22, 2018

Address Families Note that the tools are IPv4 and IPv6 capable. If a functional IPv6 address is available, that will be the default chosen. To force one or the other in the test interface, pay attention to the test members: To truly force one or the other, enter the address for a host, instead of a hostname. © 2017, perfsonar.net September 22, 2018September 22, 2018

Transition: Config to Regular Testing perfSONAR web interface is meant to be simple Enabling this on campus is the first step to seeing a simulation of performance for a bulk data tool. Ideally you would place the perfSONAR server where the users are (e.g if they are traversing a firewall still, why don’t you learn their pain)? Configuring regular tests is systematic – pick regional and far away destinations. Dust off netflow, and see where the data is going – configure tests to those locations too. © 2017, perfsonar.net September 22, 2018September 22, 2018

Regular Use The best way to get buy in (at all levels) is to use the machines: Encourage 1st line network support/help desk techs to see what the tools do. Useful for basic things: Traceroute beacon Visualization of existing regular tests Encourage 2nd line support to experiment with the command line tools. Develop reports of performance (e.g. query data from the archive and create an XLS spreadsheet/chart) © 2017, perfsonar.net September 22, 2018September 22, 2018

Importance of Regular Testing We can’t wait for users to report problems and then fix them (soft failures can go unreported for years!) Things just break sometimes Failing optics Somebody messed around in a patch panel and kinked a fiber Hardware goes bad Problems that get fixed have a way of coming back System defaults come back after hardware/software upgrades New employees may not know why the previous employee set things up a certain way and back out fixes Important to continually collect, archive, and alert on active throughput test results © 2017, perfsonar.net September 22, 2018September 22, 2018

Regular Testing There are a few schools of thought regarding this. Beacon: Let others test to you (e.g. no regular configuration is needed) Island: Pick some hosts to test to – you store the data locally. No coordination with others is needed. Can be done via web interface Mesh: full coordination between you and others (e.g. consume a testing configuration that includes tests to everyone, and incorporate into a visualization) © 2017, perfsonar.net September 22, 2018September 22, 2018

Regular Testing - Beacon The beacon setup is typically employed by a network provider (regional, backbone, exchange point) A service to the users (allows people to test into the network) Can be configured with Layer 2 connectivity if needed If no regular tests are scheduled, minimum requirements for local storage. Makes the most sense to enable all services (bandwidth and latency) © 2017, perfsonar.net September 22, 2018September 22, 2018

Regular Testing – Island The island setup allows a site to test against any number of the 2000+ perfSONAR nodes around the world, and store the data locally. No coordination required with other sites Allows a view of near horizon testing (e.g. short latency – campus, regional) and far horizon (backbone network, remote collaborators). OWAMP is particularly useful for determining packet loss in the previous cases. Throughput will not be as valuable when the latency is small © 2017, perfsonar.net September 22, 2018September 22, 2018

Regular Testing - Mesh A full mesh requires more coordination: A full mesh means all hosts involved are running the same test configuration A partial mesh could mean only a small number of related hosts are running a testing configuration In either case – bandwidth and latency will be valuable test cases © 2017, perfsonar.net September 22, 2018September 22, 2018

Develop a Test Plan What are you going to measure? Achievable bandwidth 2-3 regional destinations 4-8 important collaborators 4-8 (more if you are willing, especially to start) times per day to each destination 20-30 second tests within a region, longer across oceans and continents Loss/Availability/Latency OWAMP: ~10-20 collaborators over diverse paths Interface Utilization & Errors (via SNMP) What are you going to do with the results? NAGIOS Alerts Reports to user community Dashboard © 2017, perfsonar.net September 22, 2018September 22, 2018

perfSONAR Deployment Locations Critical to deploy such that you can test with useful semantics perfSONAR hosts allow parts of the path to be tested separately Reduced visibility for devices between perfSONAR hosts Must rely on counters or other means where perfSONAR can’t go Effective test methodology derived from protocol behavior TCP suffers much more from packet loss as latency increases TCP is more likely to cause loss as latency increases Testing should leverage this in two ways Design tests so that they are likely to fail if there is a problem Mimic the behavior of production traffic as much as possible Note: don’t design your tests to succeed The point is not to “be green” even if there are problems The point is to find problems when they come up so that the problems are fixed quickly © 2017, perfsonar.net September 22, 2018September 22, 2018

Sample Site Deployment © 2017, perfsonar.net September 22, 2018September 22, 2018

perfSONAR Dashboard - MaDDash © 2017, perfsonar.net September 22, 2018September 22, 2018

MaDDash and The Mesh Config Measurement results are more useful when they can be “seen”, because this implies they will be acted on. MaDDash is a software package that can be used to visualize the results of many perfSONAR tests The Mesh Config is a way to manage multiple nodes, by giving them a uniform testing schedule E.g. this is in contrast to the other method of configuration shown – manually entering our tests. Changes node from ‘testing as an island’ to being a part of a larger testing strategy Start Here: http://docs.perfsonar.net/multi_overview.html © 2017, perfsonar.net September 22, 2018September 22, 2018

Basic Configuration & Deployment South Carolina State University SPAN Workshop Matt Zekauskas, matt@internet2.edu 2017-05-19 This document is a result of work by the perfSONAR Project (http://www.perfsonar.net) and is licensed under CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/).