1. Good Things Come In Small Packages
Creative perfSONAR
Antoine Delvaux
PSNC — GÉANT perfSONAR Service Manager
Linz — Austria
31 May 2017
You don’t always need big devices or powerful machines to improve things.
We wanted to have more user adoption for perfSONAR, all we knew was: we had to be creative.

2. More perfSONAR? Get creative!
What is this about?
Small Nodes!
Do you operate Small Nodes?
As easy as can be!
The Mesh, The Dashboard and The Colors
You want examples?
We’ve got examples!
Useful? Really?
So, tell me, what’s next?

3. TNC16: Small Nodes in GÉANT Launch
Objectives:
Increase perfSONAR adoption amongst GÉANT partners and users
Early 2016: 50 european pS nodes in 1300 nodes worldwide (4 %)
Show case the current perfSONAR
Demonstrate perfSONAR usability on low cost devices
Give first hand experience to new users, update the experience of old time users
Increase the global perfSONAR footprint
BoF session TNC16: Lots of interest
20 small nodes to 20 organisations
5 Universities
12 NREN (8 europeans + 3 africans + 1 asian)
GÉANT + NORDUnet
1 outside the R&E community (RIPE NCC)
pS adoption one year ago was still low
More pS deployments benefits all pS users
Packed room, and it was during lunch time!

4. A Small Node? Hardware Software The Small Nodes Low cost: around 200 €
Capable of 1 gbps throughput
Disk for local storage
Hardware
BRIX BACE-3150 devices
Intel Celeron 4 cores
Single 1 gbps NIC
Software
CentOS 6
perfSONAR toolkit pre-configured
DHCP setup
Auto-updates (CentOS and perfSONAR)
200 € is low cost compared to thousands of € a fully fledged server can cost.
To have 1 gbps need to have 1 gbps interface but also sufficient CPU power and RAM.
Intel Celeron makes it easy to install regular CentOS.

5. The Small Nodes — Inside

6. Tell the group your node is online
The Small Nodes — Setup
3 installation steps
Plug the network
Plug the power
Turn it on
3 configuration steps
Assign IP addresses (v4 and v6, preferably)
Fill administrative details (Name, Country, Organisation, Geo info)
Select closest NTP servers (automatically or manually)
Tell the group your node is online
Check forward and reverse DNS are functional and correct
Quick Start guide distributed with devices — just in case
Automatic
Lookup Service registration
24 h later: first measurements
Make it as easy as possible

7. Central Management of the mesh
Managing the nodes
Users and GÉANT
Communication through mailing list
A GÉANT SA2 activity — perfSONAR Service
Monitoring
SNMP polling, MRTG based
Central Management of the mesh
Mesh definition
Central MA access
Auto updates
Managed by YUM
CentOS + perfSONAR
Can be skipped if too long.
No alerting, don’t want to bother users/admins at this point.
Should be reliable enough.
Central MA access through IP based authentification.

8. New features and changes:
perfSONAR 4.0 upgrade
Released on April 17th
New features and changes:
New graphs
pScheduler to replace BWCTL and regular testing daemon
MaDDash 2.0 with alerting
CentOS7, full Debian support
Recording of webinar:
Small nodes in automatic upgrade mode
Yum config
Unattended upgrade
All went well
Dashboard
New matrix: routes/paths

9. Controlled from central server
Measurement Mesh
Controlled from central server
Single configuration file
Measurement mesh agent on small nodes, automatically picks up mesh config changes
IPv4 mesh
20 small nodes
5 GÉANT MP
IPv6 mesh
16 small nodes
Choice of GÉANT MP?
Network / Geographical dispersion
Not too loaded (chosen with GÉANT Ops advice)
Reliable measurement points
perfSONAR is most useful when regular measurements are scheduled
Easier way to do that: setup a measurement mesh
Centrally managed, so the small nodes users don’t need to be perfSONAR experts to start using pS
GÉANT MP? Reliable MP both in terms of uptime/availability and of network and server performance.

10. Measurement Mesh — Visualisation
Central dashboard: MaDDash
Colored matrix showing latest measurement results and thresholds
Detailed and historical graphs
Configured measurements:
Throughput up to 1 gbps
Loss
Latency: OWD
Routes / Paths changes
MaDDash is part of the perfSONAR distribution
It provides a global overview of the network performance and helps quickly spot issues

11. Measurement Mesh — Example 1
First steps with mesh:
Make sure all nodes are taking measurements
Make sure all measurements are stored in the central MA
Look for Orange squares: means no data
Node unreachable
No measurements happening
Reason: Firewall issue
Outgoing measurements ok
Incoming measurements impossible
First steps as a mesh administrator (us, GÉANT SA2)
It is also possible to change dashboard colors.
There is no example A, screen shot taken from the report.

12. Measurement Mesh — Example 2
Dashboard color meaning:
Green: good performance
Yellow: degraded performance
Red: poor performance
Orange: no data
(other color scheme possible)
Red square: poor performance
Thresholds are indicated top of matrix
Throughput < 500 Mbps
Thresholds are configurable
Reason: Saturated / Low bandwidth uplink

13. What can the users do with a Small Node?
Owners can use the perfSONAR web interface
Look at mesh measurements results
Configure new measurements
Owners can use the CLI
On demand measurements
Look at the way perfSONAR works
Small Nodes are public
Mesh measurements are public (dashboard)
Small Nodes are listed on pS Lookup Service
Can be target of measurements
Getting more comfortable with perfSONAR and performance measurements in general
Is it useful?

14. Loss measurements: GÉANT Budapest MP
Yes, it is useful!
One example of usefulness, happened just after a few weeks of operation.
Dashboard showing losses to one GÉANT MP
Happened just after a few weeks of operation

15. Details: Budapest MP to JANET
Packet loss ~0.3%
Loss rate is relatively small 0.3 % or 3 packets per 1000

16. Details: Budapest MP to PIONIER
Packet loss ~0.3%
Similar loss rate

17. Measurement mesh operations: some guidelines
perfSONAR is a multi-domain performance monitoring tool
Need to involve all parties
MP operators, network operators
GÉANT Budapest MP investigation example
Contact with GÉANT Ops team
Heavy traffic on the switch during the night (backup)
Packet drops resulted
Remedy: Connect perfSONAR MP directly to router
Need to be in close relation with other partners in the mesh (here GÉANT Ops team)
Some MP get replaced
Some MP load increase
We changed the measurement matrix from 4 MP to 5 MP
If you use a foreign MP regularly, get in touch with the owners!
Always remember that perfSONAR is made for multi-domain and the pSmall mesh is a multi-domain setup.

18. User Survey in November 2016 Outcomes:
User Acceptance
User Survey in November 2016
13 answers out of 20 project participants
Outcomes:
Overall average rating of 4.77 out of 5
100% of respondents recommends project continuation
100% of respondents recommends usage of perfSONAR on low cost devices
92% would like to have more GÉANT managed small nodes in their network
75% interested in installing and running perfSONAR on small nodes themselves
58% interested in installing and running fully-fledged perfSONAR server themselves
58% are interested in having their own measurement mesh in their network
Survey was 6 months after the start of the project.

19. Increased perfSONAR coverage in Europe
Tangible results
Increased perfSONAR coverage in Europe
50 out of about 1300 early 2016 : 4% (estimation, from map)
275 out of 2093 August 2016: 13% (number of interfaces registered to LS)
557 out of 2231 May 2017 : 25% (number of interfaces registered to LS)
About 10 times more pS deployments in 18 months!
More user interest

20. Continuing activity and nodes management Extending the mesh
What’s next?
Continuing activity and nodes management
Extending the mesh
Objective: 1 Small Node per NREN
Open to new self funded small nodes
Ready to take part in the mesh?
Subscribe to list at
Announce yourself!
Get in touch with us!
Considering possibility to buy and distribute more Small Devices