1. Connect. Communicate. Collaborate The Performance Enhancement Response Team: Origins and Evolution Ann Harding, HEAnet ann.harding@heanet.ie Toby Rodwell, DANTE toby.rodwell@dante.org.uk Michal Przybylski, PSNC, michalp@man.poznan.pl

2. Connect. Communicate. Collaborate Overview Origins Evolution – from trial to pilot GÉANT – the PERT in action –Organisation –Tools PERT case –FermiLab-Renater throughput

3. Connect. Communicate. Collaborate Origins Development of high capacity WAN, NREN and campus backbones –Still seeing end-to-end performance problems –Need to look beyond the network Internet 2 End to End Performance Initiative (E2EPI) –Initial proposal of a PERT in 2001 –A group of specialists who would be to network performance what CERT is to network security –Link user and expert to solve performance problems –Internet2 concept remained theoretical

4. Connect. Communicate. Collaborate Evolution – European Trial PERT 2002 TF-NGN meeting discussion –Trial PERT for GÉANT Y4 Principle participants –GARR (IT), TERENA, DANTE, SWITCH (CH), CESNET (CZ), HEAnet (IE) and UKERNA (UK) –Entirely dependent on volunteer effort Format –Mailing list hosted by SWITCH (Dec 2002) –Free, open source issue tracker (Roundup) hosted by SWITCH (Mar 2004)

5. Connect. Communicate. Collaborate What’s in a name? First mail to ‘pert-discuss’ stated PERT stood for ‘Performance Emergency Response Team’ Second mail said the ‘E’ should be Enhancement! It was agreed that PERT should properly be ‘Performance Enhancement & Response Team’ –Connotations of the CERT –Removed the misleading ‘Emergency’ element from the title

6. Connect. Communicate. Collaborate Evolution – GÉANT2 Pilot PERT November 2004 GÉANT2 Pilot PERT –Service Activity 3 - Performance and Allocated Capacity for End-users (PACE) New e-mail address for reporting PERT cases Roster of duty Case Managers –Duty Case Manager spending to 2/3 hours per day on open issues PERT Wiki –Diary, to track successes and failures of the pilot –Preliminary knowledgebase

7. Connect. Communicate. Collaborate GÉANT2 Production PERT Connect. Communicate. Collaborate

8. GÉANT2 Production PERT Who? –PERT customers –PERT staff What? –Any academic networked system performance problem –Guaranteed investigation of problems –Consultancy service When? –March 2005 GÉANT production PERT

9. Connect. Communicate. Collaborate Production PERT - Organisation PERT Participants –PERT managers –Full Time Case Managers –Subject Matter Experts –PERT customer –PERT forum moderators

10. Connect. Communicate. Collaborate Production PERT - Tools PERT Ticketing System (PTS) –Ticket management & notification –E-mail and Jabber integration –PERT Diary PERT Knowledgebase (KB) –Wiki-based public knowledgebase –Organised, categorised PERT knowledge –Updatable by any PERT member PERT Public Forum

11. Connect. Communicate. Collaborate Production PERT PTS

12. Connect. Communicate. Collaborate PERT Case - Fermilab to Renater:The Problem Case example from Pilot PERT Problem observed transferring files from fnal.gov (FermiLab) to a machine in Strasbourg The data consisted of many 15MByte files, totalling a few hundred gigabytes The transferring application was "rsync" The bottleneck links were 100Mbps, but the achieved transfer rate was typically 5Mbps

13. Connect. Communicate. Collaborate PERT Case - FermiLab to Renater: The Tests Test machines in similar locations Use web100 tools Memory-memory routinely achieved 90+Mbps, using nttcp –limited system & disk i/o capability on the receiving machine Alternative receiving test machine –long path, fast machines on both ends; –data via ssh via TCP slower than accountable by crypto overhead –highlighted the ssh/ssl buffer limitations

14. Connect. Communicate. Collaborate PERT Case - FermiLab to Renater: Conclusions The FermiLab sender-side rsync server had small TCP buffers The Renater receive-side TCP buffers were too small On Linux, use auto-buffer-tuning on send and receive You've got to have at least 8MBytes of buffer space available for 1xGigE across an ocean Final throughput –memory-memory: 429Mbits/sec –disk-disk: ~20Mbytes/sec (~160Mbits/sec)

15. Connect. Communicate. Collaborate Conclusions Wide problem domain –SMEs crucial to success –Areas of networking, applications, protocols, systems Only service of this kind in the world! –Each new case enhances the service –Here to help

16. Connect. Communicate. Collaborate Acknowledgements The authors would like to acknowledge the pioneering work done in the PERT by Simon Leinen (SWITCH), Victor Reijs (HEAnet) and Sven Ubik (CESNET), and Larry Dunn (Cisco) for his analysis of the FermiLab-Renater case.