1. NOC Tools Donal O’Cearbhaill HEAnet Ltd.

2. Ireland’s National Education and Research Network Provides Internet services to Irish Universities 2005 - Broadband for Schools

3. Free ‘always on’ broadband connectivity to Schools 3 Year Agreement –Dept of Education/Dept of Communication/TIF 3,925+ Schools 7 Access Providers HEAnet backbone network Onward connectivity to Internet & Educational Networks HEAnet Managed Services: Network; Security; E-Mail Broadband for Schools

4. Challenges 4,000 schools Highly contended links A lot of satellite connections SLA/Contract enforcement

5. Installation Rate

6. Monitoring/ISP Infrastructure 28 Debian/Ubuntu servers 4 Fibrenetix disk arrays –Disk based backup –rsync & application level dumps –Syslog nodes PostgreSQL database Aggregation Routers –7301 –PPPoE –GRE Border/Services Routers –6500, 3750

8. Tools SmokePing Nagios Rancid Cacti Netflow

9. SmokePing Latency measurement tool Runs probes in parallel >3,800 hosts RRD backend –Reporting Historical view Acceptance testing Tuning –FPing timeouts decreased –Total number of probes reduced –Satellite frequency reduced

10. Nagios 4,131 services on 3,905 hosts Top 5 number of hosts on nagios.org Populated by SmokePing and memcache –Nagios runs checks serially –>1 hour vs. 15 mins Nagios populates –sidebar alarms –Schools Up Graph

11. Rancid Really Awesome New Cisco confIg Differ 3,296 Router configs Maintains history of changes –Mails changes

12. Cacti 3,900 hosts Data gathering –SNMP –External Perl scripts Graph templating Database driven Cricket: 27 mins –Perl Cacti: <5 mins –Cactid –Custom multithreaded C application

13. Cacti Weathermap

14. Interconnects

15. Netflow NfSen is a graphical web based front end for the nfdump netflow tools Query abuse reports Usage reporting

16. Reporting Daily Reports DNS log reporting Report infected PCs –Top MX lookups –Misconfigurations –Active Directory Netflow –IPs –Schools usage Gigabytes downloaded by schools on 22/03/07: 332 Gigabytes uploaded by schools on 22/03/07 : 48 Total MegaBytes downloaded for Digiweb Satellite: 12834 Total MegaBytes uploaded for Digiweb Satellite: 1202 Total MegaBytes downloaded for Digiweb Wireless: 77578 Total MegaBytes uploaded for Digiweb Wireless: 10217 Total MegaBytes downloaded for ESATBT ADSL: 54352 Total MegaBytes uploaded for ESATBT ADSL: 6632 Total MegaBytes downloaded for HSData Wireless: 3047 Total MegaBytes uploaded for HSData Wireless: 575 …..

17. Logging Syslog server per PoP –Servers –Routers Logcheck –Logfile scanner IP to school identifier –Mapping IP to school

18. Server Monitoring SSH keys –Sharing keys/fingerprints –High overhead SNMP –Less configurable Memcache –Local Perl script –Easy to rollout –Load –Disk Space –Monitor Processes

19. Memcache Distributed memory caching system Low overhead Speed up dynamic database-driven websites by caching data and objects in memory Developed for LiveJournal –Slashdot –Wikipedia –SourceForge Schools –Nagios –Maps –Server status

20. Subversion Modern replacement for CVS Provisioning System –Configs ViewCVS Checkins get mailed Schools-noc –Scripts stored on every server –Automatically updated –cron.d

21. Sidebar Nagios polled every minute Populated into memcache Sidebar alarms Pubcookie single sign-on

25. Provisioning System Services provisioned –CPE router config –Nagios –RADIUS –Cacti –Cisco ACS (TACACS+) –SmokePing –Fortigate (Content filtering) –Maps –DNS –Webhosting

26. Provisioning System Text::Template templating system Data stored in authoritative database PostgreSQL’s INET type is brilliant! Perl scripts generate configlets Added to Subversion Perl/Shell provisioning agents handle service restarts etc. Ability to stop all provisioning

27. Provisioning System Structure

29. Google Maps

31. Random things we’ve encountered Predictable traffic levels Smokeping, Nagios and Cricket/Cacti take a lot of tuning to monitor our network Difficult to achieve high bandwidth and high level of reliability in transparent content filter

32. donal.ocearbhaill@heanet.ie