1. National Center for Supercomputing Applications University of Illinois at Urbana–Champaign The Evolution and Future of Network Security Monitoring at NCSA CLHS 2015 Alex Withers alexw1@illinois.edu

2. Service Categories Incident Response (24/7) Preventative Security Security Monitoring Security Awareness Security Collaboration Service Menu Staff Training & Education Security Policy & Process Development Network Security Monitoring Active Response & Blocking of Attacks 24/7 Incident Response Security Hardening Guidelines Syslog Collection & Monitoring Developing Secure System Architectures Forensic Postmortems Custom Instrumented SSH monitoring Secure Bastion or Jump Hosts OAuth Portal and MyProxy CA service Vulnerability Scanning & Prioritization Security Vetting & Hardening Firewall Management Brute-force Mitigation Risk Assessments Security Auditing Cyber Security Directorate at the NCSA

3. Network Security Monitoring Network zones (define trust levels/internal monitoring) Bastion/jump Hosts (2fa) Centralized log collection One-way admin access Passive Monitoring Network monitored with Bro and Netflow Syslogs collected and monitored Keystrokes of interactive sessions logged Active response w/ BHR Vulnerability scanning internally

4. Network Security Monitoring Network zones (define trust levels/internal monitoring) Bastion/jump Hosts (2fa) Centralized log collection One-way admin access Passive Monitoring Network monitored with Bro and Netflow Syslogs collected and monitored Keystrokes of interactive sessions logged Active response w/ BHR Vulnerability scanning internally

5. Monitoring log and network data Bro and syslog logs collected Splunk and OSSEC Interactive SSH sessions logged by instrumenting sshd Gives greater visibility, don’t have to rely on shell history Data securely piped through to Bro and logged Bro triggers response to certain events: Port scanning SSH password guessing Network scanning Syslogs fed into OSSEC and rules can trigger a response: Failed attempts to login

6. Some Attack Statistics Attacks against BlueWaters ( http://www.ncsa.illinois.edu/BlueWaters/ ) http://www.ncsa.illinois.edu/BlueWaters/ In the past 6 months: Network scans: 780,000 Brute force attempts (i.e. SSH): 2800 ShellShock attacks: 700 Finding these types of attacks becomes more difficult as we move towards 100GB (more noise)

7. Active response mechanisms Blackhole Routing used to block “bad actors” Events in Bro and syslogs can trigger a block Currently using “singularity” BHR software: Based on quagga https://github.com/JustinAzoff/singularity Moving towards a new BHR software: https://github.com/justinazoff/bhr-site Supports an API, ability to query history Backend is a separate component

8. Beginnings of NSM at the NCSA Pre-2010’s Two network links: 1G and 10G Two bro nodes monitoring each link Tapping infrastructure simple and easy to maintain Bro alerts monitored and acted upon manually One node per link is too inefficient and inflexible Too slow in reacting to reconnaissance and other attacks

9. The move to 10G Effort to monitor and secure started ~4 years ago 16 10G WAN links ~12 internal network links Bro setup scaled dramatically Moved to Bro cluster architecture Myricom with proprietary “snifferv2” drivers Needed a way to manage the proliferation of WAN and Campus links… Gigamon provided the flexibility Collapse several links into a single fiber (Link Aggregation Groups) Rely on the fact that bandwidth is often under utilized

10. Present Network Setup All traffic captured bi-directionally: 10GB WAN fiber connections (XSEDE, ICCN, MREN, Internet2, etc.) Internal fiber connections Miscellaneous copper-based networks, i.e. VPN Methods for capture include inline taps and span ports 6 Gigamon G2404s work to balance and distribute the load: 4 collectors condense traffic into 10GB Link Aggregation Groups 1 aggregator hashes streams symmetrically into 8 fiber pairs 1 splitter distributes traffic to Bro clusters for analysis

12. Busy Bro nodes…

13. The 100G future Replacing many 10G WAN connections with 4 100G links Monitoring 10GB links brings many challenges, 100GB only more so… Again, rely on specialized network hardware, but machines can’t really do 100GB NICs We need more help: Cut out network traffic that 1) consumes large amounts of bandwidth and 2) we’re sure won’t interest us from a security perspective

15. Network volume and needless overhead Finding “elephant flows” Takes up large volumes of network bandwidth Takes up CPU cycles (common bottleneck) Bro examines traffic and looks for flows that are large volume such as file transfers Inform the Arista to no longer capture this flow by blacklisting the src/dest hosts-src/dest ports Flow is dropped, but we’ve still capture the initial headers, handshake, etc. and ignore the rest of the payload Bro will continue to monitor the flow and remove the block when counter no longer increments

16. Additional performance considerations Each connection monitored balance over two 10GB connections by the Arista Anticipating some flows will saturate a 10GB connection By having two 10GB connections, reduce the chances of a large flow affecting smaller flows Bro cluster nodes using two Myricom 10GB NICs with snifferv3 proprietary drivers Drivers will hash traffic and load balance between individual Bro processes

17. Summary 10GB environment managable: Networking equipment (Gigamon) Scale with more machines, CPUs and NICs 100GB environment needs more careful consideration: Networking equipment helps (Arista) Can’t just throw more machines at the problem Need to actively respond to the traffic being monitored and … Need to ensure CPU’s are utilized effectively