1. IR Awakens

2. Common problems IR teams face
What's this Talk About?
Common problems IR teams face
What actions we took to address them and results.

3. Worked in State Law Enforcement (5) Manage SOC for USC (11)
About me
Worked in State Law Enforcement (5)
Manage SOC for USC (11)
Internet Storm Center Handler (3)
SANS GSE #76

4. Centrally Manage ~3,000 Desktops
Environment
Enrollment ~42,000
Fac/Staff FT&PT ~ 18,000
8 Campus System
Distributed IT
Each Campus
Each Department
Centrally Manage ~3,000 Desktops

5. Staff (Projects/IR/Engineering/GRC) Slow response time
Challenges
Staff (Projects/IR/Engineering/GRC)
Slow response time
Slow time to collect
Slow analysis
Distributed environment
Unnecessary Response

6. Opportunity for Change
SC DOR Breach

7. 2 year project to implement change New technologies
What did we do?
2 year project to implement change
New technologies
SIEM
Full Packet Capture
Remote IR Tool
Data Discovery
Minimum security standard
Data Discovery (All Systems)
Remote IR tool (Restricted and Compliance)

8. We doubled our staff Added 2 IR staff (5) Added 2 GRC staff (3)
New Staff
We doubled our staff
Added 2 IR staff (5)
Added 2 GRC staff (3)
Added 1 PR staff (1)

9. Tech Talks (Bi-monthly) Mentoring ~18 Months to get staff self-reliant
Training
SANS
Product Training
Tech Talks (Bi-monthly)
Mentoring
~18 Months to get staff self-reliant
This is why some of the stats you’ll improve over time as staff gets more experience.

10. Automated prioritization More eyes on glass
Slow Response Time
Added a SIEM
Moved from home-grown syslog solution
Automated prioritization
More eyes on glass

11. DBIR says 30 days
The more people made a dramatic change in this.

12. Changed IR analysis process
Slow Analysis
Changed IR analysis process
(ISC Post 2016-Aug-24)
FW Log
IDS
HID
BRO
DHCP
NAC
Full
Packet
SMTP
Logs
DNS AD
DLP
Phish S P
Web Shell
C&C
Data
Exfil
Logged-in user
P (Primary) S (Secondary)

13. Reduced number of whole disk collection Full packet capture
Slow Analysis
Reduced number of whole disk collection
Due to new remote IR tool
Compliance, HR, Legal or Confirmed loss
Full packet capture
Quickly determine what happened on network

14. 2014 lull was due to implementations of new tools
2015 Several large incidents and Mentoring staff

15. Hours Per Incident

16. Time from Data Collection to Analysis
2013 (2+ Days)
Try to lookup who owned managed the system (~ 10k lines)
Check NAC,DHCP,NMAP
Contact Admin & Contact User
2015 (Within Minutes)
Pre-deployed IR tool
Slightly improved IP mgmt contact
Network Manager/User isn’t informed until after confirmed compromised
During the delay, admin and/or users had lots of time to start scanning and stomping on the systems.

17. Days to Close
Based on 24hrs a Day

18. Distributed Environment
Policy requirement for centrally managed tools
Meeting and informing Deans and Department heads what is expected
Tools that self-update and little overhead

19. Unnecessary Response
2013
Rely on the admin/users to tell us of sensitive data
In most cases we investigated all confirmed compromises
2015
Systems with commodity malware and no sensitive data we typically don’t investigate
Untold number of hours saved

20. Develop a plan to reduce key metrics
Key Takeaways
Start gather metrics
Develop a plan to reduce key metrics
Have requests ready when incidents happen
Submit them more than once (change the date)
Staff Retention
Training
Progression Path
Quality of Life

21. What Had the Most Impact
Data Discovery Tool
Endpoint Forensics
Full Packet Capture
SIEM
Note: We already had other standard tech in place (FW, SecurityOnion, Syslog)

22. Additional position to manage systems Automation of quarantine Hunting
What's next
Additional position to manage systems
Reduce response time
Automation of quarantine
Hunting
Better use of and collection of Threat Intel

23. Richard Hackley Jonathan Martin Brian Payne James Perry Jeff Whitson
Special Thanks
Richard Hackley
Jonathan Martin
Brian Payne
James Perry
Jeff Whitson

24. https://github.com/tcw3bb
Questions?
@twsecblog