AT&T Heartbleed Response (OpenSSL Brief) Adam Jones - CISM, CGEIT, CISSP, 6 σ GB ITO Global Infrastructure Operations July 24, 2014 © 2014 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. The information contained herein is not an offer, commitment, representation or warranty by AT&T and is subject to change.

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 2 OpenSSL Zero Day Vulnerability 4/7 - Cert Issued for OpenSSL 4/7 - Cloudfare.com challenges internet to hack their keys. Two participants reported success. 4/7 - Evidence of active attempts to exploit the vulnerability surfaced shortly after this event. 4/8 - CNET: "We were able to scrape a Yahoo username & password via the Heartbleed bug," tweeted Ronald Prins of security firm Fox-IT, showing a censored example. Added developer Scott Galloway, "Ok, ran my heartbleed script for 5 minutes, now have a list of 200 usernames and passwords for yahoo mail...TRIVIAL!" tweeted Ronald PrinsFox-ITcensored exampleScott Galloway 4/21 - CNBC: “Obamacare enrollees urged to change passwords over Heartbleed bug” Video Placeholder

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 3 Story Line StartupEmergingSustainingResponse Close Desktops - very low exposure Network Elements - low exposure UNIX/Linux hardware and application processes had the majority of exposure while modest given the overall enterprise. This is the high level recap of AT&T’s OpenSSL Heartbleed critical response. Risk Review - Zero day alert issued. Evaluating exposure Release management processes begin testing and staging of available patches. SWAT mode Processes confirm some exposure. Scanning processes increased. Reporting enhancements. Communication plans commence. Scans identify hardware issue. Status change to standard operations. Communication plans continue. Social media in heavy usage internally. Update for hardware issue deployed. Final issues resolved. Patching wraps up. Steps to update certificates and passwords continue. Ongoing processes continue for any new hosts coming online.

4 Lessons Learned & Best Practices © 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

5 Operational Recommendations - Lessons Learned Inventory Assets, Valid Owners, Hosted Application (Installed Applications), Application Contacts and Management (Business Unit association) Hostname, FQDN, IP Address, OS, OS Version, Patch Levels, Patch Date. Communication Plans Delivery - Application Contacts, Operations contacts (SA, DBA, Supervising Managers) Executives - SA, DBA Executives, Application Executives. Social Media - Strongly encourages for larger enterprise environments. Reporting Recommended - focus on open database relationships, common primary and secondary keys, databases of applications and each application having current accurate relations to core inventory. Online reports should be intuitive and actionable. Export functionality with pivot table structures is recommended for increased productivity. Release Management Critical - This is imperative for availability and rapid remediation. Mature processes for testing and certifying release packages prior to distribution is pivotal to success. Best practice a core functional set of teams, favorable is strong processes with cross functional teams. Layered Security Cyber defense as it is well documented is based on layered security controls. Rapid remediation and or containment is dependent on multiple controls working in harmony (IDS, IPS, Scans, Patch Management, Reporting, etc.). Slide 4

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 6 Reporting Logical View Server Tracking (Unique Servers) Response Tracking Summary Views Detailed Scheduling Application Level Tracking Response Submissions Detailed views for compliance, scheduling Evidence (Patched vs Non Patched) App Risk - Direct and SharedCompliance and Risk Views ServersAppsResults Operations Systems Level Data OS views per platform, what is compliant vs planned vs documented. Client response interfaces for organizing what clients have sent in and which apps have not sent responses. Real time audit trail. Clients App Towers (Hosted Applications) Automated Communications - App Teams. Data driven reports for GM Communications. App Instance Tracking. Interfaces for reconciling response for questions, scheduling requests and jeopardy submissions.

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 7 Communications Best Practice 1. Audience - Inventory dynamically feeds automation that sends this message to the correct audience once triggered. Target users are application contacts using impacted servers. 2. Media Types - Use multiple media forms in one communication (i.e. , slide deck, video overviews). 3. Reference Material - Have mature reference areas available (wiki, social media site, any online reporting sites, video references). 4. Required Action - Must include clear, actionable steps. No communication will be 100% successful but the steps have to be very meaningful. * Recommendations are based on standard processes for internal operations.

8 Feedback, Questions © 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

9 Questions and Feedback Audience - Our team would like your input and questions. Scanning How is your company scanning full ip address ranges for all ports internal and external? Inventory IP Address reconciliation - How is your company managing unknown IP Addresses that do not map to a known owner? Reporting How is your company discovering non Microsoft platforms? How standard is your environment? Application Availability How does your company maintain availability of your enterprise applications while expediting emergency changes into the environment? Change control? Standard change windows? Testing? Certificates - How did you handle these changes? Social Media Is your company using social media to collaborate on security remediation efforts? References: ISACA Incident Management and Response Center/Research/ResearchDeliverables/Pages/Incident- Management-and-Response.aspx Center/Research/ResearchDeliverables/Pages/Incident- Management-and-Response.aspx ISACA Security Incident Management Audit/Assurance Program Center/Research/ResearchDeliverables/Pages/Security- Incident-Management-Audit-Assurance-Program.aspx Center/Research/ResearchDeliverables/Pages/Security- Incident-Management-Audit-Assurance-Program.aspx AT&T ThreatTraq Spotlight video.cfm/2014/4/9/AT&T-ThreatTraq-Spotlight- Heartbleed video.cfm/2014/4/9/AT&T-ThreatTraq-Spotlight- Heartbleed

© 2014 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 10 Thank You Adam Jones - CISM, CGEIT, CISSP, 6 σ GB Sr. Technical Team Lead AT&T Global Infrastructure Operations Office: LinkedIn: Rebecca Finnin - CIPP, CISSP, CISA, CPA Director AT&T Chief Security Office