Effective Patch Management: How to make the pain go away Adam Shostack

Overview Why patch? Why is patching so painful? What can make it easier? Thinking about risk management How can we get out of the rat race?

Why Patch? Vendors issue patches to correct bugs Performance/Reliability Security is a subset of reliability End users apply patches to fix problems Preventative/Reaction models

Security Patches Vendors release code All code has bugs People find bugs Sometimes they tell the vendor Vendor triages, and may release a fix Some want to install it to forestall problems

Where we are Why patch? Why is patching so painful? What makes it easier? Thinking about risk management How can we get out of the rat race?

Why is patching painful? Patch notification Inventory & Roll-out Mobile Low bandwidth

Lies and Excuses! The problems of notification, inventory and roll-out, and mobile and low- bandwidth systems are roughly solved.

State of Software Tools Inventory Deploy Analyze Maturity Deployment Unicenter Tivoli ZenWorks (etc)

The Real Problems Patches are beta software Intense pressure to roll out beta software Poor data about patches Conflict between IT & IT Security Patches which can’t roll back

Uptime vs. Security IT is rated on measured uptime Every admin knows patching can break things, require reboots Security is rated on break-ins Need to deploy patches to prevent Huge fights come from different priorities.

Where We Are Why patch? Why is patching so painful? What makes it easier? Thinking about risk management How can we get out of the rat race?

Reconciling The Views Patch risk falls with time Exploit risk grows with time Can we put numbers on them? Can we engage in a risk trade-off?

Timing the Application of Security Patches for Optimal Uptime

Timing the Application... Steve Beattie, Seth Arnold, Crispin Cowan, Perry Wagle, Chris Wright, and Adam Shostack. Presented at the USENIX 16th Systems Administration Conference (LISA 2002) lisa02.pdf (Don’t copy down the URL: Google finds my homepage, that’s bullet #7)

Where We Are Why patch? Why is patching so painful? What makes it easier? Thinking about risk management How can we get out of the rat race?

Risk Management You can do this at home! Easy math leads to useful results Cost to deploy, cost to fix problems (security or broken patch) Goal is to move away from argument and worry Consider Security Risk, Patch Risk, Business Impact

Security Issues Patch criticality Software Vendor CERT metrics (ADDED: CVSS) CNN Mitigating controls Firewalls Configurations

Patch Issues How big is the patch? How many issues does it fix? Can it be backed out? Does it require a reboot? Testing (internal, external, web & lists)

Business Issues What’s the business function of the system? Is there an impending deadline? What’s your MTTR? (Mean Time To Repair)

Making it concrete Know your cost to deploy a patch Know your cost of downtime Estimate the risk of attack

Some sample numbers 1,000 node network with manual patching by $100 techies, at 1 hour/node: $100,000 to deploy a patch So what do you do if: Attack that would cost you $1,000,000 Attack that would cost you $105,000 Attack that would cost you $25,000

The $105,000 question Expected 5% ROI on cash Didn’t specify time Alternate activities? Cost of capital/ROI?

Why patch? Why is patching so painful? What makes it easier? Thinking about risk management How can we get out of the rat race?

Better Patch Mgmt SW Research and risk data Workflow Testing support Risk Management support

More Managable Deployments Use security software (Okena, Immunix, Sana, etc) to stop classes of attack Use software to deploy and manage systems Work to increase MTBF, decrease MTTR

More Secure Software The core problem is that security is not a buying criteria Make it one Push your vendor to discuss and then improve their software processes: Design, Development, Testing, Deploy

Bug (and software) Development

How To Move? It’s actually worse than that That’s a graph for a single program You deploy lots of programs

How To Get There Better software tools Internal, external Better Deployment tools Security Operations

Where The Tools Fit

Static Checkers Work with source code Lots of different languages Results generally easier to fix They’re associated with lines of code High false positive rates Find “sins of commission” like strcat() Fast

Free Static checkers RATS ITS4 Flawfinder

Static Checkers: Slicers Compiler-like technology to see what variable could be touched where Perl’s taint mode Clever techniques to deal with pointers Can be perfect on small code (20kloc) Much research

Static Checkers: Parsers Analyze variables and typing because C doesn’t Can deal with integer issues well Slower SPLINT is a free example

Static Checkers: Compilers Compile code, and analyze on the way Code is not always compiled to your processor Target a VM that has security features MOPS Dawson Engler’s Stanford GCC -Wall is not complete

Dynamic Checkers Work on binary code Never wonder if the optimizer was too clever Find “Sins of Omission” like SQL injection Slow! (Can be hours or days)

Dynamic Tools: Fuzzers Fuzz, Spike, libwhisker Mangleme http fuzzer Feed noise to the target see if it breaks And you’re surprised this is slow?

Dynamic: Attack Simulation “Second Gen fuzzers” Attack tool libraries CORE Impact, Metasploit Require skilled driver Nikto Less powerful, easy to use

Dynamic Tools: Decompilers Turn byte code/machine code into something resembling C Useful for closed source apps you need Need to analyze the decompiled source

Dynamic: Binary Differs Not a dynamic tool as much as a static tool for machine code Best for finding why a patch happened Attack/exploit creation Vendor verification: Is this patch effective? Are they being upfront about what’s in it?

Language Selection Some languages seem to be more prone to security flaws C, PHP We may not have found the classes of flaws in Java, C# New classes keep showing up (integer underflows, etc)

Things Hard to Measure Security design goodness Attack surface nmap not enough port 25 seems to have a large surface port 137 does too.

Adding Resilience to Code How to deploy operate Buggy code more securely

Free UNIX techniques chroot/jail Unprivileged daemon accounts Painful if you need fast code on port 80 Free security enhanced OSes: OpenBSD, SELinux

Techniques Harden the system: Control Attack surface Limit effect of an attack Can entail high operational cost for questionable benefit Need to evaluate what happens

More advanced tools OS hardening tools Immunix subdomain Sana kernel enhancements Application hardening Stackguard & company (Recompile vs kernel modules)

Issues with Hardening Tools How to measure their effectiveness Configuration effort Costs (percieved and real) Cash up front Speed Supportability + Vendor finger pointing

Network Intrusion Prevention Throwing Ducks at Baloons Paper by Ptacek and Newsham, 1998 Showed how to evade IDSs, IPSs The Covert Channel problem

Firewalls Move (back) Up Application Firewalls vs packet filters Inspection Snort Inline

Process Resilience Tools How to fail gracefully detect, respond, improve Measuring your process Architecture and Forensics An ounce of prevention

Selling Your Boss Or, Security folks are from Mars, businesspeople are from Wheaton

How You Buy Software Functionality, supportability, price Can you get security in there? Probably requires being able to get lots of complexity into a 1-5 score (or somesuch) The above can be used for that

Sample Scoring 0-1 point for a good language 0-1 point for documented use of tools to check code 0-1 point for unprivileged, chroot install 0-1 point for logging 0-1 point for local analysis

Deployment Budgets Cash for wires, hubs, power, air Where does security fit? What’s the real cost of a failure? (Hint, its not $1m, unless you’re a large bank)

Deployment Business Cases Cost of operations with and without tool X Cost of special events: Patching Breakins Worms Frequency of special events

Summary We’ll always have patches to deploy We can build rational decision processes We can use better tools We can push vendors to sell better SW