JBoss security: penetration, protection and patching Ruxcon 2011 David Jorm

2 SECURITY RESPONSE TEAM | RED HAT INC. Contents JBoss background & architecture JMX & JMX Console Historical vulnerabilities JBoss worm Configuration & application weaknesses Security response & counter-measures

3 SECURITY RESPONSE TEAM | RED HAT INC. JBoss Background Open source software under a commercial subscription arrangement, same model as RHEL Core product is EAP, a J2EE app server based on JBoss AS Many derivative products: SOA-P, BRMS, Portal, Web Server, etc. JBoss was acquired by Red Hat in 2006 This talk is primarily about Red Hat's JBoss products, not community releases which have no dedicated security coverage. The issues are mostly the same, however.

4 SECURITY RESPONSE TEAM | RED HAT INC. JBoss Architecture

5 SECURITY RESPONSE TEAM | RED HAT INC. JBoss Architecture Servlets handled by JBoss Web, based on Tomcat Management provided by JMX Console and Web Console Core components in JBoss AS, which is productized as EAP Other products add components to EAP: SOA, BRMS, EPP, etc. Management consoles and JBoss Web account for a large proportion of vulnerabilities Other major components spanning products include Seam and JBossWS

6 SECURITY RESPONSE TEAM | RED HAT INC. JMX (Java Management Extensions) Framework for managing and monitoring systems via MBeans Probe, Agent and Remote Management layers Source: Wikimedia Commons

7 SECURITY RESPONSE TEAM | RED HAT INC. JMX Console Web-based JMX management interface, part of the JBoss project Allows a user to invoke methods on MBeans via a web interface Included in JBoss AS, EAP and derived products Password-based authentication by default on EAP, open by default on AS A major attack surface

8 SECURITY RESPONSE TEAM | RED HAT INC. Historical Vulnerabilities

9 SECURITY RESPONSE TEAM | RED HAT INC. CVE The JMX console on EAP and derived products includes password authentication by default. The relevant tag included: GET POST Authentication was not applied to other verbs – e.g. HEAD The HEAD handler defaulted to the same code execution path as GET

10 SECURITY RESPONSE TEAM | RED HAT INC. CVE Double.parseDouble in the JRE can get into an infinite loop when converting a number to a double For example, use e-308 Can be used to effect a DoS attack Affected Java itself, but also Tomcat/JBoss Web via HTTP headers e.g. q Fixed in Tomcat/JBoss Web by no longer using Double.parseDouble for the QoS header Separate fix in Java itself

11 SECURITY RESPONSE TEAM | RED HAT INC. CVE / CVE Seam did not properly restrict the use of Expression Language (EL) during exception handling. An attacker can cause the application to throw an exception, then provide a parameter including EL. The EL can include calls to.class. and.getClass(), which can be used to invoke arbitrary code. CVE was fixed in April 2011, but the patch was incomplete and this was found by a user. CVE included a complete patch in July Both issues handled under embargo – no wild 0day

12 SECURITY RESPONSE TEAM | RED HAT INC. CVE Remote DoS in jbossws-native (web services) An attacker can make a request to XML web services (e.g. SOAP) including recursive entity resolution with embedded DTDs The issue was specific to jbossws-native (JBoss), not jbossws-cxf (Apache) Enough concurrent attack requests and the server will consume all available connections and die

13 SECURITY RESPONSE TEAM | RED HAT INC. CVE JBoss Operations Network (JON) is a centralized management system for large JBoss environments Remote privilege escalation The JON CLI allowed an unprivileged user to perform management tasks and configuration changes with the privileges of the administrator user The permissions were not being properly checked – a logic flaw

14 SECURITY RESPONSE TEAM | RED HAT INC. CVE Spring is included to support internal applications and user-deployed spring applications. We currently use spring 2.x throughout, but SOA-P uses spring 3. Spring applications which de-serialize objects from untrusted sources are vulnerable to remote code execution An attacker can serialize a proxy rather than a class instance, and use this to invoke arbitrary code using java.lang.Runtime Fixed upstream, async patch shipped for SOA-P

15 SECURITY RESPONSE TEAM | RED HAT INC. Historical Vulnerabilities – Summary There are a wide range of flaws covering a wide range of attack surfaces The vulnerabilities affect both upstream components bundled with JBoss products and JBoss project code The JMX Console and Tomcat/JBoss Web are the source of many issues Many lower impact flaws have also been found and fixed: XSS, CSRF, information disclosure

16 SECURITY RESPONSE TEAM | RED HAT INC. JBoss Worm Exploits CVE , which was patched in April 2010 Uses HEAD verb to bypass authentication, then uses the JMX Console to call bshdeployer and deploy arbitrary code to the server Installs an IRC-based command and control component for a botnet, then runs a scanner to search random blocks of IP address space for more servers to infect Also affects unsecured JBoss AS instances

17 SECURITY RESPONSE TEAM | RED HAT INC. Configuration & Application Weaknesses Incorrect application of security constraints – e.g. CVE ! Publicly exposed management interfaces, e.g. JMX Console Default admin passwords XSS, CSRF and less so SQL injection all common on deployed apps. CSRF in particular not well protected against by the development frameworks Quickstarts and samples left deployed. These have limited security coverage

18 SECURITY RESPONSE TEAM | RED HAT INC. Security response Monitoring vulnerabilities, exploits & threats Triage Escalation and troubleshooting through lifecycle Communication with other affected vendors Internal communication, documentation, advisory Responsible for errata release Metrics and feedback to engineering Single point of contact for customers

19 SECURITY RESPONSE TEAM | RED HAT INC. Red Hat SRT process

20 SECURITY RESPONSE TEAM | RED HAT INC. Embargoed vulnerabilities (50% of total, )

21 SECURITY RESPONSE TEAM | RED HAT INC. “No notice” vulnerabilities (50% of total, )

22 SECURITY RESPONSE TEAM | RED HAT INC. Triage Determine whether it affects our products Assign a severity (CVSS2) Prioritize according to severity Assign a CVE ID This is the fun part – reproducing bugs, running exploits, feeling the giddy thrill of fresh 0day in your hand

23 SECURITY RESPONSE TEAM | RED HAT INC. File Bugs Complex bug tracking regime: Bugzilla for the whole CVE Per-product bugs for affected products. Most in Bugzilla, some in JIRA, one product now heading for EOL was even in Google Code. Task bug for monitoring SRT action

24 SECURITY RESPONSE TEAM | RED HAT INC. Patch Sometimes we produce the patch for our own products Especially true for JBoss products with fewer contributors and people sharing the code In this case we need to commit our patch back upstream (embargoed) Other times we backport it from upstream Backporting means cherry picking security fixes

25 SECURITY RESPONSE TEAM | RED HAT INC. Backporting patches Apache httpd Apache httpd NEW! httpd ent httpd ent RHSA-2005:582 httpd ent RHSA-2005:608 Enterprise Linux 4

26 SECURITY RESPONSE TEAM | RED HAT INC. QE Patch Confirm fix solves the security issues No regressions introduced No performance degradation We've had issues with all of the above. A huge cost if we have to clean up one of these impacts after the patch is released.

27 SECURITY RESPONSE TEAM | RED HAT INC. Errata Packages patch as either an RPM or zip file Bundles documentation of the issues Available via RHN or FTP Triggers alert s

28 SECURITY RESPONSE TEAM | RED HAT INC. Four months in the life...

29 SECURITY RESPONSE TEAM | RED HAT INC. Counter-measures Apply your patches! For community versions there are no async patches, the only safe bet is to track the latest stable release Best practice is to deploy servers behind a reverse proxy Don't publicly expose any management interfaces, particularly the JMX Console Use an appropriate server profile for your environment Test for application vulnerabilities before deploying your apps. JBoss won't automagically stop attacks against flaws in your apps.

30 SECURITY RESPONSE TEAM | RED HAT INC. Questions?