The OWASP Foundation AppSec DC Copyright © The OWASP Foundation Permission is granted to copy, distribute and/or modify this document under the terms of the OWASP License. ModSecurity Core Rule Set (CRS) v2.0 Ryan Barnett OWASP Project Leader Director of Application Security Research, Breach Security

About The Speaker  Community Participation:  OWASP ModSecurity Core Rule Set Project Leader  Web Application Security Consortium (WASC) Board Member  WASC Distributed Open Proxy Honeypot Project Leader  Day Job:  Director of Application Security Research, Breach Security  In charge of security research through Breach Security Labs, development of ModSecurity rules and signatures.

What are we going to talk about  ModSecurity Quick Overview  The Core Rule Set (CRS) Overview  Basic Detection Categories  CRS v2.0 Improvements  Facilitating Community Collaboration  Call for Community Help 3

What is ModSecurity?  It is an open source web application firewall (WAF) module for Apache web servers   Separate Rule and Audit Engines  Allows full request/response HTTP logging capability  Deep understanding of HTTP and HTML  Robust Parsing (form encoding, multipart and XML)  Anti Evasion Features (normalization functions)  Supports Complex Rules Language  Advanced Capabilities  Transactional and Persistent Collections  Content Injection  Lua API

ModSecurity’s Apache Request Cycle Hooks

ModSecurity’s Rules Language

ModSecurity’s Rules Language Syntax Tells ModSecurity where to look (such as ARGS, ARGS_NAMES or COOKIES). Tells ModSecurity how to process Tells ModSecurity what to do if a rule matches (such as deny, exec or setvar).

The OWASP Foundation, The ModSecurity Core Rule Set (CRS) Ryan Barnett AppSec DC CRS v2.0 Overview

What is the Core Rule Set (CRS)?  A generic, plug-n-play set of WAF rules  Detection Mechanisms:  Protocol Validation  Malicious Client Identification  Generic Attack Signatures  Known Vulnerabilities Signatures  Trojan/Backdoor Access  Outbound Data Leakage  Anti-Virus and DoS utility scripts  OWASP Project Homepage  urity_Core_Rule_Set_Project

Who uses the CRS?  WASC Distributed Open Proxy Honeypot Project  “Use one of the web attacker's most trusted tools against him - the Open Proxy server. Instead of being the target of the attacks, we opt to be used as a conduit of the attack data in order to gather our intelligence”

Who uses the CRS?  Akamai WAF-in-the-Cloud Service  Converted CRS in Akamai Edge Servers  Launched in November 2009 Origin Server Akamai EdgePlatform with WAF Attacker

The OWASP Foundation, The ModSecurity Core Rule Set (CRS) Ryan Barnett AppSec DC Example Detection Categories

Detection Mechanisms: Protocol Violations  Protocol vulnerabilities such as Response Splitting, Request Smuggling, Premature URL ending  Content length only for non GET/HEAD methods  Non ASCII characters or encoding in headers  Valid use of headers (for example, content length is numerical)  Proxy Access  modsecurity_crs_20_protocol_violations.conf  Attack requests are different due to automation  Missing headers such as Host, Accept, User-Agent  Host is an IP address (common worm propagation method)  modsecurity_crs_21_protocol_anomalies.conf

HTTP Request Smuggling Example  Goal: IDS/IPS will only see one POST request to /foobar.html POST HTTP/ Content-Length: 0 Content-Length: 44 GET /cgi-bin/foo.php?cmd=`id` HTTP/1.1 Host: SITE IDS/IPS: 1. /foobar.html Server: 1./foobar.html 2./foo.php

CRS ID – Request Smuggling Attack POST /SITE/foobar.html HTTP/1.1 Host: User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: ) Gecko/ Firefox/3.0.3 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us Accept-Encoding: gzip,deflate Accept-Charset: ISO ,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Proxy-Connection: keep-alive Content-Length: 0, 44 # HTTP Request Smuggling SecRule REQUEST_HEADERS:'/(Content-Length|Transfer-Encoding)/' "," "phase:2,t:none,block,nolog,auditlog,status:400,msg:'HTTP Request Smuggling Attack.',id:'950012',tag:'WEB_ATTACK/REQUEST_SMUGGLING', severity:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+ 20,setvar:tx.web_attack_score=+1,setvar:tx.%{rule.id}- WEB_ATTACK/REQUEST_SMUGGLING-%{matched_var_name}=%{matched_var}" Apache collapses duplicate Request headers and separates the payloads with commas – this payload means there were two Content-Length headers Request Smuggling rule looks for a comma in the appropriate Request header payloads

Detection Mechanisms: Protocol Policies  Policy is usually application specific  Some restrictions can usually be applied generically  White lists can be build for specific environments  Limitations on Sizes  Request size, Upload size  # of parameters, length of parameter  modsecurity_crs_23_request_limits.conf  Items that can be allowed or restricted  Methods - Allow or restrict WebDAV, block abused methods such as CONNECT, TRACE or DEBUG  File extensions – backup files, database files, ini files  Content-Types (and to some extent other headers)  Modsecurity_crs_30_http_policy.conf

CRS ID – Request Method Not Allowed SecRule REQUEST_METHOD "!^((?:(?:POS|GE)T|OPTIONS|HEAD))$" "phase:2,t:none,block,nolog,auditlog, status:501,msg:'Method is not allowed by policy', severity:'2',id:'960032',tag:'POLICY/ METHOD_NOT_ALLOWED',setvar:tx.anomaly _score=+5,setvar:tx.policy_score=+1,s etvar:tx.%{rule.id}- POLICY/METHOD_NOT_ALLOWED- %{matched_var_name}=%{matched_var}" PUT /tr.htm HTTP/1.0 Accept-Language: pt-br, en-us;q=0.5 Translate: f Content-Length: 67 Date: Thu, 5 Nov :26:22 GMT Connection: Keep -Alive User-Agent: Microsoft Data Access Internet Publishing Provider DAV 1.1 Host: Command Tribulation was here Jesus Loves you

Detection Mechanisms: Malicious Clients  Not aimed against targeted attacks, but against general malicious internet activity  Offloads a lot of cyberspace junk & noise  Effective against comment spam  Reduce event count  Detection of Malicious Robots  Unique request attributes: User-Agent header, URL, Headers  Black list of IP addresses  Rate based detection  Detection of security scanners  Blocking can confuse security testing software (WAFW00f)  modsecurity_crs_35_bad_robots.conf

Comment SPAM – RBL Lookups SecRule &IP:SPAMMER 0" "chain,phase:1,t:none,block,nolog,auditlog,msg:' RBL Match for SPAM Source',tag:'AUTOMATION/MALICIOUS',severity:'2', skipAfter:END_RBL_CHECK" SecRule REMOTE_ADDR sbl-xbl.spamhaus.org" "t:none,setvar:'tx.msg=%{rule.msg}',setvar:tx.au tomation_score=+1,setvar:tx.anomaly_score=+20,se tvar:'tx.%{rule.id}=%{matched_var_name}=%{matche d_var}',setvar:ip.spammer=1,expirevar:ip.spammer =86400“

Detection Mechanisms: App Layer Attacks  Detect application level attacks such as those described in the OWASP top 10  SQL injection and blind SQL injection  Cross site scripting (XSS)  OS command injection and remote command access  Remote file inclusion  modsecurity_crs_40_generic_attacks.conf  modsecurity_crs_41_sql_injection_attacks.conf  modsecurity_crs_41_xss_attacks.conf

Remote File Inclusion (RFI) Example GET /XXXXXXXX.php?ADODB_DIR= HTTP/1.1 TE: deflate,gzip;q=0.3 Connection: TE, close Host: XXXXXXXXXXX User-Agent: libwww-perl/5.805 switch(substr($mcmd[0],1)) { case "restart": case "mail": //mail to from subject message case "dns": case "info": case "cmd": case "rndnick": case "php": case "exec": break; case "pscan": //.pscan case "ud.server": //.udserver case "download": case "die": case "udpflood": case "udpflood1": case "tcpflood": case "massmail": Attack Methods Control Methods SecRule ARGS "^(?:ht|f)tps?:\/\/([\d\.]+)" \ SecRule ARGS "(?:\binclude\s*\([^)]*(ht|f)tps?:\/\/)" \ SecRule ARGS "(?:ft|htt)ps?.*\?+$" \ SecRule ARGS "^(?:ht|f)tps?://(.*)\?$" \ "chain, SecRule TX:1 %{request_headers.host}” IP address in hostname Known vulnerable parameter One or more question marks at the end Domain mis- match

Detection Mechanisms: Trojans/Backdoors  Major problem in hosting environments  Uploading is allowed  Some sites may be secure while others not  Upload detection  Check uploading of files containing viruses (i.e. WORD docs)  util/modsec-clamscan.pl  Check uploading of http backdoor page  Access detection  Known signatures (x_key header)  Generic file management output (gid, uid, drwx, c:\)  modsecurity_crs_45_trojans.conf

CRS ID – Trojan File Access SecRule RESPONSE_BODY "(?: [^ |drwxr))" \ "phase:4,t:none,ctl:auditLogParts=+E,block,nolog,auditlog,s tatus:404,msg:'Backdoor access',id:'950922',tag:'MALICIOUS_SOFTWARE/TROJAN',severit y:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.trojan_score=+1,setvar:tx.anomaly_score=+20,setvar:tx.%{rule.id}- MALICIOUS_SOFTWARE/TROJAN- %{matched_var_name}=%{matched_var}"

Detection Mechanisms: Information Leakage  Monitoring outbound application data  HTTP Error Response Status Codes  SQL Information Leakage  Stack Dumps  Source Code Leakage  Last line of defense if all else fails  Provide feedback to application developers  Important for customer experience  Makes life for the hacker harder (if blocking is used)  modsecurity_crs_50_outbound.conf

CRS ID – SQL Information Leakage SecRule RESPONSE_BODY "\bYou have an error in your SQL syntax near \'" \ "phase:4,t:none,ctl:auditLogParts=+E,block,nolog,auditlog,s tatus:500,msg:'SQL Information Leakage',id:'971094',tag:'LEAKAGE/ERRORS',severity:'3',setv ar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+15,setvar: tx.%{rule.id}-LEAKAGE/ERRORS- %{matched_var_name}=%{matched_var}"

The OWASP Foundation, The ModSecurity Core Rule Set (CRS) Ryan Barnett AppSec DC CRS v2.0 Improvements

CRS V2.0 Improvements  Rules and Alert Management  Collaborative Rules/Anomaly Scoring  Conditional Rules (Weak Signatures)  Inbound+Outbound Correlation  Updated Severity Ratings  Increased Security Coverage  XSS Improvements  Converted Emerging Threats Web Attack Signatures  Converted PHPIDS Filters  Facilitate Community Collaboration  CRS Smoketest/Demo Page  JIRA Bug Tracking

The OWASP Foundation, The ModSecurity Core Rule Set (CRS) Ryan Barnett AppSec DC Rules and Alert Management

CRS <2.0 – Self Contained Rules Concept  Older (<2.0) CRS used individual, “self-contained” actions in rules  If a rule triggered, it would either deny or pass and log  No intelligence was shared between rules  Not optimal from a rules management perspective (handling false positives/exceptions)  Editing the regex could blow it up  Heavily customized rules were less likely to be updated by the user  Not optimal from a security perspective  Not every site had the same risk tolerance  Lower severity alerts were largely ignored

CRS Collaborative Rules/Anomaly Scoring  Rules logic has changed by decoupling the inspection/detection from the blocking functionality  Rules set transactional variables (tx) to store meta-data about the rule match  Rules also increase anomaly scores for both the attack category and global score  These rules are considered basic or reference events  They do not generate an event in the Apache error_log on their own by default  The anomaly score check/enforcement rules will decided whether or not to deny/log events  modsecurity_crs_49_enforcement.conf

CRS Collaborative Rules/Anomaly Scoring # # HTTP Parameter Pollution # SecRule ARGS_NAMES ".*" \ "chain,phase:2,t:none,nolog,auditlog,pass,capture,id:'950012'setvar:'tx.a rg_name_%{tx.0}=+1',msg:'Possible HTTP Parameter Pollution Attack: Multiple Parameters with the same Name.'" SecRule TX:/ARG_NAME_*/ 1" "t:none,setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+20,setvar:tx.web_attack_score=+1,setvar:tx.%{rule.id}-WEB_ATTACK/COMMAND_INJECTION- %{matched_var_name}=%{matched_var}"  Example HTTP Parameter Pollution (HPP) attack  /index.aspx?page=select 1&page=2,3 from table where id=1

CRS 2.0 – Debug Log View

CRS 2.0 – Inspecting Anomaly Scores # Alert on SQL Injection anomalies # #SecRule TX:SQLI_SCORE 0" \ # "phase:2,t:none,log,deny,msg:'SQL Injection Detected (score %{TX.SQLI_SCORE}): %{tx.msg}'" # Alert and Deny on High Anomaly Scores # SecRule TX:ANOMALY_SCORE 20" \ "phase:2,t:none,nolog,auditlog,deny,msg:'Anomaly Score Exceeded (score %{TX.ANOMALY_SCORE}): %{tx.msg}',setvar:tx.inbound_tx_msg=%{tx.msg}"

CRS 2.0 – Conditional Rules (Weak Sigs)  SQL Injection Example  Aggregate indicators to determine an attack  Strong indicators  Keywords such as: xp_cmdshell, varchar,  Sequences such as: union …. select, select … top … 1  Amount: script, cookie and document appear in the same input field  Weak indicators – meta-characters  --, ;, ', …  CRS only applies weak signatures in the event a stronger signature has previously triggered

CRS 2.0 – Conditional Rule Example

CRS 2.0 – Inbound/Outbound Correlation  Concept is to do post processing of the transactional data (in the logging phase) for event creation  modsecurity_crs_60_correlation.conf  Couple the inbound with the outbound for increased intelligence  Was there an inbound attack?  Was there an HTTP Status Code Error (4xx/5xx level)?  Was there an application information leak?  Correlation facilitates better incident response  App error without inbound attack -> Contact Ops  Inbound attack + outbound error -> Contact Security

CRS 2.0 – Updated Severity Ratings  Correlated Events  0: Emergency - is generated from correlation (inbound attack + outbound leakage)  1: Alert - is generated from correlation (inbound attack + outbound application level error)  Non-Correlated Events  2: Critical - highest severity level possible without correlation. It is normally generated by the web attack rules (40 level files)  3: Error - is generated from outbound leakage rules (50 level files)  4: Warning - is generated by malicious client rules (35 level files)  5: Notice - is generated by the Protocol policy and anomaly files  6: Info - is generated by the search engine clients (55 marketing file)

CRS 2.0 – Correlated Event Messages

The OWASP Foundation, The ModSecurity Core Rule Set (CRS) Ryan Barnett AppSec DC Increased Security Coverage

CRS 2.0 – Updated XSS Coverage  Rules added that look for all event handlers from the WASC Script Mapping Project 

CRS 2.0 – Converted Emerging Threats Rules  Breach Security Labs received authorization from ET to convert their Snort rules and include them in the CRS   Converted the following rule files  emerging-web_server.rules  emerging-web_specific_apps.rules  Identifying attacks against known vulnerabilities does have value  Raised threat level  If done correctly, lessens false positives  CRS combines the what of our generic attack payload detection with the where of ET known vuln data

CRS 2.0 – Converted Emerging Threats Rules alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID SELECT"; flow:established,to_server; uricontent:"/vehiclelistings.asp?"; nocase; uricontent:"vehicleID="; nocase; uricontent:"SELECT"; nocase; pcre:"/.+SELECT.+FROM/Ui"; classtype:web- application-attack; reference:cve,CVE ; reference:url, reference:url,doc.emergingthreats.net/ ; reference:url, bin/cvsweb.cgi/sigs/WEB_SPECIFIC_APPS/WEB_2020_Auto_ gallery; sid: ; rev:5;) Attack vector location – URI + Parameter PCRE – Weak signature

CRS 2.0 – Converted Emerging Threats Rule # (sid ) ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID SecRule REQUEST_URI_RAW "(?i:\/vehiclelistings\.asp)" "chain,phase:2,block,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:norma lisePathWin,capture,ctl:auditLogParts=+E,nolog,auditlog,logdata:'%{TX.0}',id:sid ,rev:3,msg:'ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID ',tag:‘web- application-attack',tag:'url, bin/cvsweb.cgi/sigs/WEB_SQL_INJECTION/WEB_2020_Auto_gallery'" SecRule &TX:'/SQL_INJECTION.*ARGS:vehicleID/' 0" "setvar:'tx.msg=ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID ',setvar:tx.sqli_score=+1,setvar:tx.anomaly_score=+20,setvar:tx.%{rul e.id}-SQL_INJECTION/SQL_INJECTION-%{matched_var_name}=%{matched_var}" Verify the URI of the request Verify the attack vector location from saved TX SQL Injection data exists

CRS 2.0 – Converted PHPIDS Filters   ~70 regular expression rules to detect common attack payloads  XSS  SQL Injection  RFI  Filters are heavily tested by the community and updated frequently  Breach Security Labs received authorization from PHPIDS to convert their default_filters.xml rules and include them in the CRS   Thanks to Mario Heiderich

CRS 2.0 – PHPIDS Example Filter 1 )|(?:[^\w\s]\s*\/>)|(?:>")]]> finds html breaking injections including whitespace attacks xss csrf 4

CRS 2.0 – Converted PHPIDS Example Filter SecRule ARGS|ARGS_NAMES "(?:\"[^\"]*[^- ]?>)|(?:[^\w\s]\s*\/>)|(?:>\")" "phase:2,capture,multiMatch,t:none,t:urlDecodeUni,t:cssDecod e,t:jsDecode,t:htmlEntityDecode,t:replaceComments,t:compress WhiteSpace,t:lowercase,ctl:auditLogParts=+E,block,nolog,msg: 'finds html breaking injections including whitespace attacks',id:'phpids- 1',tag:'WEB_ATTACK/XSS',tag:'WEB_ATTACK/CSRF',logdata:'%{TX. 0}',severity:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.anoma ly_score=+4,setvar:tx.%{rule.id}-WEB_ATTACK/INJECTION- %{rule.severity}-%{rule.msg}- %{matched_var_name}=%{matched_var}" Normalization functions  Combats common evasions with multiMatch action  Normal process is to only apply the operator once after the transformation function chain  With multiMath, the operator is applied before/after any transformation function that changes data

CRS 2.0 – PHPIDS Conversion/Normalization  PHPIDS combats evasions by both converting and normalizing input data before applying their regular expressions   Handles evasion issues such as:  Comments  Newlines  Charcode  Normalize Quotes  Current CRS approach is to create rules to increase the anomaly score when these are encountered vs. attempting to normalize

CRS 2.0 – PHPIDS Centrifuge  Negative security approach to combating XSS and SQL Injection is doomed to fail…  Unlimited ways to write functionally equivalent code  Obfuscation methods, however often have certain characteristics  PHPIDS has an interesting approach to identify attack payloads through heuristics  Analysis of the use of special characters  Ratio between the count of the word characters, spaces, punctuation and the non word characters  If <3.50 = malicious  Normalization and stripping of any word character and spaces including line breaks, tabs and carriage returns  Regex check in default_filters.xml catches results

The OWASP Foundation, The ModSecurity Core Rule Set (CRS) Ryan Barnett AppSec DC Facilitate Community Collaboration

CRS 2.0 – CRS Demo/Smoketest  ModSecurity/CRS finally has its own Demo/Smoketest page 

CRS 2.0 – CRS/PHPIDS Demo/Smoketest  CRS demo page is actually a front-end for the PHPIDS smoketest page   Request will go through CRS page first and then we proxy the request to the PHPIDS page  We then inspect the inbound with the outbound and provide results  CRS detected an attack  CRS did not find anything malicious but PHPIDS did  Neither CRS nor PHPIDS found anything malicious  A link is provided to report false negatives to our JIRA ticketing system 

CRS 2.0 – CRS Demo/Smoketest

The OWASP Foundation, The ModSecurity Core Rule Set (CRS) Ryan Barnett AppSec DC Call for Community Help

CRS 2.0 – Call for Community Help  We have made great strides with CRS v2.0 but there is still much work to be done  Current OWASP Project Status is Alpha  Need some help to move it to Beta -> Release Quality  Need Project Reviewers  Test out the CRS demo page and report any issues found either to the mail-list or to JIRA  Cool project idea  Port the PHPIDS Converter.php code into Lua for use in ModSecurity  Please sign up on our project mail-list if you want to help  rule-set

The OWASP Foundation AppSec DC Copyright © The OWASP Foundation Permission is granted to copy, distribute and/or modify this document under the terms of the OWASP License. Thank You! Ryan Barnett