OWASP ModSecurity Core Rule Set (CRS) Project

1
OWASP ModSecurity Core Rule Set (CRS) Project

• Ryan Barnett
• OWASP CRS Project Leader
• Senior Security Researcher

2
Ryan Barnett - Background

• Trustwave
• SpiderLabs Research Team
• Web application firewall research/development
• ModSecurity Community Manager
• Interface with the community on public mail-list
• Steer the internal development of ModSecurity
• Author
• Preventing Web Attacks with Apache

3
Community Projects

• Open Web Application Security Project (OWASP)
• Project Leader, ModSecurity Core Rule Set
• Project Contributor, OWASP Top 10
• Project Contributor, AppSensor
• Web Application Security Consortium (WASC)
• Project Leader, Web Hacking Incident Database
• Project Leader, Distributed Web Honeypots
• Project Contributor, Web Application Firewall
  Evaluation Criteria
• Project Contributor, Threat Classification
• The SANS Institute
• Courseware Developer/Instructor
• Project Contributor, CWE/SANS Top 25 Worst
  Programming Errors

4
Session Outline

• ModSecurity Quick Overview
• The Core Rule Set (CRS) Overview
• Basic Detection Categories
• Latest CRS Improvements
• CRS Demonstration Page
• Future Directions

5
ModSecurity Quick Overview
6
What is ModSecurity?

• It is an open source web application firewall
  (WAF) module for Apache web servers
• www.modsecurity.org
• Separate Rule and Audit Engines
• Allows full request/response HTTP logging
  capability
• Deep understanding of HTTP and HTML
• Robust Parsing (form encoding, multipart, XML)
• Event-based Rules Language
• Anti-Evasion Features (normalization functions)
• Advanced Capabilities
• Transactional and Persistent Collections
• Content Injection
• Lua API

7
ModSecuritys Rules Language Syntax
Tells ModSecurity how to process data (such _at_rx,
_at_pm or _at_gt).
Tells ModSecurity where to look (such as ARGS,
ARGS_NAMES or COOKIES).
Tells ModSecurity what to do if a rule matches
(such as deny, exec or setvar).
8
ModSecuritys Apache Request Cycle Hooks
9
OWASP ModSecurity Core Rule Set (CRS) Overview
10
Project Info
http//www.owasp.org/index.php/CategoryOWASP_ModS
ecurity_Core_Rule_Set_Project
11
What is the Core Rule Set (CRS)?

• A generic, plug-n-play set of WAF rules
• Choose your mode of operation
• Standard vs. Anomaly Scoring
• Detection Categories
• Protocol Validation
• Malicious Client Identification
• Generic Attack Signatures
• Known Vulnerabilities Signatures
• Trojan/Backdoor Access
• Outbound Data Leakage
• Anti-Virus and DoS utility scripts

12
Initial Configuration

• Once unpacked, edit the main config file
• modsecurity_crs_10_config.conf
• Customize the following items
• Mode of Detection Standard vs. Anomaly Scoring
• Anomaly Scoring Severity Levels
• Enable/Disable Blocking
• Blocking Threshold Levels
• Paranoid Mode aggressive inspection
• HTTP Policy Settings
• Choose where to log events (Apache error_log
  and/or ModSecuritys audit log)

13
Traditional Detection Mode Self Contained Rules
Concept

• IDS/IPS mode with self-contained rules
• Like HTTP itself the rules are stateless
• No intelligence is shared between rules
• If a rule triggers, it will execute a
  disruptive/logging action
• Easier for the new user to understand
• Not optimal from a rules management perspective
  (handling false positives/exceptions)
• Not optimal from a security perspective
• Not every site has the same risk tolerance
• Lower severity alerts are largely ignored

14
Anomaly Scoring Detection Mode Collaborative
Rules Concept

• Advanced inspection/detection mode
• Delayed blocking
• Rules set transactional variables (tx) to store
  temporary meta-data about the rule match
• Rules also increase anomaly scores for both the
  attack category and global score
• The anomaly score enforcement rules decide
  whether or not to deny transactions at the end of
  the inbound request phase
• modsecurity_crs_49_inbound_blocking.conf

15
Anomaly Scoring - Debug Log View
16
Inspecting Anomaly Scores
17
Conditional Rules (Weak Sigs)

• SQL Injection Example
• Aggregate indicators to determine an attack
• Strong indicators
• Keywords such as xp_cmdshell, varchar,
• Sequences such as union . select, select top
  1
• Amount script, cookie and document appear in the
  same input field
• Weak indicators meta-characters
• --, , ',
• CRS only applies weak signatures in the event a
  stronger signature has previously triggered

18
Conditional Rule Example
19
Event Logging Standard vs. Correlated Events

• Standard mode
• Rules log event data to both the Apache error_log
  and the ModSecurity Audit log
• Correlated mode
• Basic rules are considered reference events and
  do not directly log to the Apache error_log
• Correlation rules in the logging phase analyze
  inbound/outbound events and generate special
  events
• modsecurity_crs_60_correlation.conf

20
Inbound/Outbound Correlation

• Couple the inbound with the outbound for
  increased intelligence
• Was there an inbound attack?
• Was there an HTTP Status Code Error (4xx/5xx
  level)?
• Was there an application information leak?
• Correlation facilitates better incident response
• App error without inbound attack -gt Contact Ops
• Inbound attack outbound error -gt Contact
  Security

21
Event Severity Ratings

• Correlated Events
• 0 Emergency - is generated from correlation
  (inbound attack outbound leakage)
• 1 Alert - is generated from correlation (inbound
  attack outbound application level error)
• Non-Correlated Events
• 2 Critical - highest severity level possible
  without correlation. It is normally generated by
  the web attack rules (40 level files)
• 3 Error - is generated from outbound leakage
  rules (50 level files)
• 4 Warning - is generated by malicious client
  rules (35 level files)
• 5 Notice - is generated by the Protocol policy
  and anomaly files
• 6 Info - is generated by the search engine
  clients (55 marketing file)

22
Correlated Event Messages
23
Detection Mechanisms Protocol Violations

• Protocol vulnerabilities such as Response
  Splitting, Request Smuggling, Premature URL
  ending
• Content length only for non GET/HEAD methods
• Non ASCII characters or encoding in headers
• Valid use of headers (for example, content length
  is numerical)
• Proxy Access
• modsecurity_crs_20_protocol_violations.conf
• Attack requests are different due to automation
• Missing headers such as Host, Accept, User-Agent
• Host is an IP address (common worm propagation
  method)
• modsecurity_crs_21_protocol_anomalies.conf

24
Detection Mechanisms Protocol Policies

• Policy is usually application specific
• Some restrictions can usually be applied
  generically
• White lists can be build for specific
  environments
• Limitations on Sizes
• Request size, Upload size
• of parameters, length of parameter
• modsecurity_crs_23_request_limits.conf
• Items that can be allowed or restricted
• Methods - Allow or restrict WebDAV, block abused
  methods such as CONNECT, TRACE or DEBUG
• File extensions backup files, database files,
  ini files
• Content-Types (and to some extent other headers)
• Modsecurity_crs_30_http_policy.conf

25
Detection Mechanisms Malicious Clients

• Not aimed against targeted attacks, but against
  general malicious internet activity
• Offloads a lot of cyberspace junk noise
• Effective against comment spam
• Reduce event count
• Detection of Malicious Robots
• Unique request attributes User-Agent header,
  URL, Headers
• Black list of IP addresses
• Rate based detection
• Detection of security scanners
• Blocking can confuse security testing software
  (WAFW00f)
• modsecurity_crs_35_bad_robots.conf

26
Detection Mechanisms App Layer Attacks

• Detect application level attacks such as those
  described in the OWASP top 10
• SQL injection and blind SQL injection
• Cross site scripting (XSS)
• OS command injection and remote command access
• Remote file inclusion
• modsecurity_crs_40_generic_attacks.conf
• modsecurity_crs_41_sql_injection_attacks.conf
• modsecurity_crs_41_xss_attacks.conf

27
Known Vulnerability Signatures

• SpiderLabs received authorization from ET to
  convert their Snort rules and include them in the
  CRS
• http//www.emergingthreats.net/
• Converted the following rule files
• emerging-web_server.rules
• emerging-web_specific_apps.rules
• Identifying attacks against known vulnerabilities
  does have value
• Raised threat level
• If done correctly, lessens false positives
• CRS combines the what of our generic attack
  payload detection with the where of ET known vuln
  data

28
Example Emerging Threats Rule
Attack vector location URI Parameter
alert tcp EXTERNAL_NET any -gt HTTP_SERVERS
HTTP_PORTS (msg"ET WEB_SPECIFIC_APPS 20/20 Auto
Gallery SQL Injection Attempt --
vehiclelistings.asp vehicleID SELECT"
flowestablished,to_server uricontent"/vehicleli
stings.asp?" nocase uricontent"vehicleID"
nocase uricontent"SELECT" nocase
pcre"/.SELECT.FROM/Ui" classtypeweb-applicati
on-attack referencecve,CVE-2006-6092
referenceurl,www.securityfocus.com/bid/21154
referenceurl,doc.emergingthreats.net/2007504
referenceurl,www.emergingthreats.net/cgi-bin/cvsw
eb.cgi/sigs/WEB_SPECIFIC_APPS/WEB_2020_Auto_galler
y sid2007504 rev5)
PCRE Weak signature
29
Converted Emerging Threats Rule
Verify the URI of the request
(sid 2007508) ET WEB_SPECIFIC 20/20 Auto
Gallery SQL Injection Attempt --
vehiclelistings.asp vehicleID SecRule
REQUEST_URI_RAW "(?i\/vehiclelistings\.asp)"
"chain,phase2,block,tnone,turlDecodeUni,thtmlE
ntityDecode,tnormalisePathWin,capture,ctlauditLo
gPartsE,nolog,auditlog,logdata'TX.0',idsid2
007508,rev3,msg'ET WEB_SPECIFIC 20/20 Auto
Gallery SQL Injection Attempt --
vehiclelistings.asp vehicleID ',tagweb-applicati
on-attack',tag'url,www.emergingthreats.net/cgi-bi
n/cvsweb.cgi/sigs/WEB_SQL_INJECTION/WEB_2020_Auto_
gallery'" SecRule TX'/SQL_INJECTION.ARGSvehic
leID/' "_at_gt 0" "setvar'tx.msgET WEB_SPECIFIC
20/20 Auto Gallery SQL Injection Attempt --
vehiclelistings.asp vehicleID ',setvartx.sqli_sco
re1,setvartx.anomaly_score20,setvartx.rule
.id-SQL_INJECTION/SQL_INJECTION-matched_var_nam
ematched_var"
Verify the attack vector location from saved TX
SQL Injection data exists
30
Detection Mechanisms Trojans/Backdoors

• Major problem in hosting environments
• Uploading is allowed
• Some sites may be secure while others not
• Upload detection
• Check uploading of files containing viruses (i.e.
  WORD docs)
• util/modsec-clamscan.pl
• Check uploading of http backdoor page
• Access detection
• Known signatures (x_key header)
• Generic file management output (gid, uid, drwx,
  c\)
• modsecurity_crs_45_trojans.conf

31
Detection Mechanisms Information Leakage

• Monitoring outbound application data
• HTTP Error Response Status Codes
• SQL Information Leakage
• Stack Dumps
• Source Code Leakage
• Last line of defense if all else fails
• Provide feedback to application developers
• Important for customer experience
• Makes life for the hacker harder (if blocking is
  used)
• modsecurity_crs_50_outbound.conf

32
Latest CRS Improvements v2.0.9
33
Latest Improvements

• Lua port of PHPIDS Converter.php code
• Advanced normalization functions
• More accurate use of PHPIDS Filters
• Centrifuge Generic Attack Payload Detection
• Experimental Rules
• Generic attack payload detection
• CRS Demonstration Page
• Request Header Tagging

34
Lua port of PHPIDS

• http//phpids.net/
• 70 regular expression rules to detect common
  attack payloads
• XSS
• SQL Injection
• RFI
• Filters are heavily tested by the community and
  updated frequently
• Trustwave SpiderLabs worked with PHPIDS lead to
  port code to Lua for use in ModSecuritys API
• https//svn.php-ids.org/svn/trunk/lib/IDS/Converte
  r.php
• https//svn.php-ids.org/svn/trunk/lib/IDS/default_
  filter.xml
• Thanks to Mario Heiderich

35
Lua port of PHPIDS

• Example normalization functions
• -- Make sure the value to normalize and monitor
  doesn't contain Regex DoS
• -- Check for comments and erases them if
  available
• -- Strip newlines
• -- Checks for common charcode pattern and
  decodes them
• -- Eliminate JS regex modifiers
• -- Converts from hex/dec entities
• -- Normalize Quotes
• -- Converts SQLHEX to plain text
• -- Converts basic SQL keywords and obfuscations
  
• -- Detects nullbytes and controls chars via
  ord()
• -- This method matches and translates base64
  strings and fragments
• -- Strip XML patterns
• -- This method converts JS unicode code points
  to regular characters
• -- Converts relevant UTF-7 tags to UTF-8
• -- Converts basic concatenations
• -- This method collects and decodes proprietary
  encoding types

36
PHPIDS Example Filter
ltfiltergt ltidgt1lt/idgt ltrulegtlt!CDATA(?"
"-?gt)(?\w\s\s\/gt)(?gt")gtlt/rulegt
ltdescriptiongtfinds html breaking injections
including whitespace attackslt/descriptiongt
lttagsgt lttaggtxsslt/taggt
lttaggtcsrflt/taggt lt/tagsgt
ltimpactgt4lt/impactgt lt/filtergt
37
Converted PHPIDS Example Filters
SecRule TX'/(QUERY_REQUEST_ARGS)._normalized
/' "(?\lt\w?\s(?\gt)t(?!rong))(?\ltscri)(lt
\w\w)" "phase2,capture,tnone,pass,skip1,nolo
g,auditlog,msg'Detects obfuscated script tags
and XML wrapped HTML',id'9000033',tag'WEB_ATTACK
/XSS',logdata'TX.0',severity'2',setvar'tx.ms
grule.id-rule.msg',setvartx.anomaly_score
4,setvar'tx.tx.msg-WEB_ATTACK/XSS-matched_v
ar_nametx.0'" SecRule TXPARANOID_MODE "_at_eq
1" "chain,phase2,tnone,logdata'TX.0',severit
y'2',pass,nolog,auditlog,msg'Detects obfuscated
script tags and XML wrapped HTML',id'9000033',tag
'WEB_ATTACK/XSS'" SecRule
ARGSREQUEST_BODYREQUEST_URI_RAW
"(?\lt\w?\s(?\gt)t(?!rong))(?\ltscri)(lt\w
\w)" "capture,multiMatch,tnone,turlDecodeUni,t
cssDecode,tjsDecode,thtmlEntityDecode,treplace
Comments,tcompressWhiteSpace,tlowercase,setvar'
tx.msgrule.id-rule.msg',setvartx.anomaly_s
core4,setvar'tx.tx.msg-WEB_ATTACK/XSS-matc
hed_var_nametx.0'"
38
Centrifuge Code

• Negative security approach to combating XSS and
  SQL Injection is doomed to fail
• Unlimited ways to write functionally equivalent
  code
• Obfuscation methods, however often have certain
  characteristics
• PHPIDS has an interesting approach to identify
  attack payloads through heuristics called
  Centrifuge
• Analysis of the use of special characters
• Ratio between the count of the word characters,
  spaces, punctuation and the non word characters
• If lt3.49 malicious
• Normalization and stripping of any word character
  and spaces including line breaks, tabs and
  carriage returns
• Regex check in default_filters.xml catches
  results

39
PHPIDS Lua Port Demo
40
Experimental Generic Detection Rules

• Two new experimental/beta generic detection rules
• Optional_rules/modsecurity_crs_40_experimental.con
  f
• Restricted Character Anomaly Usage
• Analyzes the number and type of meta-characters
  present in a payload
• Testing thus far shows that the detection is good
  except for free-form text fields.
• Need to adjust the anomaly scoring for you own
  site
• Repetitive use of non-Word characters
• Currently alerts if there are 4 or more special
  characters found in a row

41
Live CRS Demonstration Page
42
Live CRS Demonstration Page
http//www.modsecurity.org/demo/
43
CRS Demonstration Page

• Request will go through CRS page first and then
  we proxy the request to the PHPIDS page
• http//demo.php-ids.org/
• We then inspect the inbound with the outbound and
  provide results
• CRS detected an attack
• CRS did not find anything malicious but PHPIDS
  did
• Neither CRS nor PHPIDS found anything malicious
• A link is provided to report false negatives to
  our JIRA ticketing system
• https//www.modsecurity.org/tracker/browse/CORERUL
  ES
• We have received gt6700 attacks thus far

44
CRS Demonstration Page
45
Demonstration page Demo
46
Request Header Tagging
47
CRS Demonstration Page

• When in a distributed architecture, you can share
  WAF data with downstream hosts
• Similar to SMTP SPAM data added to mime-headers
• optional_rules/modsecurity_crs_49_header_tagging.c
  onf
• Maps to AppSensor Detection Point RP2
  (Suspicious External User Behavior)

48
Example Header Tagging
GET /path/to/foo.php?test12720or2027227272
27-- HTTP/1.1 Host www.example.com
User-Agent Mozilla/5.0 (X11 U Linux i686
en-US rv1.9.1.5) Gecko/20091109 Ubuntu/9.10
(karmic) Firefox/3.5.5 Accept text/html,applicati
on/xhtmlxml,application/xmlq0.9,/q0.8 Accep
t-Language en-us,enq0.5 Accept-Encoding
gzip,deflate Accept-Charset ISO-8859-1,utf-8q0.
7,q0.7 X-WAF-Events TX / 999935-Detects
common comment types-WEB_ATTACK/INJECTION-ARGStes
t, TX999923-Detects JavaScript location/document
property access and window access
obfuscation-WEB_ATTACK/INJECTION-REQUEST_URI_RAW,
TX950001- WEB_ATTACK/SQL_INJECTION-ARGStest X-WA
F-Score Total48 sqli2 xss Connection
Keep-Alive
49
Future Directions
50
Future Directions

• Preventing XSS with Content Injection
• Javascript Sandboxing with Active Content
  Signatures
• http//blog.modsecurity.org/2010/09/advanced-topic
  -of-the-week-xss-defense-via-content-injection.htm
  l
• http//www.modsecurity.org/demo/demo-deny-noescape
  .html
• Implementing OWASP AppSensor Detection Points
• We currently have some mappings for existing
  events
• Will be using Lua to implement other
  aggregate/behavioral/trend detection Points

51
Call for Community Help

• We have made great strides with CRS v2.0 but
  there is still much work to be done
• Test out the CRS demo page and report any issues
  found either to the mail-list or to JIRA
• Need Rule Documentation help
• Please sign up on our project mail-list if you
  want to help
• https//lists.owasp.org/mailman/listinfo/owasp-mod
  security-core-rule-set

52
Questions?

• Email Ryan.Barnett_at_owasp.org
• Twitter - _at_ryancbarnett / _at_modsecurity