ModSecurity Performance Recommendations

Sometimes we see ModSecurity users asking about performancein the mail-list. During this post I will talk about some important topics toimprove ModSecurity performance.

 1 – HTTP Caching and Acceleration

In a common web environment static contents (ie. Images) area substantial part of http traffic. Usually users don’t want to executeModSecurity rules against this kind of content. So the first recommendation issetup in front of ModSecurity a HTTP Cache and Acceleration solution.

We have interesting open sources solutions and one of themis Varnish. You can setup it in front of your ModSecurity and configure it tocache the static traffic. Once it is done, Varnish will start serving this kindof contents and ModSecurity will only see what is really necessary.

Another possible solution is setup rules to detect fileextensions you want to inspect and ignore the others:

SecRuleEngine On

SecAction "id:'1', phase:1, t:none,setvar:'tx.inspect_extensions=.html/ .php/', nolog, pass"

SecRule REQUEST_BASENAME "\.(.*)$""chain,capture,allow,setvar:tx.exts=.%(tx.1}/,phase:1, t:none,t:urlDecodeUni, t:lowercase,id:2,logdata:'%{TX.0}'"

SecRule TX:EXTS “!@within %{tx.inspect_extensions}”

However this is not the best solution because even you willskip all other rules, ModSecurity will spend time buffering, forwarding andexecuting a few rules against this kind of data.

2 – Rule Selection

Rule selection is another important topic to talk about ifyou are using the OWASP Core Rule Set.

We have many categories of rules inside CRS, you shouldreview them and decide if all categories and rules are important for you.

We recommend load all rules, however sometimes it is notpossible for performance point of view. So you should do a risk analysis andload what is primordial.

3 – Rule Execution Mode

The rules from OWASP Core Rule Set Project can be executedin two different modes:

Self-Contained Mode - Rules inherit the "deny"disruptive action.  The first rulethat matches will block.

Collaborative Detection Mode - This is a "delayedblocking" mode of operation where each matching rule will inherit the"pass" action and will only contribute to anomaly scores.

From the performance point of view the Self-Contained Modeis the best solution since it should execute lesser rules than theCollaborative Mode, reducing the overhead caused by rule engine and loggingengine. However from the False Positive point of view, Collaborative Modeshould emit lesser False Positives.

That being said you should try the default mode first(Self-Contained) and decide if it is enough for you. If doesn't, I wouldrecommend work with ModSecurity rule exceptions features first before decide tomove to Collaborative Detection Mode.

It is important to mention that you should see results fromperformance point of view only if SecRuleEngine On.

4 – Rule Pre-Filtering

If you are planning to write your own rules, specially using@rx operator with nontrivial regex to inspect large amount of data likeresponse bodies, you should consider use @pm operator as a pre-filter rule:

SecRule RESPONSE_BODY “@pm some_leak_patterns” “phase:4,chain,id:12345,deny”SecRule RESPONSE_BODY “@rxyour_nontrivial_regex_some_leak_patterns”

The @pm operator uses a fast multi-pattern match algorithmcalled Aho-Corasick and can be used to avoid execution of your regex againstall inbound and outbound buffers.

Another pre-filter ideas are:

• Immediately reject IPs from countries
• Immediately reject IPs with bad reputation
• Immediately reject transactions with not allowed number ofarguments
• Immediately reject transactions with not allowed argumentslength.

With this idea in mind you can build a small set of rulesthat will run before the CRS and will immediately reject transactions to avoidthem being inspected against all CRS rules.

5 – Buffering

ModSecurity works buffering inbound and outbound data to belater inspected by rules. The main bottleneck related to this topic isbuffering response bodies for two reasons:  it will consume a lot of RAM and usually rules placed inresponse body phase are expensive.

That being said you can consider disabling response bodyinspection setting SecResponseBodyAccess Off and conditionally enable it usingctl:responseBodyAccessOn during some specific situation for a certain typesSecResponseBodyMimeTypes..

For example:

SecResponseBodyMimeType text/plain text/html text/xml

SecResponseBodyAccess Off

SecRule REQUEST_BODY|ARGS “@pm union select”“phase:2,chain,id:1234,ctl:responseBodyAccess=On”

SecRule REQUEST_BODY|ARGS “@rxyour_nontrivial_regex_union_select”

SecRule RESPONSE_BODY “@pm some_leak_patterns”“phase:4,chain,id:12345,deny”

SecRule RESPONSE_BODY “@rxyour_nontrivial_regex_some_leak_patterns”

6 – Logging

The Logging Engine could be a performance killer if youdon’t keep attention on:

Execute – constantly - a tuning process in your rules toavoid too much false positives. Keep in mind that disk i/o is expensive and youdon’t want to spend resource logging false positives.

Do not use serial logging mode. It uses locks to protect thefile and will kill the performance. Use concurrent mode instead.

Review the audit log parts you are logging. Some parts likeK and E can increase the overhead caused by the logging engine, because itusually needs to write big amount of data to disk.

Do not enable debug log in production.

7 – PCRE-JIT

A Just-In-Time compiler support was inserted into pcrelibrary (>=8.20). Just-in-time compiling is a optimization that can greatlyspeed up pattern matching operations. It is of most benefit when the samepattern is going to be matched many times.

Release 8.20 21-Oct-2011 ------------------------

The main change in this release is the inclusion of ZoltanHerczeg's just-in-time compiler support, which can be accessed by building PCREwith --enable-jit. Large performance benefits can be had in many situations.8.20 also fixes an unfortunate bug that was introduced in 8.13 as well astidying up a number of infelicities and differences from Perl.

ModSecurity 2.7.x series can execute a regex using PCRE-JIT.This is a very good feature from performance point of view. To enable it youmust compile ModSecurity using the follow configure option:

./configure –enable-pcre-jit

Make sure your PCRE library was compiled with JIT support(using the option described in the above pcre release notes). Also ModSecurityand Apache must use the same library version. You can check it looking intoerror log.

As an example we sent a large input to be processed byrequest body rules and measured the spent time:

JIT Disabled Phase 2 rules = 422749  usecs

JIT Enabled Phase 2 rules = 115777 usecs

Looking this example pcre-jit can make rules 75% faster inaverage.

8 – Caching Lua VM

This is for people that need to execute multiple Lua scriptsin the same transaction. Normally ModSecurity will create and destroy a VM foreach lua script running in the same transaction. You can change this behaviorrecompiling ModSecurity with the option:

./configure –enable-lua-cache

Once recompiled it, ModSecurity will keep in memory the LuaVM during the whole transaction, reducing the overhead caused by create/destroyVM operations.

As an example let’s measure the performance of three scriptsexecuting in the same transaction:

Cache disabled

Lua: Executing script: /etc/apache2/modsecurity/script1.lua

Lua: Script completed in 742 usec, returning: 1.

Lua: Executing script: /etc/apache2/modsecurity/script2.lua

Lua: Script completed in 517 usec, returning: 1.

Lua: Executing script: /etc/apache2/modsecurity/script3.lua

Lua: Script completed in 489 usec, returning: 1.

Total: 1748usecs

Cache enabled

Lua: Executing script: /etc/apache2/modsecurity/script1.lua

Lua: Script completed in 592 usec, returning: 1.

Lua: Executing script: /etc/apache2/modsecurity/script2.lua

Lua: Script completed in 130 usec, returning: 1.

Lua: Executing script: /etc/apache2/modsecurity/script3.lua

Lua: Script completed in 101 usec, returning: 1.

Total: 823usecs

We can see an overhead reduction of  ~50% caused by VM create/destroy operations.

9 – Detecting expensive rules.

If there are more performance issues you can try the followsteps to find expensive rules and then have a chance to make it better.

As an example let’s suppose we are trying to detectexpensive rules running on phase 2 (request body). If you are using the latestModSecurity version (>= 2.7.4) you have all those features to work.

Create a rule to detect if phase 2 is the problem. We willassume 1000 microseconds too much

SecRule PERF_PHASE2 “@qt 1000” “id:1234,phase:3”

If you have a trigger it is indicating you have expensiverules. So let’s try to detect them adding the follow rules

# All rules that spent more than 50usecs will be present inaudit log part H

SecRulePerfTime 50

# PERF_RULES is a collection that will contain all rulesthat spent more than SecRulePerfTime vallue to run.

SecRule PERF_RULES “@qt 50” “id:1,phase:3”

So if you have expensive rules we will see alerts:

[Tue Apr 23 17:50:26 2013] [error] [client 192.168.0.103]ModSecurity: Warning. Operator GT matched 50 at PERF_RULES:960032. [file"/etc/apache2/modsecurity/owasp-modsecurity-crs-2.2.6/base_rules/modsecurity_crs_99_perl.conf"][line "2"] [id "1"] [hostname "192.168.0.104"][uri "/acao.php"] [unique_id "UXcCIsCoAGUAACvOLqcAAAAD"]

[Tue Apr 23 17:50:26 2013] [error] [client 192.168.0.103]ModSecurity: Warning. Operator GT matched 50 at PERF_RULES:958022. [file"/etc/apache2/modsecurity/owasp-modsecurity-crs-2.2.6/base_rules/modsecurity_crs_99_perl.conf"][line "2"] [id "1"] [hostname "192.168.0.104"][uri "/acao.php"] [unique_id "UXcCIsCoAGUAACvOLqcAAAAD"]

[Tue Apr 23 17:50:26 2013] [error] [client 192.168.0.103]ModSecurity: Warning. Operator GT matched 50 at PERF_RULES:973323. [file"/etc/apache2/modsecurity/owasp-modsecurity-crs-2.2.6/base_rules/modsecurity_crs_99_perl.conf"][line "2"] [id "1"] [hostname "192.168.0.104"][uri "/acao.php"] [unique_id "UXcCIsCoAGUAACvOLqcAAAAD"]

As we can see rules 960032, 958022 and 973323 are spendingmore than 50usecs to execute., but how much time exactly ? You will get thisinformation into audit log part H:

Rules-Performance-Info: "960032=622","958022=731", "973323=109".

Please check the Reference Manual to see all PERF_variables.

10 – Persistent Storage

ModSecurity’s persistent storage mechanism is disk based.That being said it is not fast as if we could share data in memory. So werecommend, if possible, setup and define your data directory in a RAMDISK.  Persistent storage files in ModSecuritycan be bigger than necessary because old entries will only overwritten whenthey are expired. By default the expiration time is 3600 seconds, It is usuallytoo much, you probably want data for a few minutes, so you can reduce thedefault timeout using SecCollectionTimeout directives.