How to Configure ModSecurity OWASP Core Rule Set on Nginx & OpenLiteSpeed

While network firewalls like UFW or iptables filter traffic at the IP address and port layer, they are blind to application-layer HTTP payloads. An attacker can easily pass through port 443 to execute SQL injection attacks, Remote Code Execution (RCE), or cross-site scripting (XSS) against unpatched web applications and WordPress plugins.

To defend production websites against sophisticated application-layer threats, enterprise sysadmins deploy a Web Application Firewall (WAF). ModSecurity (libmodsecurity) paired with the OWASP Core Rule Set (CRS v3/v4) provides a formidable open-source defense layer. By inspecting incoming HTTP headers, POST bodies, cookies, and file uploads in real time on your Linux VPS, ModSecurity identifies and blocks malicious attack signatures before they ever touch your PHP runtime.

1. Understanding the ModSecurity OWASP CRS Engine

The OWASP Core Rule Set operates on an intelligent anomaly scoring model rather than simple binary pattern matching:

• Each incoming request is evaluated across dozens of specialized detection rules: SQL injection (SQLi), Cross-Site Scripting (XSS), Local File Inclusion (LFI), Remote File Inclusion (RFI), and PHP Code Injection.
• When a rule triggers, it assigns an anomaly score based on severity (Critical: 5, Error: 4, Warning: 3, Notice: 2).
• At the conclusion of the request inspection phase, ModSecurity sums the anomaly points. If the score meets or exceeds the configured inbound threshold (typically 5 for production), the connection is immediately terminated with an HTTP 403 Forbidden status code.

2. Compiling & Installing ModSecurity 3 on Nginx

On modern Debian/Ubuntu servers, install the pre-compiled ModSecurity v3 library and dynamic Nginx connector module:

# Install prerequisites and compiler tools
sudo apt update && sudo apt install -y libmodsecurity3 libmodsecurity-dev git curl

# Download and compile libmodsecurity connector for Nginx
git clone --depth 1 https://github.com/owasp-modsecurity/ModSecurity-nginx.git /opt/modsecurity-nginx

Load the dynamic module in your primary /etc/nginx/nginx.conf file:

load_module modules/ngx_http_modsecurity_module.so;

http {
    # Enable ModSecurity globally
    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;
    ...
}

3. Downloading & Activating the OWASP Core Rule Set

Clone the official OWASP CRS repository and configure the baseline rule definitions:

sudo mkdir -p /etc/nginx/modsec && cd /etc/nginx/modsec

# Download recommended baseline configuration
sudo curl -sSL https://raw.githubusercontent.com/owasp-modsecurity/ModSecurity/v3/master/modsecurity.conf-recommended -o modsecurity.conf
sudo curl -sSL https://raw.githubusercontent.com/owasp-modsecurity/ModSecurity/v3/master/unicode.mapping -o unicode.mapping

# Switch from DetectionOnly to On (Active Blocking)
sudo sed -i 's/SecRuleEngine DetectionOnly/SecRuleEngine On/' modsecurity.conf

# Clone official OWASP CRS v4
sudo git clone https://github.com/coreruleset/coreruleset.git /etc/nginx/modsec/coreruleset
sudo cp /etc/nginx/modsec/coreruleset/crs-setup.conf.example /etc/nginx/modsec/coreruleset/crs-setup.conf

Create the master inclusion configuration file at /etc/nginx/modsec/main.conf:

# Master ModSecurity Configuration File
Include /etc/nginx/modsec/modsecurity.conf
Include /etc/nginx/modsec/coreruleset/crs-setup.conf
Include /etc/nginx/modsec/coreruleset/rules/*.conf

4. Configuring OWASP CRS on OpenLiteSpeed

If you run OpenLiteSpeed rather than Nginx, ModSecurity support is built directly into the web server engine without requiring manual compilation:

1. Log in to the OpenLiteSpeed WebAdmin Console (port 7080).
2. Navigate to Server Configuration > Security > Web Application Firewall (WAF).
3. Set Enable WAF to Yes.
4. Set Rule Set to include your cloned OWASP CRS rule directory:

   /usr/local/lsws/conf/crs/crs-setup.conf
   /usr/local/lsws/conf/crs/rules/*.conf

5. Perform a Graceful Restart of OpenLiteSpeed.

5. Managing False Positives & WordPress Exclusions

In production, strict WAF rules can occasionally trigger false positives on legitimate administrative operations (such as saving complex HTML or custom JavaScript in the WordPress Gutenberg editor). OWASP CRS provides official exclusion packages designed specifically for popular applications.

Open /etc/nginx/modsec/coreruleset/crs-setup.conf and locate Section tx.crs_exclusions:

# Enable WordPress application-specific rule exclusions
SecAction  "id:900130,  phase:1,  nolog,  pass,  t:none,  setvar:tx.crs_exclusions_wordpress=1"

This directive instructs the WAF engine to relax inspection on known Gutenberg AJAX endpoints (wp-admin/admin-ajax.php and REST API JSON schemas), eliminating false positives while maintaining ironclad protection against external exploit payloads.

6. Testing and Verifying WAF Blocking

Test your active firewall by simulating a synthetic SQL injection attack using curl:

curl -I "https://yourdomain.com/?id=1%27%20UNION%20SELECT%20null,username,password%20FROM%20users--"

ModSecurity intercepts the attack signature and returns an instantaneous HTTP 403 Forbidden status code. Inspect the detailed audit event in /var/log/modsec_audit.log to view the triggered rule IDs, decoded attack strings, and matched anomaly points.

ModSecurity Production Tuning: Paranoia Levels, Tuning & Alerting

Optimizing the OWASP Core Rule Set on production servers requires understanding Paranoia Levels (PL) and tuning false positive thresholds:

• Understanding OWASP Paranoia Levels: CRS organizes rules into 4 progressive Paranoia Levels:
  • PL1 (Default): Standard enterprise baseline. Zero or virtually no false positives. Catches standard SQLi, XSS, and LFI.
  • PL2: Adds strict regular expressions for path traversal, restricted characters, and advanced command injection. Recommended for high-security applications.
  • PL3 & PL4: Maximum security designed for financial and military systems. Rejects any non-standard HTTP encoding, requiring custom whitelist rules for virtually every form.
• Adjusting Anomaly Thresholds in crs-setup.conf: Customize the inbound blocking threshold based on your application’s risk tolerance:

  SecAction  "id:900110,  phase:1,  nolog,  pass,  t:none,  setvar:tx.inbound_anomaly_score_threshold=5,  setvar:tx.outbound_anomaly_score_threshold=4"

• Streaming WAF Audit Logs to ELK / SIEM: ModSecurity produces rich JSON audit logs detailing matched variables, request headers, and matched regex patterns. Configure SecAuditLogFormat JSON in modsecurity.conf to forward security events into Graylog, Grafana Loki, or Elasticsearch for automated alerting.
• Whitelisting Legitimate Custom API Endpoints: If your web application utilizes raw JSON webhooks (such as Stripe or PayPal payment notifications), create explicit bypass rules matching the webhook path:

  SecRule REQUEST_URI "@streq /api/v1/stripe-webhook"     "id:10001,    phase:1,    pass,    nolog,    ctl:ruleEngine=Off"