{"id":5029,"date":"2026-10-11T00:14:02","date_gmt":"2026-10-10T18:44:02","guid":{"rendered":"https:\/\/cpanelfree.com\/blog\/how-to-rate-limit-aggressive-ai-crawlers-with-openresty-and-lua-without-blocking-seo\/"},"modified":"2026-10-11T00:14:02","modified_gmt":"2026-10-10T18:44:02","slug":"how-to-rate-limit-aggressive-ai-crawlers-with-openresty-and-lua-without-blocking-seo","status":"publish","type":"post","link":"https:\/\/cpanelfree.com\/blog\/how-to-rate-limit-aggressive-ai-crawlers-with-openresty-and-lua-without-blocking-seo\/","title":{"rendered":"How to Rate Limit Aggressive AI Crawlers with OpenResty and Lua without Blocking SEO"},"content":{"rendered":"

Unregulated crawling from commercial Large Language Model scrapers and autonomous AI retrieval agents has created an unprecedented availability crisis for production Linux web infrastructure. When distributed bot swarms originating from providers like ByteDance, OpenAI, Anthropic, and Perplexity aggressively scrape dynamic endpoints on platforms hosted via CpanelFree<\/a>, they routinely saturate PHP-FPM worker pools, exhaust database connection limits, and induce crippling 504 gateway timeouts. Applying blunt IP-level throttling or crude User-Agent blacklists inevitably causes severe collateral damage by blocking verified search engine crawlers like Googlebot and Bingbot, decimating organic search visibility. By embedding Lua-driven traffic classification directly into the OpenResty Nginx event loop with non-blocking shared dictionary memory, systems architects can surgically throttle predatory AI bots to strict token-bucket envelopes while guaranteeing zero latency or crawl budget penalties for critical SEO spiders.<\/p>\n

<\/p>\n

Architectural Paradigm: Differentiating AI Data Scrapers from Search Engine Crawlers<\/h2>\n
\n

Direct Answer:<\/strong> To rate limit AI crawlers with OpenResty and Lua without blocking SEO, inspect incoming request headers in the access_by_lua_block<\/code> phase, authenticate search engine bots using cached forward-confirmed reverse DNS (FCrDNS), and apply dynamic token-bucket rate limits against AI crawler signatures using resty.limit.req<\/code> and shared memory dictionaries without dropping organic search engine spiders.<\/p>\n<\/div>\n

The operational profile of generative AI crawlers differs fundamentally from traditional search indexing spiders. Search engine bots such as Googlebot, Bingbot, and DuckDuckBot are engineered to respect webmaster crawl limits, adhere rigorously to robots.txt<\/code> directives, honor HTTP 429 Retry-After<\/code> headers, and moderate request concurrency based on measured origin server latency. Their objective is discovery, indexing, and bidirectional traffic referral. In contrast, AI scraping agents (including Bytespider, CCBot, GPTBot, ClaudeBot, and Amazonbot) are designed for massive dataset extraction, model pre-training, and near-real-time Retrieval-Augmented Generation (RAG). They frequently ignore crawl-delay directives, execute concurrent scrapers across hundreds of distributed cloud subnets, and aggressively trigger cache-busting search queries, pagination loops, and faceted filter permutations that completely bypass reverse proxy microcaching layers.<\/p>\n

When origin servers experience this scraping intensity, traditional edge defenses routinely fail. Naive User-Agent blocking is ineffective because malicious or aggressive scrapers easily rotate their headers to mimic common desktop browsers or even impersonate Googlebot. Conversely, applying global IP-based rate limiting (such as Nginx’s built-in limit_req_zone $binary_remote_addr<\/code>) inadvertently throttles Googlebot during heavy re-indexing passes, triggering Google Search Console crawl errors, de-indexing events, and substantial organic ranking drops. To establish a robust defense, engineering teams must deploy an intelligent, programmable edge proxy capable of verifying crawler identities at wire speed.<\/p>\n

\n

Architecture Note:<\/strong> Never rely on static User-Agent filtering alone to grant rate limit exemptions. Sophisticated scrapers frequently forge headers such as Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)<\/code>. True protection requires Forward-Confirmed Reverse DNS (FCrDNS) verification, executed asynchronously within the proxy and cached in shared memory to avoid DNS resolution bottlenecks.<\/p>\n<\/blockquote>\n

Benchmarking Crawler Management Architectures<\/h2>\n

Before implementing custom Lua logic, systems engineers must evaluate the operational trade-offs of existing bot mitigation strategies. Below is a comprehensive architectural comparison benchmarking standard Nginx rate limiting, cloud-hosted WAF rules, and an OpenResty Lua shared-memory architecture across critical performance and reliability dimensions:<\/p>\n

\n\n\n\n\n\n\n\n\n\n\n
Architectural Metric<\/th>\nStandard Nginx limit_req<\/code><\/th>\nCommercial Cloud WAF<\/th>\nOpenResty + Lua Shdict<\/th>\n<\/tr>\n<\/thead>\n
Bot Identification Precision<\/td>\nCrude (IP address or static string)<\/td>\nProprietary heuristics \/ Blackbox<\/td>\nSurgical (Regex + FCrDNS verification)<\/td>\n<\/tr>\n
SEO False-Positive Risk<\/td>\nHigh (Throttles shared bot IP pools)<\/td>\nModerate (CAPTCHA challenges search bots)<\/td>\nZero (Cryptographic FCrDNS whitelist)<\/td>\n<\/tr>\n
Request Latency Overhead<\/td>\n< 0.1 ms (Hash table lookup)<\/td>\n15 ms – 45 ms (External CDN hop)<\/td>\n0.2 ms – 0.4 ms (LuaJIT shared dict)<\/td>\n<\/tr>\n
Dynamic Throttling Granularity<\/td>\nFixed rate\/burst per zone<\/td>\nCoarse rate tiers per plan<\/td>\nGranular per-bot leaky bucket & delay<\/td>\n<\/tr>\n
Memory Footprint per 100k Sessions<\/td>\n~6.4 MB (64-byte binary IP nodes)<\/td>\nOffloaded to vendor cloud<\/td>\n~12 MB (Shdict LRU memory slab)<\/td>\n<\/tr>\n
Infrastructure Cost \/ Licensing<\/td>\n$0 (Standard open-source Nginx)<\/td>\n$200 – $3,000+ \/ mo (Usage tiered)<\/td>\n$0 (Open-source OpenResty stack)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

The OpenResty Request Pipeline: Non-Blocking Execution in the Access Phase<\/h2>\n

OpenResty integrates the LuaJIT runtime directly into Nginx’s asynchronous, event-driven architecture. Nginx processes HTTP transactions through eleven distinct phases, including post-read<\/code>, server-rewrite<\/code>, find-config<\/code>, rewrite<\/code>, post-rewrite<\/code>, pre-access<\/code>, access<\/code>, post-access<\/code>, pre-content<\/code>, content<\/code>, and log<\/code>. The optimal injection point for bot rate limiting is the access phase<\/strong>, executed via access_by_lua_block<\/code> or access_by_lua_file<\/code>.<\/p>\n

Executing crawler verification during the access phase ensures that unauthorized or abusive requests are intercepted and terminated before Nginx allocates resources to upstream reverse proxy buffers, PHP-FPM FastCGI processes, or Python WSGI workers. If an AI scraper violates rate thresholds, OpenResty immediately generates an HTTP 429 Too Many Requests response with standard Retry-After<\/code> headers directly from worker memory, completely shielding backend application servers and databases from CPU load.<\/p>\n

To implement this without performance degradation, OpenResty provides thread-safe shared memory dictionaries (lua_shared_dict<\/code>). These dictionaries reside in Nginx’s master process shared memory segment, making them accessible across all worker processes without inter-process locking contention or serialization overhead. By combining lua_shared_dict<\/code> with the official resty.limit.req<\/code> library, we implement a memory-efficient leaky-bucket rate limiter that throttles bots by calculating request arrival intervals and dynamically delaying or rejecting excess traffic.<\/p>\n

\n

Security Advisory:<\/strong> Never perform synchronous DNS lookups using blocking system calls (such as standard POSIX gethostbyaddr<\/code>) inside an Nginx worker process. If an influx of crawler traffic triggers dozens of simultaneous blocking DNS resolutions, the Nginx worker event loop will freeze, causing latency spikes for all concurrent web visitors. Always utilize OpenResty’s non-blocking coroutine-based resolver (resty.dns.resolver<\/code>) backed by local caching.<\/p>\n<\/blockquote>\n

Production Configuration: Step-by-Step Deployment<\/h2>\n

Below are production-ready configuration files designed for enterprise deployment on Linux web servers running OpenResty. Follow the file paths and configurations precisely to construct an autonomous, high-performance crawler control gateway.<\/p>\n

1. OpenResty Core Configuration: \/etc\/openresty\/conf.d\/ai_crawler_ratelimit.conf<\/h3>\n

This configuration defines the shared memory dictionaries for rate limiting and DNS caching, establishes non-blocking internal resolvers, and attaches the Lua access controller to incoming HTTP traffic:<\/p>\n

# \/etc\/openresty\/conf.d\/ai_crawler_ratelimit.conf\n# Define shared memory dictionaries for rate limiting and DNS validation\nlua_shared_dict ai_crawler_limit 20m;\nlua_shared_dict dns_verify_cache 10m;\n\n# Internal non-blocking DNS resolver with caching\nresolver 1.1.1.1 8.8.8.8 ipv6=off valid=600s;\nresolver_timeout 2s;\n\n# Global Lua package path\nlua_package_path \"\/etc\/openresty\/lua\/?.lua;;\";\n\nserver {\n    listen 80;\n    listen 443 ssl http2;\n    server_name example.com www.example.com;\n\n    # SSL configuration omitted for brevity\n    # ssl_certificate \/etc\/ssl\/certs\/example.com.crt;\n    # ssl_certificate_key \/etc\/ssl\/private\/example.com.key;\n\n    # Custom structured JSON access log for crawler auditing\n    log_format crawler_json escape=json '{'\n        '\"time_local\":\"$time_local\",'\n        '\"remote_addr\":\"$remote_addr\",'\n        '\"request_method\":\"$request_method\",'\n        '\"request_uri\":\"$request_uri\",'\n        '\"status\":\"$status\",'\n        '\"body_bytes_sent\":\"$body_bytes_sent\",'\n        '\"http_referer\":\"$http_referer\",'\n        '\"http_user_agent\":\"$http_user_agent\",'\n        '\"crawler_type\":\"$sent_http_x_crawler_type\",'\n        '\"rate_limited\":\"$sent_http_x_ratelimit_action\"'\n    '}';\n\n    access_log \/var\/log\/openresty\/crawler_access.log crawler_json;\n\n    location \/ {\n        # Intercept and rate limit requests during the access phase\n        access_by_lua_file \/etc\/openresty\/lua\/crawler_control.lua;\n\n        # Standard upstream application proxy\n        proxy_pass http:\/\/127.0.0.1:8080;\n        proxy_set_header Host $host;\n        proxy_set_header X-Real-IP $remote_addr;\n        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n        proxy_set_header X-Forwarded-Proto $scheme;\n        proxy_connect_timeout 5s;\n        proxy_read_timeout 60s;\n        proxy_send_timeout 60s;\n    }\n}<\/code><\/pre>\n
2. The Production Lua Engine: \/etc\/openresty\/lua\/crawler_control.lua<\/h3>\n
Save the following script to \/etc\/openresty\/lua\/crawler_control.lua<\/code>. It parses User-Agent headers, executes non-blocking Forward-Confirmed Reverse DNS (FCrDNS) checks for major search engines, caches verified IPs for 24 hours, and applies strict leaky-bucket rate limits against unverified bots and commercial AI scrapers:<\/p>\n
-- \/etc\/openresty\/lua\/crawler_control.lua\nlocal limit_req = require \"resty.limit.req\"\nlocal resolver = require \"resty.dns.resolver\"\n\nlocal user_agent = ngx.var.http_user_agent or \"\"\nlocal client_ip = ngx.var.remote_addr\n\n-- Initialize rate limiters on shared dictionary\n-- AI Crawlers: 5 requests per minute (0.083 req\/s) with burst allowance of 2\nlocal lim_ai, err_ai = limit_req.new(\"ai_crawler_limit\", 5\/60, 2)\nif not lim_ai then\n    ngx.log(ngx.ERR, \"Failed to instantiate AI rate limiter: \", err_ai)\n    return\nend\n\n-- Fallback scraper limit: 30 requests per minute with burst of 10\nlocal lim_general, err_gen = limit_req.new(\"ai_crawler_limit\", 30\/60, 10)\nif not lim_general then\n    ngx.log(ngx.ERR, \"Failed to instantiate General rate limiter: \", err_gen)\n    return\nend\n\n-- Shared dictionary for DNS resolution caching (24-hour TTL)\nlocal dns_cache = ngx.shared.dns_verify_cache\n\n-- Known aggressive AI crawlers and data brokers regex pattern\nlocal ai_pattern = \"(?i)(Bytespider|GPTBot|ChatGPT-User|ClaudeBot|Anthropic-AI|PerplexityBot|CCBot|Diffbot|Amazonbot|FacebookBot|cohere-ai|Omgilibot|OAI-SearchBot|ImagesiftBot)\"\n\n-- Legitimate search engine signatures requiring FCrDNS verification\nlocal seo_patterns = {\n    google = \"(?i)Googlebot\",\n    bing = \"(?i)bingbot\",\n    apple = \"(?i)Applebot\",\n    yandex = \"(?i)YandexBot\"\n}\n\n-- Function: Non-blocking Forward-Confirmed Reverse DNS (FCrDNS) Verification\nlocal function verify_seo_crawler(ip, bot_type)\n    local cache_key = \"fcrdns:\" .. ip\n    local cached_status = dns_cache:get(cache_key)\n    if cached_status ~= nil then\n        return cached_status == \"VERIFIED\"\n    end\n\n    local r, err = resolver:new{\n        nameservers = {\"1.1.1.1\", \"8.8.8.8\"},\n        retrans = 2,\n        timeout = 1500, -- 1.5 second non-blocking timeout\n    }\n    if not r then\n        ngx.log(ngx.WARN, \"DNS resolver initialization error: \", err)\n        return false\n    end\n\n    -- Reverse PTR lookup\n    local answers, err_ptr = r:query(r:arpa_str(ip), {qtype = r.TYPE_PTR})\n    if not answers or answers.errcode then\n        dns_cache:set(cache_key, \"FAILED\", 3600) -- Cache negative lookup for 1 hour\n        return false\n    end\n\n    local ptr_hostname = nil\n    for _, ans in ipairs(answers) do\n        if ans.ptrdname then\n            ptr_hostname = ans.ptrdname\n            break\n        end\n    end\n\n    if not ptr_hostname then\n        dns_cache:set(cache_key, \"FAILED\", 3600)\n        return false\n    end\n\n    -- Validate authoritative search engine domains\n    local valid_domain = false\n    if bot_type == \"google\" and (ptr_hostname:match(\"%.googlebot%.com$\") or ptr_hostname:match(\"%.google%.com$\")) then\n        valid_domain = true\n    elseif bot_type == \"bing\" and ptr_hostname:match(\"%.search%.msn%.com$\") then\n        valid_domain = true\n    elseif bot_type == \"apple\" and ptr_hostname:match(\"%.applebot%.apple%.com$\") then\n        valid_domain = true\n    elseif bot_type == \"yandex\" and (ptr_hostname:match(\"%.yandex%.ru$\") or ptr_hostname:match(\"%.yandex%.com$\") or ptr_hostname:match(\"%.yandex%.net$\")) then\n        valid_domain = true\n    end\n\n    if not valid_domain then\n        ngx.log(ngx.WARN, \"Spoofed search bot detected from IP: \", ip, \" with PTR: \", ptr_hostname)\n        dns_cache:set(cache_key, \"FAILED\", 86400)\n        return false\n    end\n\n    -- Forward lookup to verify reverse mapping matches client IP\n    local forward_ans, err_fwd = r:query(ptr_hostname, {qtype = r.TYPE_A})\n    if not forward_ans or forward_ans.errcode then\n        dns_cache:set(cache_key, \"FAILED\", 3600)\n        return false\n    end\n\n    for _, ans in ipairs(forward_ans) do\n        if ans.address == ip then\n            dns_cache:set(cache_key, \"VERIFIED\", 86400) -- Cache verified bot IP for 24 hours\n            return true\n        end\n    end\n\n    dns_cache:set(cache_key, \"FAILED\", 86400)\n    return false\nend\n\n-- Step 1: Detect and verify legitimate search engines\nfor bot_name, pattern in pairs(seo_patterns) do\n    if ngx.re.find(user_agent, pattern, \"jo\") then\n        local is_legit = verify_seo_crawler(client_ip, bot_name)\n        if is_legit then\n            ngx.header[\"X-Crawler-Type\"] = \"Verified-\" .. bot_name\n            ngx.header[\"X-RateLimit-Action\"] = \"BYPASS\"\n            return -- Legitimate search engine: bypass all rate limits\n        else\n            -- Spoofed bot signature: reject immediately with 403 Forbidden\n            ngx.header[\"X-Crawler-Type\"] = \"Spoofed-\" .. bot_name\n            ngx.header[\"X-RateLimit-Action\"] = \"BLOCKED\"\n            ngx.status = ngx.HTTP_FORBIDDEN\n            ngx.say('{\"error\":\"Forbidden\",\"message\":\"Spoofed search crawler identity detected.\"}')\n            ngx.exit(ngx.HTTP_FORBIDDEN)\n        end\n    end\nend\n\n-- Step 2: Rate limit aggressive AI crawlers\nif ngx.re.find(user_agent, ai_pattern, \"jo\") then\n    ngx.header[\"X-Crawler-Type\"] = \"AI-Scraper\"\n    local delay, err = lim_ai:incoming(client_ip, true)\n    if not delay then\n        if err == \"rejected\" then\n            ngx.header[\"X-RateLimit-Action\"] = \"RATE_LIMITED\"\n            ngx.header[\"Retry-After\"] = \"60\"\n            ngx.status = 429\n            ngx.header[\"Content-Type\"] = \"application\/json\"\n            ngx.say('{\"error\":\"Too Many Requests\",\"message\":\"AI scraping rate limit exceeded. Retry in 60 seconds.\"}')\n            ngx.exit(429)\n        end\n        ngx.log(ngx.ERR, \"Failed to evaluate AI rate limit: \", err)\n        return\n    end\n\n    if delay >= 0.001 then\n        ngx.sleep(delay) -- Apply non-blocking sub-second sleep to smooth burst\n    end\n    ngx.header[\"X-RateLimit-Action\"] = \"THROTTLED\"\n    return\nend\n\n-- Step 3: Default unclassified traffic passes through cleanly\nngx.header[\"X-Crawler-Type\"] = \"Standard-Client\"\nngx.header[\"X-RateLimit-Action\"] = \"ALLOWED\"<\/code><\/pre>\n
3. Kernel & Network Buffer Tuning: \/etc\/sysctl.d\/99-openresty-highconcurrency.conf<\/h3>\n
When handling thousands of crawler requests while maintaining microsecond response times for end users, the underlying Linux kernel TCP stack must be tuned to prevent socket exhaustion and SYN packet dropping:<\/p>\n
# \/etc\/sysctl.d\/99-openresty-highconcurrency.conf\n# Maximize socket listen backlog for high connection bursts\nnet.core.somaxconn = 65535\nnet.ipv4.tcp_max_syn_backlog = 65535\n\n# Increase network interface input queue\nnet.core.netdev_max_backlog = 16384\n\n# Enable rapid reuse of TIME_WAIT sockets for outgoing proxy connections\nnet.ipv4.tcp_tw_reuse = 1\nnet.ipv4.tcp_fin_timeout = 15\n\n# Expand ephemeral port range to support high upstream proxy density\nnet.ipv4.ip_local_port_range = 1024 65535\n\n# Increase system-wide file descriptor limit\nfs.file-max = 2097152\n\n# Protect against TCP SYN flood attacks during distributed scraping runs\nnet.ipv4.tcp_syncookies = 1<\/code><\/pre>\n
Apply these kernel parameters immediately without rebooting by executing:<\/p>\n
sudo sysctl -p \/etc\/sysctl.d\/99-openresty-highconcurrency.conf<\/code><\/pre>\n
Operational Validation & Production Telemetry<\/h2>\n
Once deployed, verify that the Lua rate limiter accurately distinguishes between legitimate search engines, spoofed crawlers, and aggressive AI scrapers using targeted curl<\/code> diagnostic probes:<\/p>\n
# Test 1: Simulate spoofed Googlebot from non-Google IP (Expect 403 Forbidden)\ncurl -i -H \"User-Agent: Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)\" https:\/\/example.com\/\n\n# Test 2: Simulate ByteSpider AI scraper within threshold (Expect 200 OK with throttled headers)\ncurl -i -H \"User-Agent: Mozilla\/5.0 (compatible; Bytespider; +https:\/\/zhanzhang.toutiao.com\/)\" https:\/\/example.com\/\n\n# Test 3: Rapid burst test against AI rate limiter (Expect 429 Too Many Requests with Retry-After)\nfor i in {1..6}; do\n    curl -s -o \/dev\/null -w \"%{http_code}\\n\" -H \"User-Agent: Mozilla\/5.0 (compatible; GPTBot\/1.0; +https:\/\/openai.com\/gptbot)\" https:\/\/example.com\/\ndone<\/code><\/pre>\n
Reviewing the structured JSON log at \/var\/log\/openresty\/crawler_access.log<\/code> allows monitoring tools (such as Vector, Fluentbit, or Prometheus) to generate real-time metrics tracking the volume of throttled AI scrapers versus unhindered search engine requests.<\/p>\n
Transitioning from Staging to Enterprise Production Infrastructure<\/h2>\n
Implementing custom OpenResty reverse proxy architectures on staging platforms like CpanelFree<\/a> provides an outstanding zero-cost environment for profiling Lua regex expressions, calibrating token-bucket burst limits, and stress-testing DNS resolution coroutines. However, when migrating high-concurrency web properties to production where sustained AI scraping coexists with millions of human pageviews, infrastructure underlying the edge proxy becomes paramount.<\/p>\n
Operating shared dictionaries, SSL termination, and dynamic LuaJIT execution under massive scraper volume requires deterministic CPU thread allocation, ultra-low memory latency, and carrier-grade network transit. For enterprise web deployments that cannot tolerate jitter, throttling overhead, or surprise renewal rate spikes, hosting your edge infrastructure on MeraHost Enterprise Cloud<\/a> guarantees dedicated AMD EPYC and Intel Xeon compute resources with enterprise NVMe storage arrays. By combining OpenResty’s non-blocking Lua engine with high-performance bare-metal or cloud instances from MeraHost, your applications remain impervious to aggressive scraper storms while delivering peak Core Web Vitals to human visitors and search crawlers alike.<\/p>\n
Frequently Asked Questions<\/h2>\n
\nWhy not simply block all AI crawlers using robots.txt directives?<\/summary>\n
While reputable search engines adhere strictly to robots.txt<\/code>, many commercial AI scrapers and third-party data brokers routinely disregard disallow rules or interpret them selectively. Furthermore, completely disallowing AI crawlers in robots.txt<\/code> prevents your content from being cited in emerging AI search engines (such as Perplexity and SearchGPT). Rate limiting via OpenResty allows you to control the exact request cadence and bandwidth consumption of AI bots without severing visibility in AI search summaries.<\/p>\n<\/details>\n
\nDoes returning HTTP 429 status codes to AI crawlers hurt Google rankings?<\/summary>\n
No, because our OpenResty Lua architecture authenticates Googlebot and Bingbot using Forward-Confirmed Reverse DNS (FCrDNS) and completely exempts verified search crawlers from the rate limiter. Search engine spiders receive normal HTTP 200 responses with full crawl budget allocation. Only unverified bots and matching AI User-Agents receive HTTP 429 status codes, preserving your organic SEO rankings.<\/p>\n<\/details>\n
\nHow does OpenResty handle DNS resolution without slowing down incoming web traffic?<\/summary>\n
Our implementation utilizes the resty.dns.resolver<\/code> module, which executes DNS lookups inside non-blocking coroutines rather than standard synchronous system calls. Furthermore, verified search bot IPs are stored in a 10MB shared memory dictionary (lua_shared_dict dns_verify_cache<\/code>) with a 24-hour TTL. After the initial lookup, subsequent requests from that IP are validated in approximately 0.2 milliseconds directly from memory.<\/p>\n<\/details>\n
\nWhat is the memory overhead of maintaining shared dictionaries for high-traffic sites?<\/summary>\n
Shared dictionaries in OpenResty are allocated as fixed-size shared memory slabs when Nginx boots. A 20MB rate limit zone and a 10MB DNS cache zone consume exactly 30MB of system RAM, regardless of traffic volume. OpenResty manages entries using an internal LRU (Least Recently Used) algorithm, ensuring that active client and crawler states are retained without risks of memory fragmentation or unbounded RAM consumption.<\/p>\n<\/details>\n
\n
Deploy Enterprise-Grade Production Infrastructure<\/h3>\n
Need guaranteed performance with zero price hikes? Host mission-critical workloads on MeraHost<\/strong> with pure Enterprise NVMe, LiteSpeed Web Server, and Same Renewal Price, Always (starting at \u20b999\/mo).<\/p>\n
\n
Explore MeraHost NVMe Cloud →<\/a><\/div>\n
Deploy Free Staging on CpanelFree<\/a><\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Rate limit aggressive AI bots using OpenResty and Lua without penalizing search engines. Implement dynamic shdict throttling and rDNS crawler checks.<\/p>\n","protected":false},"author":1,"featured_media":5028,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[57,177,87,226,101],"class_list":["post-5029","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-web-hosting-news","tag-almalinux","tag-databases-performance","tag-devops","tag-geo-ai-crawlers","tag-sysadmin"],"_links":{"self":[{"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/posts\/5029","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/comments?post=5029"}],"version-history":[{"count":0,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/posts\/5029\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/media\/5028"}],"wp:attachment":[{"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/media?parent=5029"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/categories?post=5029"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/tags?post=5029"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}