Securing Self-Hosted AI API Endpoints with OAuth2 Proxy and Rate Limiting on Nginx

Exposing self-hosted Large Language Model (LLM) inference engines such as vLLM, Ollama, and Text Generation Inference (TGI) directly to production networks presents an existential infrastructure risk, as unauthenticated queries or runaway client loops can exhaust enterprise GPU VRAM, trigger severe thermal throttling, and cause denial-of-wallet resource starvation. Deploying native inference runtimes behind an enterprise-grade reverse proxy layer with centralized identity verification and granular request throttling transforms raw GPU compute nodes into secure, highly available API gateways. At CpanelFree, our infrastructure engineering teams isolate bare-metal AI acceleration clusters behind hardened Nginx reverse proxies coupled with OAuth2 Proxy subrequest authentication to enforce zero-trust access control without degrading streaming token latency.

Architectural Overview: The Zero-Trust AI Gateway Pattern

Traditional web application architectures often bundle authentication directly into the application runtime. However, modern self-hosted inference servers (e.g., vLLM executing PagedAttention or TensorRT-LLM) are optimized exclusively for high-throughput CUDA tensor parallelization, not complex cryptographic authentication handshakes, session token caching, or distributed OAuth2 token validation. Coupling identity verification into the inference daemon introduces security vulnerabilities, bloats process memory, and exposes vulnerable Python runtimes directly to hostile external network traffic.

The decoupled Zero-Trust AI Gateway Architecture solves this operational challenge through strict separation of concerns:

• Edge Ingress & TLS Termination (Nginx): Nginx intercepts all inbound HTTP/2 and HTTP/3 client traffic, manages Let’s Encrypt or corporate TLS certificates, handles TCP connection pooling, and enforces kernel-level socket optimizations.
• Subrequest Authentication (OAuth2 Proxy): Utilizing the Nginx ngx_http_auth_request_module, Nginx fires an internal HTTP GET /oauth2/auth subrequest to OAuth2 Proxy before admitting client payloads. OAuth2 Proxy validates incoming session cookies, Bearer JWT tokens, or API keys against an OpenID Connect (OIDC) Identity Provider (Keycloak, Okta, Authentik, or Google Workspace).
• Dual-Tier Token Bucket Rate Limiting: Nginx inspects the returned authentication headers (such as X-Auth-Request-User) to apply dual-tier rate limiting. Unauthenticated ping endpoints are throttled per IP, while heavy /v1/chat/completions requests are governed per authenticated user account with strict burst ceilings.
• Isolated Backend Inference (vLLM / Ollama): Protected backend daemons bind strictly to localhost (127.0.0.1:8000) or an isolated internal private bridge network, shielded from direct network access.

Performance Matrix: Default vs. Tuned Production Gateway

Deploying subrequest authentication and rate limiting introduces potential latency bottlenecks if TCP sockets and subrequest pipelines are unoptimized. Below is an empirical comparison matrix benchmarking an untuned default Nginx deployment against our production-hardened AI gateway stack under a sustained load of 5,000 concurrent streaming inference requests.

Step 1: Kernel Hardening for Streaming SSE Inference

High-concurrency LLM inference involves long-lived, streaming HTTP connections. A single prompt generation can keep an HTTP socket open for 15 to 120 seconds while tokens are generated sequentially. To prevent ephemeral port starvation, file descriptor exhaustion, and connection queue drops, create a dedicated kernel optimization profile in /etc/sysctl.d/99-ai-proxy.conf.

# /etc/sysctl.d/99-ai-proxy.conf
# Linux Kernel Optimization for High-Concurrency Streaming AI Reverse Proxies

# Expand maximum open file descriptors for massive socket concurrency
fs.file-max = 2097152

# Socket backlog tuning for incoming connection spikes
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 32768
net.core.netdev_max_backlog = 16384

# Enable TCP BBR congestion control for ultra-low streaming latency
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# Socket reuse and ephemeral port range expansion
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535

# Keepalive intervals to detect abandoned client streams without wasting GPU slots
net.ipv4.tcp_keepalive_time = 120
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 4

# Buffer memory allocations for high throughput
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

Apply the new kernel parameters immediately without rebooting:

sudo sysctl --system

Step 2: Deploying and Configuring OAuth2 Proxy

OAuth2 Proxy acts as the identity verification gatekeeper. In an AI infrastructure architecture, it can validate session cookies for human engineers accessing web-based chat playgrounds (e.g., Open WebUI) while simultaneously validating Bearer JWT tokens for programmatic API callers accessing /v1/chat/completions. We configure OAuth2 Proxy to bind to a high-speed local loopback port and leverage Redis for high-speed token and session caching.

Create the production configuration file at /etc/oauth2-proxy/oauth2-proxy.cfg:

# /etc/oauth2-proxy/oauth2-proxy.cfg
# Production configuration for AI API Gateway Authentication

## HTTP Ingress
http_address = "127.0.0.1:4180"
reverse_proxy = true

## Provider Details (OpenID Connect / Keycloak / Authentik)
provider = "oidc"
oidc_issuer_url = "https://auth.example.com/realms/ai-cluster"
client_id = "ai-gateway-proxy"
client_secret = "SECURE_OIDC_CLIENT_SECRET_CHANGE_ME"

## Token & Header Management
pass_access_token = true
pass_authorization_header = true
pass_basic_auth = false
set_xauthrequest = true
pass_user_headers = true

## API Bearer Token Support (Crucial for Programmatic LLM API Callers)
skip_jwt_bearer_tokens = true
extra_jwt_issuers = ["https://auth.example.com/realms/ai-cluster=ai-gateway-proxy"]

## Scope and Email Domain Filtering
scope = "openid email profile"
email_domains = ["example.com"]

## Session Storage (High-Speed Local Redis)
session_store_type = "redis"
redis_connection_url = "redis://127.0.0.1:6379"

## Cookie Security
cookie_name = "_ai_gateway_oauth2"
cookie_secret = "32_BYTE_RANDOM_COOKIE_SECRET_KEY_STRING=="
cookie_secure = true
cookie_httponly = true
cookie_samesite = "lax"
cookie_expire = "12h"
cookie_refresh = "1h"

Next, define the systemd unit file to manage the OAuth2 Proxy process with production security sandboxing:

# /etc/systemd/system/oauth2-proxy.service
[Unit]
Description=OAuth2 Proxy Authentication Service for AI Gateways
After=network.target redis.service
Wants=redis.service

[Service]
Type=simple
User=oauth2-proxy
Group=oauth2-proxy
ExecStart=/usr/local/bin/oauth2-proxy --config=/etc/oauth2-proxy/oauth2-proxy.cfg
Restart=always
RestartSec=5s

# Security Hardening Directives
ProtectSystem=strict
ProtectHome=true
NoNewPrivileges=true
PrivateTmp=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now oauth2-proxy.service
sudo systemctl status oauth2-proxy.service

Step 3: Hardened Nginx Gateway with Dual-Tier Rate Limiting

With OAuth2 Proxy running, we configure Nginx to intercept all inbound client traffic. This configuration accomplishes four vital architectural mandates:

1. Subrequest Verification: Nginx invokes auth_request /oauth2/auth;. If OAuth2 Proxy returns HTTP 202/200, Nginx forwards the request to the inference backend. If it returns HTTP 401, Nginx blocks the request or redirects web users to login.
2. User-Aware Token-Bucket Rate Limiting: Two separate rate-limiting zones are maintained in shared memory. One zone throttles based on client IP (preventing port scans and brute force attacks), while the second zone extracts $auth_user from the verified OAuth2 Proxy response header to enforce per-user inference quotas.
3. SSE Real-Time Streaming Optimization: Buffer management directives (proxy_buffering off;, proxy_cache off;) ensure tokens generated by vLLM or Ollama stream instantly to the client without buffering latency.
4. Security Header Hardening: Strips server technology fingerprints and injects modern defense-in-depth headers.

Create the virtual host configuration at /etc/nginx/conf.d/ai_gateway.conf:

# /etc/nginx/conf.d/ai_gateway.conf
# Hardened Nginx AI Gateway with OAuth2 Subrequest Auth & Rate Limiting

# 1. Rate Limiting Shared Memory Zones
# Zone 1: Rate limit by Client IP (20 requests/second for general API ingress)
limit_req_zone $binary_remote_addr zone=ip_ai_limit:20m rate=20r/s;

# Zone 2: Rate limit by Verified User ID passed from OAuth2 Proxy (5 requests/second for heavy inference)
limit_req_zone $auth_user zone=user_inference_limit:20m rate=5r/s;

# Custom 429 JSON response payload instead of generic HTML error
limit_req_status 429;

# Upstream Backend Pool (vLLM OpenAI-Compatible API)
upstream vllm_backend {
    server 127.0.0.1:8000 max_fails=3 fail_timeout=10s;
    keepalive 64;
}

# Upstream OAuth2 Proxy
upstream oauth2_auth_backend {
    server 127.0.0.1:4180;
    keepalive 32;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name ai.example.com;

    # SSL TLS Certificates and Cryptographic Hardening
    ssl_certificate /etc/letsencrypt/live/ai.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ai.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:20m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;

    # Security & Defense-in-Depth Headers
    add_header X-Frame-Options "DENY" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

    # Client payload bounds (Set according to max context window and image size for multimodal models)
    client_max_body_size 64M;

    # 2. OAuth2 Proxy Internal Endpoint Handlers
    location /oauth2/ {
        proxy_pass       http://oauth2_auth_backend;
        proxy_set_header Host                    $host;
        proxy_set_header X-Real-IP               $remote_addr;
        proxy_set_header X-Scheme                $scheme;
        proxy_set_header X-Auth-Request-Redirect $request_uri;
        proxy_connect_timeout 3s;
        proxy_read_timeout 10s;
    }

    # Internal subrequest location called by auth_request
    location = /oauth2/auth {
        internal;
        proxy_pass       http://oauth2_auth_backend;
        proxy_set_header Host             $host;
        proxy_set_header X-Real-IP        $remote_addr;
        proxy_set_header X-Scheme         $scheme;
        # Pass full authorization header containing Bearer JWT
        proxy_set_header Authorization   $http_authorization;
        proxy_pass_request_body          off;
        proxy_set_header Content-Length  "";
        
        # Keepalive upstream optimization
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }

    # Custom 401 Handler for API vs Web
    error_page 401 = @error401;
    location @error401 {
        # Check if caller expects JSON API response or browser HTML redirect
        if ($http_accept ~* "application/json") {
            return 401 '{"error": "unauthorized", "message": "Valid Bearer JWT or OAuth2 session token required"}';
        }
        return 302 https://$host/oauth2/start?rd=$scheme://$host$request_uri;
    }

    # Custom 429 JSON Handler for Rate Limit Exceeded
    error_page 429 = @error429;
    location @error429 {
        add_header Retry-After 5 always;
        add_header Content-Type application/json always;
        return 429 '{"error": "rate_limited", "message": "Inference token quota exceeded. Please throttle requests."}';
    }

    # 3. Public Health Check (Bypasses Auth & Rate Limits for Load Balancers)
    location = /healthz {
        access_log off;
        return 200 '{"status": "healthy", "service": "ai-gateway"}';
        add_header Content-Type application/json;
    }

    # 4. Protected AI Inference API Endpoint (/v1/chat/completions, /v1/models, /v1/embeddings)
    location /v1/ {
        # Enforce OAuth2 Subrequest Authentication
        auth_request /oauth2/auth;

        # Capture authenticated identity headers from OAuth2 Proxy
        auth_request_set $auth_user  $upstream_http_x_auth_request_user;
        auth_request_set $auth_email $upstream_http_x_auth_request_email;
        auth_request_set $auth_token $upstream_http_x_auth_request_access_token;

        # Dual-Tier Rate Limiting:
        # Limit by IP: burst 10 requests, delay excess
        limit_req zone=ip_ai_limit burst=10 nodelay;
        # Limit by Verified User: burst 5 requests, strict queuing
        limit_req zone=user_inference_limit burst=5;

        # Upstream proxy to vLLM / Ollama
        proxy_pass http://vllm_backend;
        proxy_http_version 1.1;

        # CRITICAL: Disable Buffering for Real-Time SSE Token Streaming
        proxy_buffering off;
        proxy_cache off;
        chunked_transfer_encoding on;
        tcp_nodelay on;

        # Extended timeouts for long-context generative generation
        proxy_connect_timeout 5s;
        proxy_send_timeout 300s;
        proxy_read_timeout 300s;

        # Pass authenticated audit context downstream to vLLM
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Consumer-User $auth_user;
        proxy_set_header X-Consumer-Email $auth_email;

        # Enable upstream socket keepalive
        proxy_set_header Connection "";
    }
}

Step 4: End-to-End Testing and Verification

Once the gateway is active, verify that unauthenticated requests, valid Bearer JWT calls, and rate-limiting limits behave as expected.

1. Testing Unauthenticated Request Rejection

Querying the protected /v1/models endpoint without an authorization header must immediately return a 401 Unauthorized response from Nginx without hitting the inference backend:

curl -i -H "Accept: application/json" https://ai.example.com/v1/models

Expected HTTP response:

HTTP/2 401 
server: nginx
content-type: application/json
content-length: 78

{"error": "unauthorized", "message": "Valid Bearer JWT or OAuth2 session token required"}

2. Testing Authenticated Streaming Inference

Execute an authenticated streaming completion request by passing a valid OIDC Bearer token in the Authorization header:

curl -N -X POST https://ai.example.com/v1/chat/completions   -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI..."   -H "Content-Type: application/json"   -d '{
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "messages": [{"role": "user", "content": "Explain Linux eBPF ring buffers in two sentences."}],
    "stream": true
  }'

Because proxy_buffering off; and tcp_nodelay on; are enabled, server tokens will arrive sequentially in real time via Server-Sent Events without batching delays.

3. Verifying Rate Limit Enforcement

Simulate a concurrent burst of automated requests to verify the user_inference_limit token bucket:

for i in {1..15}; do
  curl -s -o /dev/null -w "%{http_code}
" -X POST https://ai.example.com/v1/chat/completions     -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI..."     -H "Content-Type: application/json"     -d '{"model": "test", "messages": [{"role": "user", "content": "ping"}]}' &
done
wait

Requests exceeding the burst threshold of 5 will return HTTP 429 Too Many Requests with the customized JSON error payload, shielding the GPU scheduling queue from thread lockups.

Frequently Asked Questions (FAQ)