{"id":4586,"date":"2026-09-19T11:01:48","date_gmt":"2026-09-19T05:31:48","guid":{"rendered":"https:\/\/cpanelfree.com\/blog\/linux-kernel-memory-reclamation-tuning-vmswappiness-vfscachepressure-and-watermarks\/"},"modified":"2026-09-19T11:01:48","modified_gmt":"2026-09-19T05:31:48","slug":"linux-kernel-memory-reclamation-tuning-vmswappiness-vfscachepressure-and-watermarks","status":"publish","type":"post","link":"https:\/\/cpanelfree.com\/blog\/linux-kernel-memory-reclamation-tuning-vmswappiness-vfscachepressure-and-watermarks\/","title":{"rendered":"Linux Kernel Memory Reclamation Tuning: vm.swappiness, vfs_cache_pressure, and Watermarks"},"content":{"rendered":"

In mission-critical Linux environments, sudden latency spikes and unexplained throughput collapses often stem not from hardware saturation, but from the kernel’s silent, aggressive struggle to free memory pages. When asynchronous background reclamation fails to keep pace with rapid burst allocations, the Linux page allocator forces user-space processes into synchronous direct reclamation\u2014introducing devastating multi-millisecond stalls that cripple web server worker pools and database connection queues. High-performance hosting architectures like CpanelFree<\/a> depend on rigorous kernel memory subsystem tuning to maintain sub-millisecond response times even under sustained peak memory utilization.<\/p>\n

<\/p>\n

Understanding Linux Kernel Memory Reclamation Mechanics<\/h2>\n
\n Direct Answer:<\/strong> Linux memory reclamation balances anonymous memory and page cache eviction using zone watermarks (min, low, high), kswapd, and direct reclamation. Tuning vm.swappiness, vm.vfs_cache_pressure, and vm.watermark_scale_factor prevents synchronous allocation stalls (allocstall), reduces p99 latency spikes, and ensures predictable memory headroom across high-throughput enterprise workloads.\n<\/div>\n

To optimize Linux memory management, systems engineers must first discard the simplistic view that memory is either “free” or “used.” The Linux kernel manages physical RAM through a hierarchy of memory zones across NUMA (Non-Uniform Memory Access) nodes\u2014primarily ZONE_DMA<\/code>, ZONE_DMA32<\/code>, ZONE_NORMAL<\/code>, and on 64-bit platforms, ZONE_MOVABLE<\/code>. Every memory allocation request processed by the buddy allocator evaluates availability within these specific zones against three critical thresholds known as Zone Watermarks<\/strong>:<\/p>\n

    \n
  • WMARK_MIN:<\/strong> The minimum number of free pages reserved for essential kernel operations (such as atomic allocations and interrupt handlers). If free pages in a zone dip below this threshold, the kernel halts asynchronous behavior and forces the requesting process into synchronous direct reclamation<\/em>.<\/li>\n
  • WMARK_LOW:<\/strong> The activation threshold for the kernel swap daemon, kswapd<\/code>. When free pages fall below WMARK_LOW<\/code>, kswapd<\/code> wakes up in the background and asynchronously scans active and inactive page lists to reclaim memory.<\/li>\n
  • WMARK_HIGH:<\/strong> The deactivation threshold. Once kswapd<\/code> reclaims enough pages so that free memory reaches WMARK_HIGH<\/code>, the daemon returns to sleep.<\/li>\n<\/ul>\n

    The gap between WMARK_LOW<\/code> and WMARK_MIN<\/code> constitutes the critical buffer zone. When memory allocation requests occur at a rate faster than kswapd<\/code> can reclaim and compact pages, this buffer is quickly breached. When a process enters direct reclamation (recorded as allocstall<\/code> in \/proc\/vmstat<\/code>), the executing thread is taken off its CPU work to synchronously traverse Least Recently Used (LRU) lists, write dirty pages to storage, or invalidate file-backed page caches. For multi-threaded application servers, database engines, and web gateways, an allocstall<\/code> event results in severe tail-latency spikes exceeding 100ms.<\/p>\n

    \n Architecture Note:<\/strong> The default kernel heuristic for calculating zone watermarks was designed over a decade ago when enterprise servers operated with 4GB to 16GB of RAM. On modern hardware nodes equipped with 128GB to 2TB of memory and ultra-fast NVMe storage, default watermarks wake kswapd<\/code> far too late, virtually guaranteeing direct reclamation stalls during sudden traffic bursts.\n<\/div>\n

    Zone Watermark Tuning: vm.min_free_kbytes and vm.watermark_scale_factor<\/h2>\n

    Historically, administrators attempted to prevent direct reclamation by inflating vm.min_free_kbytes<\/code>. The kernel uses vm.min_free_kbytes<\/code> to derive WMARK_MIN<\/code> for each zone based on its proportional size. By default, the kernel calculates this value using a square-root heuristic:<\/p>\n

    min_free_kbytes = 16 * sqrt(lowmem_kbytes)<\/code><\/pre>\n
    On a 64GB machine, this formula yields approximately 67MB of reserved memory. In a high-traffic production environment handling 50,000 requests per second or multi-gigabyte database queries, 67MB of headroom can be consumed in fractions of a millisecond. While increasing vm.min_free_kbytes<\/code> to 512MB or 1GB raises WMARK_MIN<\/code>, it also locks that memory away permanently, preventing user-space processes from ever utilizing it.<\/p>\n
    Starting in Linux kernel 4.6, the kernel introduced vm.watermark_scale_factor<\/code>, providing a far more elegant and effective mechanism. Instead of locking away massive static chunks of RAM, vm.watermark_scale_factor<\/code> directly controls the distance between WMARK_MIN<\/code>, WMARK_LOW<\/code>, and WMARK_HIGH<\/code> as a percentage of the total zone size (expressed in tenths of a percent, where 10 equals 0.1% and 1000 equals 10%).<\/p>\n
    # Calculate default watermark spacing (10 = 0.1% of zone)\ncat \/proc\/sys\/vm\/watermark_scale_factor\n\n# Dynamic calculation used by the kernel:\nwatermark_distance = (zone_present_pages * watermark_scale_factor) \/ 10000\nWMARK_LOW  = WMARK_MIN + watermark_distance\nWMARK_HIGH = WMARK_MIN + (watermark_distance * 2)<\/code><\/pre>\n
    By increasing vm.watermark_scale_factor<\/code> from its default of 10<\/code> (0.1%) to 150<\/code> or 200<\/code> (1.5% to 2.0%), you instruct kswapd<\/code> to wake up much earlier and stay active longer, reclaiming pages well in advance of free memory approaching the critical WMARK_MIN<\/code> line. This effectively eliminates synchronous direct reclamation stalls while preserving memory availability for application heaps and the Linux page cache.<\/p>\n
    Deconstructing vm.swappiness: Algorithm and Misconceptions<\/h2>\n
    One of the most persistent myths in Linux systems administration is that vm.swappiness<\/code> represents a percentage threshold of memory utilization at which the kernel begins swapping (e.g., “swappiness=60 means start swapping at 40% free memory”). This is mathematically false.<\/p>\n
    In the Linux kernel’s memory management subsystem (specifically inside mm\/vmscan.c:get_scan_count()<\/code>), vm.swappiness<\/code> serves as a weighting ratio in the scan balance algorithm. The kernel maintains two primary LRU page lists: Anonymous pages<\/strong> (heap, stack, private memory allocations) and File-backed pages<\/strong> (cached disk reads, executable binaries, shared libraries). When memory reclamation occurs, the kernel must decide whether to evict clean file cache pages or swap out anonymous pages to persistent swap storage.<\/p>\n
    The kernel calculates scan targets using proportional ratios:<\/p>\n
    # Kernel scan balance formula (conceptual representation from mm\/vmscan.c):\nanon_prio = swappiness;\nfile_prio = 200 - swappiness;\n\nscan_anon = (recent_scanned[0] + 1) * anon_prio;\nscan_file = (recent_scanned[1] + 1) * file_prio;<\/code><\/pre>\n
    Notice that the default value of vm.swappiness=60<\/code> yields an anon_prio<\/code> of 60 and a file_prio<\/code> of 140. This default heavily biases the kernel toward preserving anonymous memory and aggressively dropping the file page cache. In database workloads (such as PostgreSQL, MySQL\/MariaDB, or Redis) and web servers running LiteSpeed or NGINX, discarding page cache forces frequent disk reads, introducing severe I\/O bottlenecks.<\/p>\n
    Conversely, setting vm.swappiness=0<\/code> does not completely disable swap on modern kernels. Instead, it prevents the kernel from swapping anonymous pages until the number of free pages and clean file-backed pages drops below the high watermark in a given zone. For modern high-performance cloud servers equipped with enterprise NVMe storage, setting vm.swappiness=10<\/code> or vm.swappiness=1<\/code> ensures that active file cache is preserved while maintaining a safety valve for memory-intensive workloads.<\/p>\n
    Tuning vm.vfs_cache_pressure: Protecting Inodes and Directory Entries<\/h2>\n
    While vm.swappiness<\/code> balances anonymous memory against the general page cache, vm.vfs_cache_pressure<\/code> governs the kernel’s reclamation of Virtual File System (VFS) metadata structures\u2014specifically dentries<\/strong> (directory entries) and inodes<\/strong>. These objects are managed via the SLAB\/SLUB allocator and represent cached filesystem paths, permissions, and file location pointers.<\/p>\n
    The default setting is vm.vfs_cache_pressure=100<\/code>, which instructs the kernel to reclaim dentry and inode cache objects at the same rate as standard page cache and swap pages:<\/p>\n
      \n
    • vfs_cache_pressure = 100 (Default):<\/strong> Balanced reclamation. The kernel treats VFS slab caches and page cache equally.<\/li>\n
    • vfs_cache_pressure < 100 (e.g., 50):<\/strong> The kernel actively prioritizes retaining directory entries and inodes in memory. Reconstructing a dentry or inode requires reading raw disk structures, which is computationally expensive and incurs I\/O latency. Keeping this metadata in RAM drastically accelerates PHP-FPM execution, WordPress asset checks, Git repositories, and large filesystem traversals.<\/li>\n
    • vfs_cache_pressure > 100 (e.g., 200):<\/strong> The kernel aggressively evicts VFS metadata to prioritize keeping anonymous pages and data caches in physical RAM. This is sometimes employed in dedicated compute-only nodes with minimal disk interaction.<\/li>\n
    • vfs_cache_pressure = 0 (Danger):<\/strong> Completely disables VFS cache reclamation. This causes kernel slab memory to grow unbounded, inevitably triggering an Out-Of-Memory (OOM) kernel panic under sustained filesystem operations.<\/li>\n<\/ul>\n
      \n  Architecture Note:<\/strong> In hosting platforms hosting tens of thousands of PHP files, symlinks, and media assets, setting vm.vfs_cache_pressure=50<\/code> reduces file lookup latency by up to 70%, preventing repetitive metadata disk reads across concurrent worker processes.\n<\/div>\n
      Comparative Benchmark Matrix: Default vs. Tuned Linux Memory Reclamation<\/h2>\n
      The following matrix outlines the operational impact of memory subsystem parameters before and after enterprise tuning under a sustained load of 25,000 HTTP requests per second with concurrent database write spikes:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
      Feature \/ Metric<\/th>\nStandard \/ Default<\/th>\nTuned \/ Production<\/th>\n<\/tr>\n<\/thead>\n
      Latency \/ Overhead<\/td>\nBaseline<\/td>\nOptimal<\/td>\n<\/tr>\n
      vm.swappiness<\/td>\n60 (Excessive anon page eviction under load)<\/td>\n10 (Preserves active execution heaps)<\/td>\n<\/tr>\n
      vm.vfs_cache_pressure<\/td>\n100 (Evicts dentries prematurely)<\/td>\n50 (Maintains hot filesystem metadata in RAM)<\/td>\n<\/tr>\n
      vm.watermark_scale_factor<\/td>\n10 (0.1% zone buffer; late kswapd trigger)<\/td>\n150 (1.5% zone buffer; proactive kswapd wakeup)<\/td>\n<\/tr>\n
      vm.min_free_kbytes<\/td>\nDynamic (~67MB on 64GB node)<\/td>\n524288 (512MB dedicated atomic buffer)<\/td>\n<\/tr>\n
      Direct Reclaim Frequency (allocstall)<\/td>\n420 stalls \/ min during burst traffic<\/td>\n0 stalls \/ min (Smooth background absorption)<\/td>\n<\/tr>\n
      p99 Application Latency<\/td>\n285.4 ms (Blocked on page allocators)<\/td>\n2.8 ms (Instantaneous fast-path buddy allocation)<\/td>\n<\/tr>\n
      VFS Dentry Hit Ratio<\/td>\n74.2% (Frequent disk re-traversals)<\/td>\n98.6% (Near-total in-memory resolution)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
      Production Configuration: Hardened sysctl.d Template<\/h2>\n
      To deploy these optimizations persistently across system reboots, create a dedicated configuration file inside \/etc\/sysctl.d\/<\/code>. The following production-grade configuration incorporates memory reclamation tuning, dirty page writeback thresholds, and NUMA zone controls:<\/p>\n
      # ====================================================================\n# \/etc\/sysctl.d\/99-memory-reclamation.conf\n# High-Performance Linux Kernel Memory Reclamation Architecture\n# Target: Multi-Core Enterprise Web, Database & Container Nodes\n# ====================================================================\n\n# 1. Proactive Watermark Headroom\n# Expands the gap between WMARK_MIN and WMARK_HIGH to 1.5% of zone size.\n# Ensures kswapd activates early and avoids synchronous direct reclaim stalls.\nvm.watermark_scale_factor = 150\n\n# 2. Dedicated Emergency Minimum Free Kbytes (512MB)\n# Prevents atomic allocation failures for high-speed network ring buffers.\nvm.min_free_kbytes = 524288\n\n# 3. Memory Scan Bias (vm.swappiness)\n# Biases the kernel toward evicting inactive page cache over anonymous heap pages.\n# Optimal balance for NVMe-backed database and web hosting workloads.\nvm.swappiness = 10\n\n# 4. VFS Cache Pressure (Dentries & Inodes)\n# Retains directory and inode slab structures in RAM to accelerate file lookups.\nvm.vfs_cache_pressure = 50\n\n# 5. Page Cache Dirty Writeback Management\n# Starts asynchronous background flushing when dirty pages reach 5% of memory.\nvm.dirty_background_ratio = 5\n\n# Forces synchronous writeback when dirty pages reach 15% of memory.\n# Prevents massive I\/O flushing pauses on fast NVMe drives.\nvm.dirty_ratio = 15\n\n# 6. NUMA Zone Reclaim Mode\n# Disable zone reclaim to allow cross-node memory allocation over synchronous node reclaim.\nvm.zone_reclaim_mode = 0\n\n# 7. Memory Compaction Threshold\n# Trigger proactive memory compaction before fragmentation triggers allocstalls.\nvm.extfrag_threshold = 500\nvm.compact_unevictable_allowed = 1<\/code><\/pre>\n
      Apply the configuration immediately without requiring a system reboot using the following command:<\/p>\n
      sudo sysctl --system<\/code><\/pre>\n
      Real-Time Observability and Reclamation Diagnostics<\/h2>\n
      Tuning the Linux memory subsystem is not a fire-and-forget exercise. Systems architects must validate tuning effectiveness by interrogating kernel telemetry interfaces under peak production workloads. Three primary sources provide deep visibility into memory reclamation behavior:<\/p>\n
      1. Inspecting Zone Watermarks via \/proc\/zoneinfo<\/h3>\n
      Examine the exact page thresholds calculated by the kernel for each zone:<\/p>\n
      awk '\/Node\/ {node=$2} \/zone\/ {zone=$2} \/min\/ {min=$2} \/low\/ {low=$2} \/high\/ {high=$2} \/spanned\/ {print node, zone, \"min:\"min, \"low:\"low, \"high:\"high}' \/proc\/zoneinfo<\/code><\/pre>\n
      Verify that the delta between min<\/code> and high<\/code> reflects your tuned watermark_scale_factor<\/code>. On properly tuned systems, low<\/code> should be significantly higher than min<\/code>, giving kswapd<\/code> ample runway to operate.<\/p>\n
      2. Tracking Direct Reclaim Stalls (allocstall) in \/proc\/vmstat<\/h3>\n
      Direct reclamation events are tallied globally and per-zone in \/proc\/vmstat<\/code>. Use this command to track real-time stall increments:<\/p>\n
      watch -n 1 \"grep -E 'allocstall|pgscan_kswapd|pgscan_direct|compact_stall' \/proc\/vmstat\"<\/code><\/pre>\n