{"id":4636,"date":"2026-09-20T12:01:15","date_gmt":"2026-09-20T06:31:15","guid":{"rendered":"https:\/\/cpanelfree.com\/blog\/proxmox-ve-82-cluster-architecture-with-ceph-nvme-hyperconverged-storage\/"},"modified":"2026-09-20T12:01:15","modified_gmt":"2026-09-20T06:31:15","slug":"proxmox-ve-82-cluster-architecture-with-ceph-nvme-hyperconverged-storage","status":"publish","type":"post","link":"https:\/\/cpanelfree.com\/blog\/proxmox-ve-82-cluster-architecture-with-ceph-nvme-hyperconverged-storage\/","title":{"rendered":"Proxmox VE 8.2 Cluster Architecture with Ceph NVMe Hyperconverged Storage"},"content":{"rendered":"

Designing resilient, high-throughput virtualization infrastructure requires eliminating traditional storage area network (SAN) chokepoints without introducing volatile single points of failure. In high-density multi-tenant cloud platforms such as those deployed at CpanelFree<\/a>, running Proxmox VE 8.2 coupled with Ceph Reef on enterprise PCIe 4.0\/5.0 NVMe drives unlocks raw metal I\/O performance alongside automated hyperconverged failover. By unifying compute (KVM\/LXC) and distributed object storage (Ceph OSDs) within identical server nodes, infrastructure teams achieve sub-millisecond p99 write latency, deterministic split-brain prevention, and linear horizontal scalability on commodity enterprise server hardware.<\/p>\n

<\/p>\n

Hyperconverged Architecture: Proxmox VE 8.2 with Ceph NVMe Explained<\/h2>\n
\nDirect Answer:<\/strong> Proxmox VE 8.2 with Ceph NVMe hyperconverged storage combines KVM\/LXC virtualization and distributed Ceph OSDs onto the same physical compute nodes. Operating across dedicated 25\/100GbE low-latency networks, it delivers software-defined distributed block storage (Ceph RBD) directly to virtual machines, eliminating expensive external SAN arrays while enabling automatic quorum voting, continuous data scrubbing, and zero-downtime live migration.\n<\/div>\n

Traditional data center designs separate compute hypervisors from external SAN\/NAS storage arrays connected over Fibre Channel or iSCSI protocols. While this isolation simplifies capacity allocation, it introduces massive network traversal overhead, expensive multi-tier controller licensing, and rigid storage expansion boundaries. In contrast, hyperconverged infrastructure (HCI) co-locates virtual machines and storage daemons on every node. Proxmox VE 8.2, based on Debian 12 (Bookworm) and running Linux Kernel 6.8 with Ceph 18.2 (Reef), optimizes this paradigm by eliminating intermediate translation layers and utilizing direct kernel-bypass or asynchronous polling for NVMe I\/O paths.<\/p>\n

Ceph distributes data across storage nodes via the CRUSH (Controlled Replication Under Scalable Hashing) algorithm. Rather than relying on centralized lookup tables, CRUSH deterministically computes the physical OSD locations for every object replica. When paired with high-end enterprise NVMe media (such as Kioxia CM7, Samsung PM1733, or Micron 7450 PRO) and dual-redundant 25GbE or 100GbE networks, hyperconverged Proxmox clusters achieve million-IOPS aggregates while sustaining full tolerance against physical host and disk failures.<\/p>\n

Architectural Comparison: Default vs. Tuned Production Deployments<\/h2>\n

Deploying Proxmox VE and Ceph with out-of-the-box defaults typically yields acceptable performance for spinning rust (HDDs) or legacy SATA SSDs, but it severely bottlenecks enterprise NVMe drives. NVMe devices operate with massive parallelism (up to 64,000 queues with 64,000 commands per queue), whereas default Linux kernel network stacks, Corosync cluster rings, and Ceph BlueStore allocators are tuned conservatively for generic hardware.<\/p>\n\n\n\n\n\n\n\n\n\n\n
Feature \/ Metric<\/th>\nStandard \/ Default<\/th>\nTuned \/ Production<\/th>\n<\/tr>\n<\/thead>\n
Network Topology & MTU<\/td>\nSingle 10GbE Shared NIC (MTU 1500)<\/td>\nDual 25\/100GbE LACP Bond \/ Full-Mesh (MTU 9000)<\/td>\n<\/tr>\n
Corosync Cluster Ring<\/td>\nSingle ring shared with VM traffic<\/td>\nDual redundant rings on dedicated latency-isolated links<\/td>\n<\/tr>\n
Ceph Messenger Protocol<\/td>\nmsgr1 (v1 unencrypted, single-thread)<\/td>\nmsgr2 (v2 async frame, multi-core worker threads)<\/td>\n<\/tr>\n
4K Random Write Latency (p99)<\/td>\n4.8 ms (Kernel buffer & lock contention)<\/td>\n0.62 ms (Direct I\/O, polled completion, NVMe scheduler: none)<\/td>\n<\/tr>\n
BlueStore Memory Target<\/td>\n4096 MB Default per OSD<\/td>\n8192 MB – 16384 MB dynamic cache per OSD<\/td>\n<\/tr>\n
I\/O Scheduling Engine<\/td>\nmq-deadline \/ kyber<\/td>\nnone (Direct hardware queue dispatching)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\nArchitecture Note:<\/strong> When designing Ceph NVMe clusters, never use hardware RAID controllers in front of NVMe drives. Ceph BlueStore requires direct, unobstructed access to the raw NVMe namespaces to manage block allocations, WAL (Write-Ahead Log), and RocksDB metadata natively. Hardware RAID controllers introduce command serializations, cache flush locks, and blind spots during disk telemetry monitoring.\n<\/div>\n

High-Performance Network Topology: Mesh vs. Dual Switched<\/h2>\n

In a Proxmox VE Ceph cluster, network latency directly dictates synchronous write latency. When a VM issues a write to a Ceph RBD volume with a replication size of 3 (`size = 3, min_size = 2`), the primary OSD must replicate that payload across the network to two secondary OSDs and await cryptographic write confirmations before signaling write completion to the client hypervisor.<\/p>\n

Two primary network architectures are deployed in enterprise environments:<\/p>\n

1. Dual 25G\/100G Switched Backbone<\/h3>\n

Recommended for clusters with four or more nodes. Two leaf switches configured with MLAG (Multi-Chassis Link Aggregation) or EVPN-VXLAN provide dual active-active links to each server node via Linux 802.3ad bonding. Network traffic is strictly partitioned into distinct VLANs:<\/p>\n

    \n
  • VLAN 10 (Corosync Ring 0):<\/strong> Cluster heartbeats, quorum votes, and state machine transitions. Prioritized with 802.1p CoS 5 (Voice\/Realtime).<\/li>\n
  • VLAN 20 (Corosync Ring 1):<\/strong> Secondary failover heartbeat path routed through an alternate physical NIC and switch.<\/li>\n
  • VLAN 30 (Ceph Public Network):<\/strong> Handles hypervisor-to-OSD client communications, VM disk I\/O, and status inquiries.<\/li>\n
  • VLAN 40 (Ceph Cluster\/Private Network):<\/strong> Dedicated exclusively to inter-OSD data replication, dynamic PG rebalancing, and deep background scrubbing.<\/li>\n
  • VLAN 50 (VM Guest & Management Traffic):<\/strong> Outbound public internet access, private VLAN bridge interfaces, and Web GUI connectivity.<\/li>\n<\/ul>\n

    2. Full-Mesh Broadcast Topology (3-Node Hyperconverged)<\/h3>\n

    For high-performance 3-node edge deployments, dedicated switches can be bypassed entirely. By installing dual-port 25GbE\/100GbE Mellanox ConnectX-5 or ConnectX-6 NICs in each node, direct DAC (Direct Attach Copper) cables link Node 1 to Node 2, Node 2 to Node 3, and Node 3 to Node 1. Using routed point-to-point subnets with broadcast\/static routes or a routed BGP\/OSPF mesh, nodes exchange Ceph replication traffic at wire speed with sub-microsecond switchless transit times.<\/p>\n

    Production Configuration Files<\/h2>\n

    Below are complete, validated production configuration files extracted from an enterprise Proxmox VE 8.2 cluster running Ceph Reef over dual 25GbE Mellanox NICs.<\/p>\n

    1. Network Interface Configuration (`\/etc\/network\/interfaces`)<\/h3>\n

    This configuration defines an 802.3ad LACP bonded interface with MTU 9000 (Jumbo Frames) for Ceph storage networks, alongside dedicated VLANs for Corosync and guest VM bridges.<\/p>\n

    # \/etc\/network\/interfaces - Production Proxmox VE 8.2 Ceph NVMe Cluster\nauto lo\niface lo inet loopback\n\n# Physical Management & Corosync Ring 0 (1GbE\/10GbE)\niface eno1 inet manual\n\n# High-Speed 25GbE SFP28 Interfaces for Ceph Storage\niface enp129s0f0np0 inet manual\niface enp129s0f1np1 inet manual\n\n# Corosync Ring 0 Dedicated Network\nauto eno1.10\niface eno1.10 inet static\n    address 10.10.10.11\/24\n    vlan-raw-device eno1\n    comment \"Corosync Ring 0 Heartbeat\"\n\n# LACP Bond for Ceph Storage (Public + Cluster)\nauto bond0\niface bond0 inet manual\n    bond-slaves enp129s0f0np0 enp129s0f1np1\n    bond-miimon 100\n    bond-mode 802.3ad\n    bond-xmit-hash-policy layer2+3\n    mtu 9000\n\n# Ceph Public Network (Client VM I\/O)\nauto bond0.30\niface bond0.30 inet static\n    address 192.168.30.11\/24\n    vlan-raw-device bond0\n    mtu 9000\n    comment \"Ceph Public Network\"\n\n# Ceph Cluster Network (OSD Replication & Recovery)\nauto bond0.40\niface bond0.40 inet static\n    address 192.168.40.11\/24\n    vlan-raw-device bond0\n    mtu 9000\n    comment \"Ceph Private Replication Network\"\n\n# Corosync Ring 1 (Redundant Heartbeat over 25G Bond)\nauto bond0.20\niface bond0.20 inet static\n    address 10.10.20.11\/24\n    vlan-raw-device bond0\n    mtu 1500\n    comment \"Corosync Ring 1 Heartbeat\"\n\n# VM Bridge Interface for Guest Traffic\nauto vmbr0\niface vmbr0 inet static\n    address 172.16.10.11\/24\n    gateway 172.16.10.1\n    bridge-ports eno1\n    bridge-stp off\n    bridge-fd 0\n    comment \"Management and Public VM Traffic\"<\/code><\/pre>\n
    2. Ceph Configuration File (`\/etc\/pve\/ceph.conf`)<\/h3>\n
    Ceph Reef introduces critical improvements to the BlueStore engine and async messenger. The following parameters configure high-throughput thread pools, BlueStore NVMe cache limits, and prevent OSD thrashing during transient network blips.<\/p>\n
    [global]\n    auth_client_required = cephx\n    auth_cluster_required = cephx\n    auth_service_required = cephx\n    cluster_network = 192.168.40.0\/24\n    public_network = 192.168.30.0\/24\n    fsid = 8a67ef84-48cd-47d3-9bc6-32d8478f7e1b\n    mon_host = 192.168.30.11 192.168.30.12 192.168.30.13\n    ms_bind_msgr1 = false\n    ms_bind_msgr2 = true\n    ms_async_op_threads = 6\n\n[osd]\n    osd_memory_target = 8589934592\n    osd_op_num_threads_per_shard = 2\n    osd_op_num_shards = 8\n    bdev_flock_retry = 10\n    bdev_aio_poll_ms = 250\n    bluestore_cache_autotune = true\n    bluestore_cache_meta_ratio = 0.45\n    bluestore_cache_kv_ratio = 0.45\n    bluestore_cache_data_ratio = 0.10\n    bluestore_min_alloc_size = 4096\n    bluestore_prefer_deferred_size = 0\n    bluestore_compression_mode = none\n\n[mon]\n    mon_warn_on_pool_no_redundancy = true\n    mon_pg_warn_max_per_osd = 400\n    mon_osd_down_out_interval = 600\n    mon_osd_min_down_reporters = 2<\/code><\/pre>\n
    \nPerformance Optimization:<\/strong> Setting bluestore_min_alloc_size = 4096<\/code> aligns BlueStore allocation granules precisely with the 4KB native sectors of enterprise NVMe devices. This completely prevents write amplification on small random 4K write workloads, reducing SSD wear while doubling random write IOPS.\n<\/div>\n
    3. Kernel & Network Stack Tuning (`\/etc\/sysctl.d\/99-ceph-nvme-performance.conf`)<\/h3>\n
    To eliminate network queue drops at 25GbE\/100GbE line rates and optimize Linux kernel memory management for hyperconverged nodes, apply the following sysctl parameters:<\/p>\n
    # \/etc\/sysctl.d\/99-ceph-nvme-performance.conf\n# Increase network receive\/transmit queue lengths\nnet.core.netdev_max_backlog = 250000\nnet.core.somaxconn = 65535\n\n# Maximize socket buffer sizes for 25G\/100G line rate\nnet.core.rmem_default = 33554432\nnet.core.wmem_default = 33554432\nnet.core.rmem_max = 67108864\nnet.core.wmem_max = 67108864\nnet.ipv4.tcp_rmem = 4096 87380 67108864\nnet.ipv4.tcp_wmem = 4096 65536 67108864\n\n# TCP Congestion Control and Low Latency\nnet.ipv4.tcp_congestion_control = bbr\nnet.ipv4.tcp_slow_start_after_idle = 0\nnet.ipv4.tcp_notsent_lowat = 16384\nnet.ipv4.tcp_timestamps = 1\nnet.ipv4.tcp_sack = 1\n\n# Memory and Swappiness Settings for Hyperconverged Node\nvm.swappiness = 10\nvm.vfs_cache_pressure = 50\nvm.dirty_background_ratio = 5\nvm.dirty_ratio = 10\nvm.min_free_kbytes = 1048576\n\n# Increase ARP table cache for large VM footprints\nnet.ipv4.neigh.default.gc_thresh1 = 4096\nnet.ipv4.neigh.default.gc_thresh2 = 8192\nnet.ipv4.neigh.default.gc_thresh3 = 16384<\/code><\/pre>\n
    4. NVMe I\/O Scheduler Udev Rule (`\/etc\/udev\/rules.d\/60-nvme-scheduler.rules`)<\/h3>\n
    By default, the Linux kernel may assign the `mq-deadline` or `kyber` scheduler to block devices. For enterprise NVMe devices, hardware queue management is orders of magnitude faster than kernel-level reordering. Setting the scheduler to `none` allows Ceph OSD worker threads to submit commands directly to the NVMe controller.<\/p>\n
    # \/etc\/udev\/rules.d\/60-nvme-scheduler.rules\n# Set I\/O scheduler to 'none' for all NVMe block devices\nACTION==\"add|change\", KERNEL==\"nvme[0-9]*n[0-9]*\", ATTR{queue\/scheduler}=\"none\"\n\n# Optimize NVMe read-ahead and request queue depth\nACTION==\"add|change\", KERNEL==\"nvme[0-9]*n[0-9]*\", ATTR{queue\/read_ahead_kb}=\"128\"\nACTION==\"add|change\", KERNEL==\"nvme[0-9]*n[0-9]*\", ATTR{queue\/nr_requests}=\"1024\"<\/code><\/pre>\n
    Validating Performance: Benchmarks and Telemetry<\/h2>\n
    Once your cluster is provisioned and tuned, validation must occur at both the storage pool layer (Ceph RADOS) and the guest hypervisor layer (KVM via `fio`).<\/p>\n
    RADOS Benchmark Execution<\/h3>\n
    Run a direct RADOS benchmark across the cluster to verify network throughput, OSD responsiveness, and replication efficiency without filesystem or VM overhead:<\/p>\n
    # Run 4MB sequential write benchmark for 60 seconds\nrados bench -p nvme-pool 60 write --no-cleanup\n\n# Run 4K random read benchmark with 32 concurrent threads\nrados bench -p nvme-pool 60 rand -t 32\n\n# Clean up benchmark objects\nrados -p nvme-pool cleanup<\/code><\/pre>\n
    FIO Guest Benchmark<\/h3>\n
    Inside a production test VM configured with a VirtIO-SCSI single controller and an enterprise NVMe-backed RBD disk with `io_uring` enabled, execute an asynchronous random write workload:<\/p>\n
    # \/root\/fio-nvme-test.job\n[global]\nioengine=io_uring\ndirect=1\nruntime=60\ntime_based=1\ngroup_reporting=1\nfilename=\/dev\/vdb\n\n[random-write-4k]\nbs=4k\nrw=randwrite\niodepth=64\nnumjobs=4<\/code><\/pre>\n
    A properly tuned 3-node Proxmox VE 8.2 cluster with three enterprise NVMe drives per node (9 OSDs total) over dual 25GbE should consistently yield over 220,000 random 4K write IOPS and sub-800 microsecond p99 latency, even with 3x replication enabled.<\/p>\n
    Frequently Asked Questions<\/h2>\n
    \nCan I run a stable Ceph cluster with only 3 Proxmox VE nodes?<\/summary>\n
    Yes, a 3-node configuration is the minimum recommended production deployment for Ceph. With three nodes running Ceph Monitors and Managers, the cluster maintains strict majority quorum (2 of 3 votes). When using a standard replication rule of size = 3, min_size = 2<\/code>, the cluster can tolerate a complete node failure without downtime or data unavailability. However, during node maintenance, the cluster operates in a degraded state until the node is restored.<\/p>\n<\/details>\n
    \nWhy is Corosync network separation critical in Ceph hyperconverged setups?<\/summary>\n
    Corosync handles cluster membership and heartbeats. If Ceph replication traffic saturates a shared network interface, Corosync packets may experience jitter exceeding 200ms. When heartbeats are dropped, Proxmox assumes the node has failed, triggers a cluster fence, and abruptly reboots the host via watchdog. Separating Corosync onto dedicated physical links or applying strict 802.1p traffic prioritization completely avoids split-brain fencing.<\/p>\n<\/details>\n
    \nShould I enable Ceph BlueStore compression on NVMe pools?<\/summary>\n
    In almost all NVMe workloads, Ceph BlueStore compression should be disabled (bluestore_compression_mode = none<\/code>). While LZ4 or Snappy compression saves disk space, the CPU compression and decompression cycles introduce noticeable latency penalties on high-speed NVMe drives. If storage efficiency is paramount, restrict compression only to cold, sequentially-read bulk storage pools.<\/p>\n<\/details>\n
    \nWhat is the advantage of VirtIO-SCSI single controller with io_uring in Proxmox?<\/summary>\n
    Selecting VirtIO SCSI Single<\/code> with IO Thread<\/code> enabled and setting the async I\/O engine to io_uring<\/code> assigns a dedicated QEMU I\/O thread and submission\/completion ring buffer to each virtual disk. This bypasses the traditional QEMU Big Lock (BQL), enabling the virtual machine to saturate host NVMe queues and achieve native bare-metal IOPS performance.<\/p>\n<\/details>\n
    \n
    Ready to Deploy High-Performance Infrastructure?<\/h3>\n
    Experience blazing-fast NVMe storage, unmetered bandwidth, and enterprise LiteSpeed caching on CpanelFree.<\/p>\n
      Get Started with Free Cloud Hosting →<\/a>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
    Master Proxmox VE 8.2 cluster design with hyperconverged Ceph NVMe storage. Implement low-latency mesh networking, production sysctl tuning, and failover.<\/p>\n","protected":false},"author":1,"featured_media":4635,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[179],"tags":[57,180,177,87,101],"class_list":["post-4636","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-architecture","tag-almalinux","tag-cloud-architecture","tag-databases-performance","tag-devops","tag-sysadmin"],"_links":{"self":[{"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/posts\/4636","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/comments?post=4636"}],"version-history":[{"count":0,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/posts\/4636\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/media\/4635"}],"wp:attachment":[{"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/media?parent=4636"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/categories?post=4636"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cpanelfree.com\/blog\/wp-json\/wp\/v2\/tags?post=4636"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}