{"id":4817,"date":"2026-09-24T01:02:27","date_gmt":"2026-09-23T19:32:27","guid":{"rendered":"https:\/\/cpanelfree.com\/blog\/how-to-monitor-nvidia-gpu-temperature-power-and-vram-with-prometheus-and-grafana\/"},"modified":"2026-09-24T01:02:27","modified_gmt":"2026-09-23T19:32:27","slug":"how-to-monitor-nvidia-gpu-temperature-power-and-vram-with-prometheus-and-grafana","status":"publish","type":"post","link":"https:\/\/cpanelfree.com\/blog\/how-to-monitor-nvidia-gpu-temperature-power-and-vram-with-prometheus-and-grafana\/","title":{"rendered":"How to Monitor NVIDIA GPU Temperature, Power and VRAM with Prometheus and Grafana"},"content":{"rendered":"

Operating enterprise LLM inference clusters and high-concurrency CUDA microservices requires granular, millisecond-accurate hardware telemetry to prevent silent thermal throttling, power cap degradation, and catastrophic Out-Of-Memory (OOM) faults. Without automated instrumentation, transient memory spikes across Tensor Cores degrade inference latencies before standard Linux kernel watchdogs can alert operations teams. At CpanelFree<\/a>, our bare-metal infrastructure engineers leverage deep Prometheus time-series scraping and Grafana dashboards to enforce real-time visibility across high-density GPU clusters.<\/p>\n

<\/p>\n

Enterprise NVIDIA GPU Observability with Prometheus & Grafana<\/h2>\n
Quick Answer:<\/strong> To monitor NVIDIA GPU metrics on Linux, install NVIDIA Data Center GPU Manager (DCGM) along with dcgm-exporter<\/code> to expose hardware counters via HTTP on port 9400. Configure Prometheus to scrape these metrics over TLS, and import Grafana Dashboard ID 12239 for instant visibility into thermal headroom, power cap throttling, and VRAM allocations.<\/div>\n
Architecture Note:<\/strong> NVIDIA Persistence Mode (nvidia-smi -pm 1<\/code> or the nvidia-persistenced<\/code> daemon) maintains initialized device descriptors in kernel memory even when no CUDA workloads are active. Without persistence mode enabled, Prometheus scraping incurs up to 150ms of driver spin-up latency on idle devices, leading to spurious scrape timeout errors and jitter in Grafana graphs.<\/div>\n\n\n\n\n\n\n\n\n\n\n
Feature \/ Metric<\/th>\nStandard \/ Default<\/th>\nTuned \/ Production<\/th>\n<\/tr>\n<\/thead>\n
Latency \/ Overhead<\/td>\nBaseline (800ms – 2500ms shell fork)<\/td>\nOptimal (< 12ms direct memory channel)<\/td>\n<\/tr>\n
Sampling Resolution<\/td>\n30s to 60s coarse intervals<\/td>\n1s to 5s continuous streaming<\/td>\n<\/tr>\n
Kernel Lock Contention<\/td>\nHigh (NVML mutex blocking under load)<\/td>\nZero (Asynchronous DCGM telemetry buffer)<\/td>\n<\/tr>\n
Hardware Metric Scope<\/td>\nBasic temp, VRAM, and power<\/td>\nTensor Core active cycles, PCIe replay, NVLink<\/td>\n<\/tr>\n
MIG Partition Support<\/td>\nUnstructured text parsing required<\/td>\nNative Prometheus instance & profile labels<\/td>\n<\/tr>\n
Alerting Pipeline<\/td>\nFragile shell script cron triggers<\/td>\nNative PromQL Alertmanager escalation<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Why Legacy NVML Polling Fails in High-Density AI Workloads<\/h3>\n

Modern Linux servers hosting accelerated computing workloads\u2014such as model fine-tuning with PyTorch, distributed inference via vLLM or Triton Inference Server, and generative image rendering pipelines\u2014place unprecedented stress on GPU silicon. Traditional administrative scripts historically relied on executing nvidia-smi --query-gpu=... --format=csv<\/code> inside crontabs or lightweight Bash daemons. While functional on single-workstation setups, this legacy approach breaks down catastrophically in multi-GPU production environments.<\/p>\n

Every execution of nvidia-smi<\/code> invokes a new userspace process, dynamically links against the NVIDIA Management Library (NVML), opens file descriptors against the character devices in \/dev\/nvidia*<\/code>, and locks the NVML mutex inside the kernel module. When eight GPUs are operating under 100% compute saturation with concurrent CUDA kernels running, this mutex lock introduces kernel thread contention. In extreme conditions, rapid polling via nvidia-smi<\/code> causes the monitoring process to hang, consumes significant CPU cycles, and can even induce driver context timeouts. Furthermore, shell-based polling cannot provide millisecond-scale visibility into transient thermal spikes, clock throttling events, or PCIe replay counters that degrade tensor throughput.<\/p>\n

To eliminate this bottleneck, NVIDIA engineered the Data Center GPU Manager (DCGM). Operating as an asynchronous, low-overhead daemon, DCGM interfaces directly with the NVIDIA kernel driver via shared memory ring buffers. It samples hardware telemetry at user-defined microsecond intervals without taking blocking driver locks. By coupling DCGM with the open-source dcgm-exporter<\/code>, systems engineers can expose standard Prometheus metrics over an HTTP endpoint with near-zero CPU overhead.<\/p>\n

Pre-Flight System Hardening: Driver Persistence and Systemd Setup<\/h3>\n

Before deploying the Prometheus exporter, the Linux operating system must be tuned to ensure the NVIDIA kernel modules remain permanently loaded and initialized. On headless Linux servers (such as Ubuntu 22.04 LTS or Debian 12), the Linux kernel will aggressively unload the nvidia<\/code> and nvidia-uvm<\/code> drivers whenever all active CUDA processes terminate. When Prometheus attempts to scrape metrics from an idle GPU whose driver has unloaded, the scrape request blocks while the kernel reloads the driver, causing high scrape latency and inaccurate time-series data.<\/p>\n

To prevent this, configure the NVIDIA Persistence Daemon to run automatically on system boot. Create a systemd drop-in override for the persistence service:<\/p>\n

# \/etc\/systemd\/system\/nvidia-persistenced.service.d\/override.conf\n[Service]\nExecStart=\nExecStart=\/usr\/bin\/nvidia-persistenced --user=nvidia-persistenced --no-persistence-mode --verbose\nRestart=always\nRestartSec=5s<\/code><\/pre>\n
Reload systemd and verify the persistence daemon status across all installed physical accelerators:<\/p>\n
# Enable and start persistence service\nsudo systemctl daemon-reload\nsudo systemctl enable --now nvidia-persistenced\n\n# Verify persistence mode is active on all GPUs (Persistence-M: Enabled)\nnvidia-smi -q | grep -i \"persistence mode\"<\/code><\/pre>\n
Deploying NVIDIA DCGM Exporter as an Enterprise Systemd Service<\/h3>\n
The dcgm-exporter<\/code> binary can be deployed either via a lightweight OCI container using Docker\/Podman or natively as a standalone systemd binary. For bare-metal infrastructure where container engine dependencies are minimized, running dcgm-exporter<\/code> natively under systemd provides maximum determinism and security sandboxing.<\/p>\n
Below is the production-hardened systemd unit file for dcgm-exporter<\/code>, configured to bind strictly to internal management networks and drop unnecessary Linux capabilities:<\/p>\n
# \/etc\/systemd\/system\/dcgm-exporter.service\n[Unit]\nDescription=NVIDIA DCGM Exporter for Prometheus Telemetry\nAfter=network-online.target nvidia-persistenced.service\nWants=network-online.target nvidia-persistenced.service\n\n[Service]\nType=simple\nUser=root\nGroup=root\nWorkingDirectory=\/etc\/dcgm-exporter\nExecStart=\/usr\/bin\/dcgm-exporter     -f \/etc\/dcgm-exporter\/custom-counters.csv     -a 0.0.0.0:9400     -c 5000     -d 10000     -r 127.0.0.1:5555\nRestart=on-failure\nRestartSec=10s\nLimitNOFILE=65536\nCapabilityBoundingSet=CAP_SYS_ADMIN\nNoNewPrivileges=true\nProtectSystem=strict\nProtectHome=true\nReadWritePaths=\/var\/log \/run\n\n[Install]\nWantedBy=multi-user.target<\/code><\/pre>\n
Configuring Granular Telemetry Counters (custom-counters.csv)<\/h3>\n
By default, dcgm-exporter<\/code> exposes a minimal subset of metrics. In production AI\/ML clusters, you must capture granular telemetry including junction temperatures, power limits, PCIe link replay rates, and memory copy utilization. This is controlled via a custom CSV configuration file that maps DCGM Field IDs directly to Prometheus metric identifiers:<\/p>\n
# \/etc\/dcgm-exporter\/custom-counters.csv\n# Format: Field ID, Metric Name, Metric Type, Metric Help\nDCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_GPU_TEMP, gauge, Core GPU temperature in Celsius.\nDCGM_FI_DEV_MEMORY_TEMP, DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory junction temperature in Celsius.\nDCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_POWER_USAGE, gauge, Real-time electrical power draw in Watts.\nDCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Cumulative energy consumption in millijoules.\nDCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_USED, gauge, Framebuffer VRAM memory used in Megabytes.\nDCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_FB_FREE, gauge, Framebuffer VRAM memory free in Megabytes.\nDCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_GPU_UTIL, gauge, Streaming Multiprocessor (SM) compute utilization percentage.\nDCGM_FI_DEV_MEM_COPY_UTIL, DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory copy engine utilization percentage.\nDCGM_FI_DEV_SM_CLOCK, DCGM_FI_DEV_SM_CLOCK, gauge, Current Streaming Multiprocessor clock frequency in MHz.\nDCGM_FI_DEV_MEM_CLOCK, DCGM_FI_DEV_MEM_CLOCK, gauge, Current memory clock frequency in MHz.\nDCGM_FI_DEV_PCIE_REPLAY_COUNTER, DCGM_FI_DEV_PCIE_REPLAY_COUNTER, counter, Total PCIe link transmission replay retries.\nDCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_XID_ERRORS, gauge, Value of the last critical driver Xid error code.\nDCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles where Tensor Cores were active.<\/code><\/pre>\n
Integrating with Prometheus Server Scrape Pipelines<\/h3>\n
Once dcgm-exporter<\/code> is listening on port 9400, configure your central Prometheus server to scrape the endpoint. Because hardware thermal events can escalate in seconds under unconstrained matrix multiplication loads, configure a dedicated 5-second scrape interval for the GPU fleet:<\/p>\n
# \/etc\/prometheus\/prometheus.yml (Snippet)\nscrape_configs:\n  - job_name: \"nvidia-dcgm\"\n    scrape_interval: 5s\n    scrape_timeout: 4s\n    metrics_path: \/metrics\n    scheme: http\n    static_configs:\n      - targets:\n          - \"gpu-node-01.infra.internal:9400\"\n          - \"gpu-node-02.infra.internal:9400\"\n        labels:\n          cluster: \"production-ai-inference\"\n          region: \"datacenter-west\"\n    relabel_configs:\n      - source_labels: [__address__]\n        regex: \"([^:]+):.*\"\n        target_label: instance\n        replacement: \"${1}\"<\/code><\/pre>\n
Production Prometheus Alerting Rules for Thermal, VRAM, and Power Anomalies<\/h3>\n
Observability without actionable alerting creates operational blind spots. Below is an enterprise alert rule file defining automated alerts for critical thermal thresholds, runaway memory leaks, and PCIe hardware degradation:<\/p>\n
# \/etc\/prometheus\/rules\/nvidia-gpu-alerts.yml\ngroups:\n  - name: nvidia_gpu_hardware_alerts\n    rules:\n      - alert: GPUHighTemperatureWarning\n        expr: DCGM_FI_DEV_GPU_TEMP > 80\n        for: 2m\n        labels:\n          severity: warning\n        annotations:\n          summary: \"GPU high temperature warning on {{ $labels.instance }} GPU {{ $labels.gpu }}\"\n          description: \"Core temperature on GPU {{ $labels.gpu }} has exceeded 80C for over 2 minutes (Current: {{ $value }}C).\"\n\n      - alert: GPUCriticalThermalThrottling\n        expr: DCGM_FI_DEV_GPU_TEMP >= 88\n        for: 30s\n        labels:\n          severity: critical\n        annotations:\n          summary: \"GPU thermal throttling imminent on {{ $labels.instance }} GPU {{ $labels.gpu }}\"\n          description: \"GPU core temperature reached {{ $value }}C. Hardware downclocking is actively degrading tensor compute throughput.\"\n\n      - alert: GPUVRAMExhaustionRisk\n        expr: (DCGM_FI_DEV_FB_USED \/ (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)) * 100 > 94\n        for: 3m\n        labels:\n          severity: warning\n        annotations:\n          summary: \"VRAM allocation exceeding 94% on {{ $labels.instance }} GPU {{ $labels.gpu }}\"\n          description: \"GPU {{ $labels.gpu }} has sustained 94% VRAM utilization for 3 minutes. Immediate risk of CUDA Out-Of-Memory (OOM) abort.\"\n\n      - alert: GPUPCIeReplayRateElevated\n        expr: rate(DCGM_FI_DEV_PCIE_REPLAY_COUNTER[2m]) > 5\n        for: 1m\n        labels:\n          severity: warning\n        annotations:\n          summary: \"PCIe replay errors detected on {{ $labels.instance }} GPU {{ $labels.gpu }}\"\n          description: \"PCIe replay rate is {{ $value }} errors\/sec. Potential physical PCIe riser degradation or signal integrity faults.\"<\/code><\/pre>\n
Grafana Dashboard Construction & Essential PromQL Queries<\/h3>\n
With Prometheus aggregating time-series data, you can build Grafana dashboards that present executive-level summaries alongside micro-level hardware diagnostics. Community Dashboard 12239<\/code> provides an excellent starting framework, but production teams should configure dedicated panels utilizing the following tuned PromQL queries:<\/p>\n