How to Monitor NVIDIA GPU Temperature, Power and VRAM with Prometheus and Grafana

Operating enterprise LLM inference clusters and high-concurrency CUDA microservices requires granular, millisecond-accurate hardware telemetry to prevent silent thermal throttling, power cap degradation, and catastrophic Out-Of-Memory (OOM) faults. Without automated instrumentation, transient memory spikes across Tensor Cores degrade inference latencies before standard Linux kernel watchdogs can alert operations teams. At CpanelFree, our bare-metal infrastructure engineers leverage deep Prometheus time-series scraping and Grafana dashboards to enforce real-time visibility across high-density GPU clusters.

Enterprise NVIDIA GPU Observability with Prometheus & Grafana

Why Legacy NVML Polling Fails in High-Density AI Workloads

Modern Linux servers hosting accelerated computing workloads—such as model fine-tuning with PyTorch, distributed inference via vLLM or Triton Inference Server, and generative image rendering pipelines—place unprecedented stress on GPU silicon. Traditional administrative scripts historically relied on executing nvidia-smi --query-gpu=... --format=csv inside crontabs or lightweight Bash daemons. While functional on single-workstation setups, this legacy approach breaks down catastrophically in multi-GPU production environments.

Every execution of nvidia-smi invokes a new userspace process, dynamically links against the NVIDIA Management Library (NVML), opens file descriptors against the character devices in /dev/nvidia*, and locks the NVML mutex inside the kernel module. When eight GPUs are operating under 100% compute saturation with concurrent CUDA kernels running, this mutex lock introduces kernel thread contention. In extreme conditions, rapid polling via nvidia-smi causes the monitoring process to hang, consumes significant CPU cycles, and can even induce driver context timeouts. Furthermore, shell-based polling cannot provide millisecond-scale visibility into transient thermal spikes, clock throttling events, or PCIe replay counters that degrade tensor throughput.

To eliminate this bottleneck, NVIDIA engineered the Data Center GPU Manager (DCGM). Operating as an asynchronous, low-overhead daemon, DCGM interfaces directly with the NVIDIA kernel driver via shared memory ring buffers. It samples hardware telemetry at user-defined microsecond intervals without taking blocking driver locks. By coupling DCGM with the open-source dcgm-exporter, systems engineers can expose standard Prometheus metrics over an HTTP endpoint with near-zero CPU overhead.

Pre-Flight System Hardening: Driver Persistence and Systemd Setup

Before deploying the Prometheus exporter, the Linux operating system must be tuned to ensure the NVIDIA kernel modules remain permanently loaded and initialized. On headless Linux servers (such as Ubuntu 22.04 LTS or Debian 12), the Linux kernel will aggressively unload the nvidia and nvidia-uvm drivers whenever all active CUDA processes terminate. When Prometheus attempts to scrape metrics from an idle GPU whose driver has unloaded, the scrape request blocks while the kernel reloads the driver, causing high scrape latency and inaccurate time-series data.

To prevent this, configure the NVIDIA Persistence Daemon to run automatically on system boot. Create a systemd drop-in override for the persistence service:

# /etc/systemd/system/nvidia-persistenced.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/nvidia-persistenced --user=nvidia-persistenced --no-persistence-mode --verbose
Restart=always
RestartSec=5s

Reload systemd and verify the persistence daemon status across all installed physical accelerators:

# Enable and start persistence service
sudo systemctl daemon-reload
sudo systemctl enable --now nvidia-persistenced

# Verify persistence mode is active on all GPUs (Persistence-M: Enabled)
nvidia-smi -q | grep -i "persistence mode"

Deploying NVIDIA DCGM Exporter as an Enterprise Systemd Service

The dcgm-exporter binary can be deployed either via a lightweight OCI container using Docker/Podman or natively as a standalone systemd binary. For bare-metal infrastructure where container engine dependencies are minimized, running dcgm-exporter natively under systemd provides maximum determinism and security sandboxing.

Below is the production-hardened systemd unit file for dcgm-exporter, configured to bind strictly to internal management networks and drop unnecessary Linux capabilities:

# /etc/systemd/system/dcgm-exporter.service
[Unit]
Description=NVIDIA DCGM Exporter for Prometheus Telemetry
After=network-online.target nvidia-persistenced.service
Wants=network-online.target nvidia-persistenced.service

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/etc/dcgm-exporter
ExecStart=/usr/bin/dcgm-exporter     -f /etc/dcgm-exporter/custom-counters.csv     -a 0.0.0.0:9400     -c 5000     -d 10000     -r 127.0.0.1:5555
Restart=on-failure
RestartSec=10s
LimitNOFILE=65536
CapabilityBoundingSet=CAP_SYS_ADMIN
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log /run

[Install]
WantedBy=multi-user.target

Configuring Granular Telemetry Counters (custom-counters.csv)

By default, dcgm-exporter exposes a minimal subset of metrics. In production AI/ML clusters, you must capture granular telemetry including junction temperatures, power limits, PCIe link replay rates, and memory copy utilization. This is controlled via a custom CSV configuration file that maps DCGM Field IDs directly to Prometheus metric identifiers:

# /etc/dcgm-exporter/custom-counters.csv
# Format: Field ID, Metric Name, Metric Type, Metric Help
DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_GPU_TEMP, gauge, Core GPU temperature in Celsius.
DCGM_FI_DEV_MEMORY_TEMP, DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory junction temperature in Celsius.
DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_POWER_USAGE, gauge, Real-time electrical power draw in Watts.
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Cumulative energy consumption in millijoules.
DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_USED, gauge, Framebuffer VRAM memory used in Megabytes.
DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_FB_FREE, gauge, Framebuffer VRAM memory free in Megabytes.
DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_GPU_UTIL, gauge, Streaming Multiprocessor (SM) compute utilization percentage.
DCGM_FI_DEV_MEM_COPY_UTIL, DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory copy engine utilization percentage.
DCGM_FI_DEV_SM_CLOCK, DCGM_FI_DEV_SM_CLOCK, gauge, Current Streaming Multiprocessor clock frequency in MHz.
DCGM_FI_DEV_MEM_CLOCK, DCGM_FI_DEV_MEM_CLOCK, gauge, Current memory clock frequency in MHz.
DCGM_FI_DEV_PCIE_REPLAY_COUNTER, DCGM_FI_DEV_PCIE_REPLAY_COUNTER, counter, Total PCIe link transmission replay retries.
DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_XID_ERRORS, gauge, Value of the last critical driver Xid error code.
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles where Tensor Cores were active.

Integrating with Prometheus Server Scrape Pipelines

Once dcgm-exporter is listening on port 9400, configure your central Prometheus server to scrape the endpoint. Because hardware thermal events can escalate in seconds under unconstrained matrix multiplication loads, configure a dedicated 5-second scrape interval for the GPU fleet:

# /etc/prometheus/prometheus.yml (Snippet)
scrape_configs:
  - job_name: "nvidia-dcgm"
    scrape_interval: 5s
    scrape_timeout: 4s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets:
          - "gpu-node-01.infra.internal:9400"
          - "gpu-node-02.infra.internal:9400"
        labels:
          cluster: "production-ai-inference"
          region: "datacenter-west"
    relabel_configs:
      - source_labels: [__address__]
        regex: "([^:]+):.*"
        target_label: instance
        replacement: "${1}"

Production Prometheus Alerting Rules for Thermal, VRAM, and Power Anomalies

Observability without actionable alerting creates operational blind spots. Below is an enterprise alert rule file defining automated alerts for critical thermal thresholds, runaway memory leaks, and PCIe hardware degradation:

# /etc/prometheus/rules/nvidia-gpu-alerts.yml
groups:
  - name: nvidia_gpu_hardware_alerts
    rules:
      - alert: GPUHighTemperatureWarning
        expr: DCGM_FI_DEV_GPU_TEMP > 80
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "GPU high temperature warning on {{ $labels.instance }} GPU {{ $labels.gpu }}"
          description: "Core temperature on GPU {{ $labels.gpu }} has exceeded 80C for over 2 minutes (Current: {{ $value }}C)."

      - alert: GPUCriticalThermalThrottling
        expr: DCGM_FI_DEV_GPU_TEMP >= 88
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "GPU thermal throttling imminent on {{ $labels.instance }} GPU {{ $labels.gpu }}"
          description: "GPU core temperature reached {{ $value }}C. Hardware downclocking is actively degrading tensor compute throughput."

      - alert: GPUVRAMExhaustionRisk
        expr: (DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)) * 100 > 94
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "VRAM allocation exceeding 94% on {{ $labels.instance }} GPU {{ $labels.gpu }}"
          description: "GPU {{ $labels.gpu }} has sustained 94% VRAM utilization for 3 minutes. Immediate risk of CUDA Out-Of-Memory (OOM) abort."

      - alert: GPUPCIeReplayRateElevated
        expr: rate(DCGM_FI_DEV_PCIE_REPLAY_COUNTER[2m]) > 5
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "PCIe replay errors detected on {{ $labels.instance }} GPU {{ $labels.gpu }}"
          description: "PCIe replay rate is {{ $value }} errors/sec. Potential physical PCIe riser degradation or signal integrity faults."

Grafana Dashboard Construction & Essential PromQL Queries

With Prometheus aggregating time-series data, you can build Grafana dashboards that present executive-level summaries alongside micro-level hardware diagnostics. Community Dashboard 12239 provides an excellent starting framework, but production teams should configure dedicated panels utilizing the following tuned PromQL queries:

• VRAM Memory Utilization Percentage:
  (DCGM_FI_DEV_FB_USED{instance=~"$instance"} / (DCGM_FI_DEV_FB_USED{instance=~"$instance"} + DCGM_FI_DEV_FB_FREE{instance=~"$instance"})) * 100
  Provides an instantaneous allocation ratio across individual GPU indices, independent of differing total memory configurations (e.g. 24GB RTX 4090 vs 80GB H100).
• Dynamic Power Draw Relative to TDP Limit:
  (DCGM_FI_DEV_POWER_USAGE{instance=~"$instance"} / DCGM_FI_DEV_POWER_LIMIT{instance=~"$instance"}) * 100
  Tracks whether GPUs are hitting power capping caps. Power throttling often masquerades as software pipeline latency when the board VRMs throttle SM clocks.
• Thermal Headroom Margin:
  88 - DCGM_FI_DEV_GPU_TEMP{instance=~"$instance"}
  Displays the degree margin remaining before automatic hardware throttling triggers at 88°C. This allows proactive load balancer re-routing before customer requests stall.
• Tensor Core Compute Saturation:
  DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{instance=~"$instance"} * 100
  Measures whether the underlying model architecture is genuinely exploiting mixed-precision FP16/BF16 matrix multiplication units or stalling on host memory transfers.

Infrastructure Scaling & Bare-Metal Architectural Foundations

While monitoring and telemetry give you real-time visibility into GPU thermal dynamics and memory bottlenecks, the underlying host architecture dictates your operational ceiling. High-performance machine learning inference servers, fast vector search indexes, and real-time API web heads require rock-solid upstream servers that never choke on disk I/O or network throughput.

When provisioning dedicated infrastructure for production databases, reverse proxy clusters, or staging pipelines, raw hardware reliability and transparent hosting economics are paramount. For production systems where stability cannot be compromised by hypervisor oversubscription or surprise renewal bills, MeraHost Enterprise Cloud delivers enterprise NVMe arrays, optimized LiteSpeed Web Server stacks, and a contractual Same Renewal Price, Always guarantee starting at ₹99/mo.

Frequently Asked Questions