NVIDIA AI Operations (Day-2) Review
Purpose
Review operational posture of NVIDIA GPU fleets against the NCP-AIO body of knowledge: DCGM and DCGM-Exporter coverage, MIG partitioning lifecycle, Xid error classification and runbooks, fleet driver/firmware upgrade orchestration, and GPU-aware incident response.
Lean operating rules
- Prefer live evidence (
dcgmi diag,dcgmi health,nvidia-smi --query-gpu=..., Prometheus DCGM metrics) when the active client exposes it; otherwise fall back to NVIDIA DCGM documentation, sanitized dashboards, and Xid reference tables. - Separate confirmed facts from inference. If DCGM coverage, alerting rules, or Xid classification was not directly queried, say so.
- Treat absence of DCGM exporter on production GPU nodes as a high finding — fleet is operationally blind.
- Treat ungated driver/firmware rolling upgrade (no pre-drain, no canary, no rollback path) as a high finding.
- Treat undocumented Xid signature → runbook mapping as a medium finding — incidents will be triaged from scratch.
- Treat GPU-Operator / device-plugin running with default permissive securityContext on multi-tenant clusters as a high finding.
- Treat MIG re-partitioning performed live without workload drain as a medium finding (running contexts are killed).
- Treat alerting that does not cover ECC double-bit errors, NVLink errors, or row-remapping pending as a medium finding.
Response minimum
Return, at minimum:
- the scoped target (cluster/fleet, exporter version) and evidence level,
- DCGM exporter coverage and alert posture,
- MIG partitioning lifecycle posture,
- Xid → runbook coverage,
- driver/firmware upgrade discipline,
- safe next actions and assumptions or blockers.