A traditional data center runs a diverse portfolio of workloads. Web servers, databases, batch jobs, internal tools. They are mostly independent, mostly tolerant of a single node disappearing, and mostly measured in availability. If one of a thousand web servers dies, a load balancer routes around it and nobody files an incident.
An AI factory is the opposite shape. It runs a small number of enormous, tightly coupled jobs across thousands of accelerators joined by a high speed fabric, and it is measured in output: tokens produced, or model checkpoints advanced, per unit of power and time. The word factory is doing real work in that phrase. A factory is judged on yield, meaning useful output over input, not on whether the lights are on.
That difference changes everything about how failure behaves. In a training job running synchronous data parallelism across 16,000 GPUs, every GPU waits for the slowest one at every step. A single device running 10 percent slow does not degrade 1 in 16,000 of your throughput. It degrades all of it. The blast radius of one bad component is the entire job.
And the scale makes it worse, because at current sizes components fail constantly. The most useful public number remains Meta's Llama 3 pre training disclosure: over a 54 day snapshot on 16,384 H100 GPUs, they recorded 466 interruptions, of which 419 were unexpected. That is roughly one unexpected failure every three hours. GPU issues including NVLink accounted for 148 of them and HBM3 memory for another 72, so GPU and memory together caused about half. Only three incidents required significant manual intervention, and the team still held above 90 percent effective training time, which is a testament to their automation rather than to the hardware.
So the operating assumption for an AI factory is not "things work and occasionally break." It is "things break continuously and the job must survive it." Yield is what you have left after you pay for all of that.
Power has become the other binding constraint. In the DSX announcements at GTC Berlin the day before Minerva launched, NVIDIA cited over 300 billion dollars of equipment backlogged and over 200 GW of US projects sitting in interconnection queues. When you cannot get more power allocated, the only way to produce more tokens is to waste fewer of the watts you already have. Yield again.
This is the context Minerva was built for.
The Problem: Degradation, Not Failure
A hard failure is the easy case. A GPU falls off the bus, XID 79 fires, your scheduler drains the node, the job restarts from checkpoint. Painful but legible.
The expensive case is the one that never trips a threshold. NVIDIA's own framing names two examples: a GPU running hotter than its peers, and an NVMe drive losing performance under load. Neither is broken. Neither is alerting. Both will quietly slow every step of a job that depends on them, and because the slowdown is distributed through a collective operation, the symptom shows up everywhere except at the cause.
The second problem is organizational. In most operations the data center is split into separate views for separate teams. Networking has fabric telemetry. Compute has GPU metrics. Site reliability has facility and power. The ML team has training loss curves and step times. The failure mode is predictable: the ML team sees step time creep up 8 percent, opens a ticket, and four teams each look at their own dashboard, see nothing abnormal in their own domain, and hand it back.
The answer requires something none of those dashboards have, which is topology. Knowing that these eight GPUs, that SuperNIC, that switch port and that specific cable are all on the path of this specific job is what converts four clean dashboards into one root cause.
What Minerva Is
NVIDIA launched Minerva on October 22, 2026 at GTC Berlin, describing it as an AIOps platform that combines infrastructure and AI workload telemetry and applies purpose built AI models to turn that data into actionable insight.
Working through the pieces.
The dynamic knowledge graph is the centerpiece and the part that distinguishes this from a metrics pipeline. It maps GPUs, SuperNICs, switches, cables and the jobs running on them. Dynamic matters because job placement changes constantly, so the graph edge from "job 4417" to "cable in rack 12" has to be rebuilt as the scheduler moves work around. This is what makes the correlation step possible at all.
The telemetry scope is wide and NVIDIA shaped. Per partner descriptions at launch, it covers GPUs, ConnectX SuperNICs, BlueField DPUs, NVLink switches, Spectrum-X Ethernet, NCCL, schedulers and AI workloads. Note how much of that is networking. This is a networking organization's view of observability, which makes sense given Minerva's product page sits under NVIDIA's networking section.
The ML layer processes what NVIDIA describes as billions of data points per hour using a mix of supervised and unsupervised machine learning. The stated capabilities are learning normal behavior, detecting previously unseen anomalies, and predicting emerging failures. The unsupervised half is the important claim: threshold alerting cannot catch a failure mode nobody has written a rule for, and at this scale novel failure modes are routine.
Correlation and consolidation is where the operational value lands for anyone who has been on call for a large cluster. Minerva correlates related events using factory topology and groups event storms into focused incidents with a likely root cause. One bad component in a fabric generates alerts from every device that notices, and the current state of the art for most teams is a human reading forty pages and guessing.
Sherlock is a built in multi agent assistant that answers natural language questions grounded in live telemetry. Treat the agent framing with normal skepticism, but the underlying capability of querying a topology graph conversationally is genuinely useful for the "which job is affected by this switch" class of question that is tedious to express in PromQL.
An MCP server lets external agentic tools connect to the platform. That is a notable architectural choice: rather than making Minerva the only interface, it exposes itself as a tool other agents can call, which fits the direction the rest of the agent ecosystem has taken this year.
Remediation is recommended rather than forced, and teams can apply fixes manually or automate them through infrastructure APIs. The human in the loop default is correct for anything that drains a node out from under a running job.
Deployment wise, partner descriptions say it can run standalone or integrate with NVIDIA DSX OS, the open source AI factory operations software announced earlier in the year.
AI Factory Yield as the Metric
Minerva is explicitly built around a metric NVIDIA calls AI factory yield, defined as the useful work the infrastructure produces, measured through GPU hours, goodput, cluster uptime and token production.
Goodput is the one worth dwelling on, because it is the metric that makes AI infrastructure different and the one most teams do not track. Throughput is work attempted. Goodput is work that survived to a useful result. A training job that runs for two hours and then restarts from a checkpoint ninety minutes old did an hour and a half of throughput and thirty minutes of goodput. Everything Minerva claims to do, earlier detection, fewer restarts, faster repair, is ultimately a goodput argument.
Being fair about the framing: AI factory yield is a vendor defined composite metric, and a vendor defined composite that a vendor's product is built to improve deserves a raised eyebrow. The underlying components are all real and measurable independently, which is the saving grace. You can hold NVIDIA to goodput and mean time between interruptions without adopting their index.
How It Behaves in an Incident
The scenario Minerva is designed for, as described at launch:
The two numbers partners reported at launch map onto exactly that flow. Bitdeer AI described an NVMe power fault where mean time to repair fell from a multi day outage to six hours, and a case where 81 noisy events were consolidated into a single alert, which they credited with saving 54 hours of cluster uptime.
Those are the most concrete figures available, and they come from a partner's internal testing with no published baseline definitions, sample sizes or test conditions. I would read them as illustrations of the mechanism rather than as benchmarks. The 81 to 1 consolidation ratio is believable to anyone who has watched a fabric incident fill a pager; the 54 hours attributed to it depends entirely on what the counterfactual was.
Comparison: Where This Sits Against What You Already Run
This is the section that matters if you operate GPU infrastructure today, because you are not starting from nothing.
| DCGM Exporter + Prometheus + Grafana | Commercial APM (Datadog, Dynatrace) | Scheduler level tools | NVIDIA Minerva | |
|---|---|---|---|---|
| Collects | Per GPU utilization, memory, temperature, power, ECC, XID errors | Host, container, application metrics and traces | Job queue, allocation, utilization per job | GPUs, SuperNICs, DPUs, NVLink and Spectrum-X fabric, storage, schedulers, NCCL, workloads |
| Topology awareness | None. Metrics are per device, joined by labels at query time | Service dependency maps, not physical fabric | Job to node mapping only | Dynamic graph of devices, cables and the jobs on them |
| Detection model | Static thresholds and rate rules you write | Thresholds plus some anomaly detection | Mostly none | Supervised plus unsupervised ML on learned baselines |
| Novel failure modes | Only if you wrote a rule | Partially | No | Claimed, via unsupervised detection |
| Failure prediction | No | Limited | No | Claimed |
| Event storm handling | Alertmanager grouping by labels you define | Correlation features, varying quality | No | Topology based correlation into one incident |
| Fabric visibility | Not covered by DCGM. Separate tooling needed | Generally weak on InfiniBand and NVLink | No | First class, it is a networking product |
| Goodput as an SLI | You have to build it | You have to build it | Partial | Built into the product framing |
| Cost model | Open source, you run it | Per host or per metric, expensive at GPU scale | Varies | Not disclosed |
| Vendor coupling | NVIDIA GPUs, otherwise portable | Portable | Varies | Deeply NVIDIA specific |
The honest read of the DCGM stack is that it is excellent at what it does and structurally unable to do what Minerva claims. DCGM Exporter runs as a DaemonSet, exposes per GPU metrics on a Prometheus endpoint, and gives you utilization, framebuffer, temperature, power, ECC and XID codes, with XID 79 as the one that means a GPU fell off the bus. That is genuinely the right foundation, and if you do not have it, build it before you evaluate anything else.
What it cannot tell you is why. As one practitioner guide puts it, a utilization drop could be a crashed serving process, a traffic routing issue, or a reloaded model, and the metric alone does not distinguish them. DCGM also does not cover the interconnect, which for multi node training is where a large share of real incidents live. Threshold tuning is left to you, and the standard advice of alerting on change rather than absolute value for utilization is an admission that fixed thresholds do not work well here.
So Minerva is not competing with your DCGM stack. It is claiming to be the correlation and inference layer above it, in the same relationship that an APM product has to raw metrics. The two real questions are whether the correlation is good enough to be worth a new dependency, and what it costs.
What to Take From This Even If You Never Deploy It
Most readers of this will not run a factory with hundreds of thousands of GPUs, and Minerva is quite clearly aimed at those who do: the named partners are IREN, Aolani, Bitdeer AI, IBM, Jio, Lambda, Quake AI and Tesla, with Dell, Lenovo and Supermicro offering it in their AI factory portfolios. That is a neocloud and hyperscaler list.
The ideas are portable at any scale, though, and four of them are worth stealing:
Build the topology graph, even a crude one. You do not need a product to record which GPUs, NICs, switch ports and cables a job is using. A table populated from your scheduler and your fabric inventory, joined at query time, converts "this GPU is hot" into "this GPU is hot and it is on the critical path of job 4417." That single join is most of the value.
Make goodput an SLI with a dashboard. Throughput is vanity at this scale. Track work that survived to a checkpoint or a served response, and track the gap between the two. If you only instrument one new thing this quarter, instrument the difference between attempted and useful work.
Alert on deviation from peer, not on absolute thresholds. The canonical AI factory failure is one device behaving differently from its identical siblings. A rule comparing each GPU's temperature or step time against the distribution of its peers in the same job catches that class of problem without any machine learning, and it catches it before any absolute threshold fires.
Consolidate before you page. If one fabric event can generate 81 alerts, your on call rotation is being trained to ignore alerts. Grouping by topology, even approximately, is worth more than better individual alerts.
The Open Questions
Being direct about what is not known, because the launch material is uniformly positive and the independent evidence is thin.
The models are undisclosed. "Purpose built AI models" with supervised and unsupervised components is all that has been said. Without architecture, training data or accuracy figures, there is no way to assess false positive rates, and false positive rate is the entire difference between an anomaly detector that gets adopted and one that gets muted in week two.
Every number is vendor or partner sourced. NVIDIA's own material reports outcomes qualitatively: longer time between interruptions, faster time to repair, fewer and more relevant alerts. The quantitative figures come from one partner's internal tests without methodology. There are no independent benchmarks yet.
Pricing, licensing and general availability are unclear. Several named companies were described as planning to deploy rather than running it in production. For a platform that would sit at the center of your operations, that is a significant unknown.
Scale is asserted rather than demonstrated. Support for hundreds of thousands of GPUs and billions of data points per hour are claims, and plausible ones given who is making them, but not yet shown publicly.
Vendor coupling is the strategic question. The telemetry scope is ConnectX, BlueField, NVLink, Spectrum-X, NCCL. If your fleet is entirely NVIDIA, that depth is the product. If you run or plan to run AMD, TPUs or custom accelerators alongside, you need to know what happens to the correlation graph when half the fabric is invisible to it. Nothing in the launch material addresses heterogeneous fleets.
It overlaps with things you may already pay for. If you run a commercial APM at GPU scale, you are already paying a lot for partial coverage. Minerva is not obviously additive to that, and nobody has published a sensible answer on how the two should coexist.
Questions Worth Asking Before You Buy
If you are in a position to evaluate this, the things I would want answered:
- What is the false positive rate on the anomaly detection, measured on a cluster like mine, and how is it tuned?
- How long is the learning period before the baselines are usable, and what happens during a hardware refresh when the baseline shifts?
- What exactly does automated remediation touch, and what is the rollback path when it drains the wrong node?
- Does the knowledge graph degrade gracefully on non NVIDIA components, or does it have blind edges?
- Is the telemetry exportable, or does adopting Minerva mean the correlated view only exists inside Minerva?
- What are the retention and cardinality limits at my GPU count, and what does that cost?
- How does this interact with my existing Prometheus and Alertmanager setup, and am I expected to replace or supplement them?
That list is not hostile. It is the same list you would bring to any observability purchase, and the fact that the launch material answers almost none of it is normal for day one.
Conclusion
The framing is the most interesting part of this release. NVIDIA is arguing that AI infrastructure should be measured like a factory, on yield, and that the dominant threat to yield is not failure but degradation that no current dashboard surfaces. That argument is correct and it is supported by the public failure data: roughly one unexpected interruption every three hours on a 16,000 GPU cluster, with about half traced to GPUs and memory, is an environment where continuous partial degradation is the normal operating state rather than an exception.
Minerva's specific answer, a dynamic topology graph joining devices to jobs, unsupervised detection on learned baselines, consolidation of event storms into single incidents, and remediation that a human can approve, is a sensible architecture for that problem. It is also, at launch, almost entirely unverified outside NVIDIA and its partners, deeply coupled to NVIDIA silicon and fabric, and priced unknown.
For the small number of organizations running factories at the scale of the named partners, it is clearly worth an evaluation, with the question list above. For everyone else, the useful move is to take the four portable ideas, especially peer relative alerting and goodput as a tracked SLI, and implement them on the DCGM and Prometheus stack you already have. Those will improve your yield this quarter regardless of what you eventually buy.
References
- NVIDIA Blog: NVIDIA Unveils Minerva to Bring End-to-End Observability for AI Factories, October 22, 2026
- NVIDIA Blog: GTC Berlin 2026 live updates
- Unite.ai: NVIDIA Launches Minerva AIOps Platform for AI Factory Observability
- Verdict: Nvidia launches Minerva AIOps platform for AI factory observability
- Tom's Hardware: Faulty H100 GPUs and HBM3 memory caused half of failures during Llama 3 training
- KubeNatives: GPU Monitoring with DCGM Exporter: The Metrics That Matter
- NVIDIA Developer Blog: Get Real-Time Visibility into GPU Usage Across Kubernetes Clusters
- NVIDIA product page: Minerva