Skip to content
Update4 min read

NVIDIA Cluster Readiness Engine: Certify a GPU Cluster Before Workloads Land

NVIDIA has open-sourced the Cluster Readiness Engine (NVCRE), a Kubernetes controller that certifies a GPU cluster before production workloads touch it. It runs real communication and training workloads across the cluster, measures them, and reports every node that failed and why. NVIDIA introduced it on September 23, 2026; version 0.5.0 followed on September 28.

The problem it goes after

NVIDIA's own framing is the clearest: a GPU cluster can pass every health check and still fail to run an AI workload. Some faults only show up under distributed load. In a synchronous job, one slow GPU drags every other GPU down to its speed, and almost nothing in the telemetry says which one.

So NVCRE does not inspect the cluster. It loads it.

The short version

NVCRE
WhatKubernetes controller plus a kubectl nvcre CLI
LicenceApache 2.0
Latest releasev0.5.0, September 28, 2026
API maturityv1alpha1, so expect it to change
Communication testsFive NCCL variants: all-reduce, all-gather, all-to-all, loopback, loopback across NVSwitch
Diagnostic testsDCGM level-4 suite
Training testsMulti-node NeMo pretraining, measured by goodput
Platforms detectedAWS, GCP, Azure, OCI, several GPU clouds, on-premises
NeedsNVIDIA GPU Operator; Prometheus Operator CRDs for metrics

How a certification run works

Trace05 stops

One certification, from request to a named list of failed nodes.

  1. CERT.RUNCertificationYou pick categories from the catalog. One workflow is created per category.
  2. GROUP.NODESTopology groupsNodes are grouped by where they sit: within a node, within a rack, or the full cluster.
  3. JOB.LOADReal workloadNCCL or a training job runs on each group while node health is watched.
  4. MEASUREMeasurementPod logs are parsed for bus bandwidth and goodput, against thresholds.
  5. REPORTReportStatus per category, measured numbers, and each failed node with a reason.
The third stop is the difference from a health check: the cluster is tested by doing the work.

Getting a first result is three commands:

curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash
kubectl nvcre setup init
kubectl nvcre certification run --category communication/nccl-all-reduce --wait

Finding the one bad node

A failed group of thirty-two nodes is not an answer. In diagnostic mode NVCRE narrows it down by itself.

Trace04 stops

Adaptive fault isolation: split, rerun, repeat.

  1. GROUP.FAILA group failsA workload on a group of nodes misses its threshold.
  2. GROUP.SPLITSplit in halfThe failing group is divided into two smaller groups.
  3. JOB.RERUNRerun the halvesEach half runs the same workload. A passing half is cleared.
  4. NODE.SUSPECTSuspect nodesSplitting stops at the minimum group size, leaving the nodes to look at.
The result is a short list of suspects rather than a whole cohort under suspicion.

What it does next is deliberate: nothing. NVCRE records the failed nodes and reasons. It never cordons, taints or otherwise modifies a node. Whether to quarantine one stays with the platform operator, which keeps it from fighting whatever remediation system is already in place.

Why this matters for agents

Most agentic systems call a hosted model and never see a GPU. The ones that do see GPUs are the ones with the least room for surprises: private, self-hosted, on-premises or air-gapped deployments, where the model serving layer is yours to run. There, a marginal node does not announce itself as a hardware fault. It shows up as an agent that is slow, or times out mid-task, on some requests and not others.

NVCRE's catalog is built around training and interconnect tests, not inference serving, so it does not tell you how your model server will behave. What it gives you is a floor: evidence that the hardware under the inference layer is sound before you start tuning anything above it. That is worth having before a private deployment goes live, and again whenever nodes are added or replaced.

We have not run NVCRE ourselves yet. Two things to weigh before you adopt it: the API is v1alpha1, and the repository is about two months old, so pin a version and expect breaking changes.

Sources

Related.