Nvidia's GPUs are widely treated as the trustworthy half of the machine. The driver is the part people blame when a training run diverges. That framing is backwards. Silicon does not promise correct arithmetic; the driver and CUDA runtime assemble that promise out of patches, ordering rules, and regression tests. This piece explains where that promise lives, how it is maintained, and where it quietly fails.

The Myth of Hardware Correctness

A floating-point unit is a physical circuit, and physical circuits round. Tensor cores trade precision for throughput in ways that are documented but easy to forget. A matrix multiply that accumulates in reduced precision can produce a result that is close to correct and not identical to the reference. The hardware did what it was built to do. Nothing in the chip raises a flag.

Errata sheets exist for a reason. Successive GPU generations have shipped with known bugs that firmware or the driver works around, sometimes by disabling a fast path or inserting a fixup. Those workarounds are invisible to the application. A kernel that ran correctly on one board revision may take a slower route on another and still return the same numbers, or nearly the same numbers.

The driver absorbs the blame because it is the visible surface. When a loss curve spikes or a gradient turns to NaN, the first suspect is the software stack. Sometimes that is right. Often the fault is a rounding behavior that was always there and only now matters because the model got deeper or the batch got larger.

What the Driver Actually Guarantees

CUDA defines a memory model, and the model is the real contract. It says when one thread's write becomes visible to another, what ordering the runtime enforces, and which operations are atomic. The driver implements that model on hardware whose caches and schedulers were not designed with it in mind. Correctness here means the implementation matches the model closely enough that programs written to the model behave.

There is no formal specification of the whole stack that a user can check against. There are headers, programming guides, and an enormous body of tests inside Nvidia. The gap between the documented model and the shipped behavior is patched continuously, and the patch notes are terse. A behavior change that fixes one workload can alter results in another.

Firmware sits between the driver and the silicon and carries its own workarounds. A firmware update can change how a unit handles a corner case, and the change may not appear in any API documentation. This site has argued in a related piece that signing keys and update provenance deserve scrutiny before any firmware push, and the same caution applies to GPU firmware.

The Cost of Silent Data Corruption

Undetected bit flips are the expensive failure mode. A flipped bit in a weight matrix may cost a fraction of a percent of accuracy, which is invisible in a loss curve and expensive in a benchmark. A flipped bit in an index can crash a kernel or, worse, produce plausible garbage that propagates through a checkpoint.

The industry has a rough sense of how often this happens, and the numbers are uncomfortable. Large fleet operators have reported silent-error rates in the low single digits per thousand GPU-hours on some workloads, though the figure moves by an order of magnitude depending on memory technology, duty cycle, and how aggressively the operator screens. What matters for planning is not the exact rate but the shape: errors are rare enough that a single job rarely sees one, and common enough that a fleet running continuously will see them every week.

When corruption is suspected, the debugging cost is brutal. Teams rerun jobs to see whether the failure repeats, and each rerun consumes GPU hours that were budgeted for training. Reproducing a fault may require the same driver version, the same firmware, the same board revision, and the same access pattern. Any difference and the bug vanishes.

Validation suites grow with every release because each new architecture adds cases. The suite becomes a record of everything that has gone wrong before. That is a reasonable engineering response, and it means the test surface is always chasing the hardware rather than leading it.

How Nvidia Engineers Correctness

Nvidia runs large internal regression farms that execute kernels across driver versions and board configurations. Failures are replayed deterministically where possible, which requires controlling scheduling and memory layout. The replay infrastructure is as much a part of the product as the compiler.

Driver teams own numerical contracts for the operations they expose. When a new instruction is added, someone has to decide what its rounding behavior is and how it will be tested. Customer workloads feed back into this process. A large customer reporting a divergence becomes a test case, which is efficient and also means coverage follows revenue.

The methodology is closer to browser rendering than to silicon verification. A related piece on this site notes that Chrome and Firefox repaint differently under the same CSS, which is the same category of problem: two implementations of a shared model that diverge in the corners. GPU math has the same shape, with higher stakes.

The Trade-off Nobody Wants to Price

There is a real tension between speed of release and the completeness of a numerical contract, and the vendor has chosen one side of it. Shipping a driver every few weeks keeps new frameworks working and keeps bug fixes flowing. It also means the documentation is always a release behind the binary, and the errata that would let a careful team reason about their own risk never gets written down.

The alternative is not obviously better. A vendor that froze the stack to document every corner case would fall behind framework releases, and most users would ignore the detail anyway. The cost of the current arrangement is concentrated in the small number of teams whose workloads sit close enough to the hardware that rounding behavior matters. Those teams pay in debugging time, and they pay again when a driver upgrade silently changes a fast path.

Checksums and redundancy are the usual mitigation, and they carry their own price. Running the same reduction twice to compare results doubles the compute for that step. Storing a checksum beside every checkpoint adds I/O and complicates the pipeline. For most teams the arithmetic is uncomfortable: the expected cost of a rare silent error is lower than the certain cost of doubling every validation pass, until the day it isn't.

Where the Burden Breaks Down

New architectures outpace test coverage. A chip taped out this year ships into frameworks that were written for last year's instruction set, and the interaction surface is larger than any internal farm can enumerate. The first months of a new generation are effectively a public beta, and the users are not told.

Third-party kernels bypass the checks. A hand-written CUDA kernel or a custom Triton kernel may use instructions in ways the driver team never tested. Libraries like PyTorch and JAX add another layer, and each layer assumes the one below is correct. When the assumption fails, the report travels up the stack slowly.

Documentation lags behavior. A workaround lands in a driver release, the release notes mention a fix, and the programming guide still describes the old semantics. Users become unwitting testers of a contract nobody wrote down. The objection from inside the vendor is fair: documenting every errata would slow releases and confuse most readers. The cost lands on the teams that need the detail.

Practical Steps for Practitioners

Pin driver versions per workload and change them deliberately. Treat a driver upgrade the way you treat a dependency bump: read the release notes, run the validation suite, and keep the previous version available for rollback.

Run bitwise reproducibility checks on a small fixed input before trusting a new stack. If two runs on the same hardware and driver do not match bit for bit, you have learned something important before the training job starts.

Log GPU error counters continuously. Corrected errors, thermal throttling, and XID events are early warnings, and they are cheap to collect. A dashboard that shows them next to loss curves will save a week of debugging.

Demand numerical contracts from vendors in writing. Ask what rounding mode an operation uses, what precision a fused kernel accumulates in, and which errata affect your board revision. If the answer is not documented, that is the finding.

The broader lesson matches what this site has said about build systems outliving their authors. The layer that holds the promise together is code, it is maintained by people, and it needs the same rigor as anything else in production.