Hello, Agent:

Why extreme co-design beats faster chips with Ian Finder at NVIDIA

Hosted by

Alexander Gallego

Guest

Ian Finder

September 10, 2026

Show notes

Transcript

Or listen on:

Guest:

Topics

Show Notes:

Most conversations about AI infrastructure stop at the GPU. This one goes underneath it, to the CPU design decisions that determine whether all that GPU capacity actually gets used.

We speak with Ian Finder, Head of Data Center CPU at NVIDIA, about the design philosophy behind the Vera CPU, NVIDIA's clean-sheet core built for the agentic era. Ian traces the lineage from Grace, a GPU host processor built on an off-the-shelf ARM core, to Vera, a ground-up design organized around Amdahl's Law: as GPUs get faster and workloads parallelize further, the CPU can never be allowed back into the critical path.

The conversation covers the engineering trade-offs behind chiplet-based coherency fabrics, why deterministic tail latency matters as much as raw throughput, and how agentic workloads, branchy, memory-hungry, and always saturating, are reshaping what a data center CPU needs to do. We also get into NVIDIA's internal culture around clean-sheet projects, the taste and intuition behind balancing conflicting design constraints, and Ian's home-lab experiments running local open-weight models to orchestrate hardware simulation work.

‍

Key Takeaways:

(00:00) Introduction.

(01:00) Grace to Vera: Grace was understood as a GPU host processor built on an off-the-shelf ARM core; Vera dropped in a purpose-built core designed around Amdahl's Law from the start.

(02:50) The Scalable Coherency Fabric was NVIDIA's first investment in CPU design: a shallow, wide, highly-subscribed L3 mesh built before the core itself.

(03:42) Predictable tail latency, not just throughput, is the real design target: the GPU is a synchronous batch machine, so the CPU's memory subsystem has to guarantee data arrives on time.

(07:37) Chiplets let NVIDIA isolate fixed-latency paths (I/O, memory) onto separate dies, freeing the compute die to maximize core-to-core subscription on the coherency mesh.

(10:40) The "full-tilt at saturation" design point: use available memory bandwidth, make performance deterministic, and build a core that can actually saturate both.

(11:52) Every CPU design point is a balance of competing goals, not a single-axis optimization, illustrated with a car-dealership analogy of speed vs. efficiency vs. capacity.

(15:08) Inside NVIDIA's clean-sheet culture: Vera came out of a small, dedicated team building shared intuition from first principles before ever pulling in data.

(16:17) Agentic tool-calling increases the CPU load line industry-wide, requiring more sustained compute at home and in the data center alike.

(18:08) Amdahl's Law is a lens for balance, not an argument for extremes: most workloads (ETL, sandboxed Python, agent harnesses) need a triangulated design point, not maximum single-thread performance.

(20:53) Agentic workloads are fundamentally CPU-bound: fetching data from databases, generating and compiling code, and running regex and branchy logic all sit off the GPU's critical path.

(24:51) Agent deployments need distinct SLA classes: reinforcement-learning sandboxes need very high QoS, persistent low-intensity agents (like an email assistant) need almost none, and long-running coding agents sit in between.

(26:17) RL rollouts and GPU utilization are tightly coupled: if a CPU-run simulation misses its window, the training run doesn't stall, but it does lose a cycle of on-policy feedback, slowing how fast the model improves.

(29:09) Agent state swapping and GPU KV cache offload are the same underlying problem: a fixed pool of memory that has to be saturated and rapidly hydrated or dehydrated as work shifts between tasks.

(31:37) Open-weight frontier models like KIMI-3 and GLM-5.2 are enabling a shift toward smaller orchestrated models (echoing Stanford's Minions paper), where a frontier model coordinates lightweight local models doing constant, CPU-bound work.

(34:53) Ian's home lab: recreating an SGI Indigo workstation on an FPGA using a local open-weight model on a DGX Spark to orchestrate single-threaded RTL simulation runs and call out to a larger model for fixes.

(46:48) "Users click, agents swarm": today's infrastructure assumptions were built around human duty cycles, and need to be rethought for systems that self-correct without constant supervision.

(50:33) Human-in-the-loop and machine-to-machine workloads will coexist, but the latter, high-throughput, always-on, no-human-intervention automation, is the design point Redpanda is increasingly building toward.

(53:04) Closing thought: hardware sets the wave that software eventually rides, and both NVIDIA's CPU roadmap and Redpanda's architecture were shaped by betting early on where saturation-driven, latency-sensitive design was heading.