How Amazon Turns Real Failures Into Better AI Models
How does Amazon build its agentic AI? Michael Giannangeli, Head of Product for Amazon Nova and Agentic AI, breaks down evals, RL gyms, and model routing.
He also explains why the bottleneck in software has shifted away from engineering hours and what takes its place.
In this video, we cover:
- The eval lifecycle: building from real failure modes, saturation, and why 100% means delete
- RL gyms: training models on real environments like migrations, DevOps, and pen testing
- Model routing, cost-per-token trade-offs, and why routing isn't solved
- The agent stack of an Amazon product lead: Claude Code, Codex, and Kiro
- Autonomous migrations, trust, and how much human-in-the-loop survives
For engineers and product people building with AI agents who want to see how a frontier lab actually closes its feedback loops.
Recorded at the AI4 conference 2026.
Timestamps:
00:00:00 - Intro
00:00:36 - The Agents an Amazon Product Lead Uses Daily
00:03:36 - Why Nobody's Heard of Amazon Nova
00:04:55 - Model Costs and the Routing Problem
00:08:10 - Why Building Good Evals Is So Hard
00:10:05 - When Evals Saturate and Get Deleted
00:12:17 - Turning Real Failure Modes Into Hundreds of Evals
00:15:26 - Improving Models Without Training on Customer Data
00:18:26 - If Everyone Uses Agents, You Need Agents
00:20:22 - The Bottleneck Is No Longer Engineering Hours
00:23:20 - Ship Fast to Validate the Right Thing
00:26:44 - Staying at the Frontier Amid Constant Noise
00:29:37 - Spend 10-20% of Your Time Experimenting
00:32:54 - RL Gyms: How Models Learn From Failure
00:37:09 - Will Migrations Become Fully Autonomous?
Guest: Michael Giannangeli - Head of Product, Agentic AI & Amazon Nova at Amazon