Production engineering: the last mile is most of the journey

Both articles stress that going from prototype to production was harder than expected, because errors in long-running stateful agents compound. A minor glitch that would just slow down ordinary software can completely derail an agent's research trajectory. The key production lessons:

• State management and recovery. Agents run for a long time across many tool calls and cannot simply restart from scratch when something fails. The team built the ability to resume from failure points, and combined the model's adaptability with deterministic safeguards like retry logic and regular checkpoints. They also let agents handle tool failures gracefully by informing them and allowing adaptive responses.
• Debugging non-determinism. Because the same prompt can lead to different paths, ordinary logging was not enough. They added production tracing that monitored high-level decision patterns, while deliberately not storing the contents of individual user conversations, for privacy.
• Safe deployment. Updating code could break agents that were running mid-task. The solution was rainbow deployments, which gradually shift traffic from the old version to the new one while keeping both alive, so in-progress sessions are not disrupted.
• A known bottleneck: synchronous execution. The lead currently waits for each batch of subagents to finish before continuing. This keeps coordination simple but slows things down and prevents real-time steering. Asynchronous execution would unlock more parallelism but adds hard problems in coordinating results, keeping state consistent, and propagating errors. The team names this as future work.