How Microsoft Ships AI Agents at Enterprise Scale: Breaking a Production Agent into 5 Layers
- Microsoft Core AI VP of Product Marco Casalaina detailed how the Microsoft Foundry platform supports over 80,000 enterprises building and running AI agents.
- He breaks a production-grade agent system into 5 layers: the reasoning layer (swappable models), the agent runtime (orchestration loop), the observability and governance layer, the identity layer, and the context layer.
- Retrieval has been redesigned as a planning, retrying "sub-agent" (agentic retrieval) — no longer the one-shot classic RAG that's done after a single lookup.
- Evaluation upgrades from generic metrics (does it run) to rubric-based scoring targeting specific behaviors, paired with Agent Optimizer, which automatically rewrites prompts and swaps models to fix problems.
- The piece offers two principles that don't depend on Foundry and any team can copy directly: retrieval should be built as a retryable loop, and any agent that takes action must have an auditable identity.
The demo that ran fine falls apart the moment it ships
On the Microsoft Foundry platform, more than 80,000 enterprises are already building and running AI agents; Microsoft's own Microsoft 365 Copilot runs on the same platform, serving over 20 million users.
Marco starts with what's changed over the past two years: "We're moving past the Q&A phase of AI. In 2026, more and more customers are using voice as the front end, so we're also moving past the chatbot era." The old form and the new form are very different.
This shift is exactly what makes the engineering problem completely different. A demo can be thrown together in a single evening — the model's smart enough, the test prompts all run fine, the demo looks impressive, and the pilot ships in a week. Where things actually crack is in production, which exposes problems the demo never could.
Things real users ask that you never imagined in the demo.
The material the agent depends on quietly goes out of date, and the answers go wrong with it.
A vendor ships an update, the agent's behavior shifts, and no one notices until customers complain.
The agent runs on a shared system account, so when something goes wrong, there's no way to trace who did it.
It confidently says something it shouldn't.
You have no idea whether quality is getting better or worse.
Asked what the biggest lesson has been from running these systems, Marco's answer was: the harness matters as much as the model. The harness means the entire set of things wrapped around the model — the runtime, tools, context retrieval, identity layer, guardrails, evaluators, and deployment pipeline.
Models change every few weeks, and you can't treat that like a database version upgrade. Bump a Postgres version, and you can generally expect it to just work; models aren't like that — every model has its own quirks, and the harness has to be retuned to match. After Anthropic shipped Claude Opus 4.8, Microsoft's GitHub Copilot CLI team had to retune the harness and rerun the evaluation suite before they could ship it.
