Deep Dive · XiaoHu Explains

Ploy Switches Production AI from Opus 4.8 to GPT-5.6 Sol: Time Cut by More Than Half, Cost Down 27%, No Quality Loss

Swapping models isn't just swapping an API: four invisible traps — eval framework, tool parameters, caching, reasoning records — unpacked and fixed one by one
TL;DR
  • Ploy switched the default model behind its website-building AI agent from Claude Opus 4.8 to GPT-5.6 Sol, which launched that same day
  • Its own benchmarks show: for the same kind of website-building task, time dropped from 8:00 to 3:42, cost fell from $3.06 to $2.22, and the visual score rose from 0.936 to 0.970
  • During migration, they found GPT-5.6 invents default values for tool-call parameters it doesn't need (like offset: 0), which at one point silently caused 52% to 64% of file reads to return empty content
  • OpenAI's prompt caching mechanism is completely different from Anthropic's; after redesigning the cache layering with a workspace-level cache key, the first-call cache hit rate went from 0% to 83.7%
  • The team also found their own eval framework had a hidden bias toward the old model it was used to, so the framework itself had to be fixed before its results could be trusted
⚑Stance note: This piece is Ploy's official engineering blog — a vendor's own account. The time, cost, visual scores, cache hit rates, parameter-inclusion rates, and other figures cited here all come from Ploy's own eval framework and production traffic, and the benchmark standards are Ploy's own. What follows just relays the facts and practices it recorded.
1Background · Model Switch

Ploy Switched Its Default Model to GPT-5.6 Sol

On July 9, 2026, AI website-building product Ploy switched its agent's default model from Claude Opus 4.8 to GPT-5.6 Sol, which OpenAI released that same day.

Switching models turned out to be far more complicated than swapping an API endpoint. Ploy's agent has to plan pages, read the codebase, write components, generate images, screenshot its own output, and then judge whether it's actually done. Moving this whole pipeline from Claude to GPT-5.6, the team discovered — one failed eval after another — that what gets treated as "the model" is actually each provider's own set of behavioral habits, and the whole stack had quietly grown into a shape built around the old model.

📊
Same production-grade agent, same Vercel AI SDK — only the default model changed. In Ploy's own benchmarks, for the same kind of website-building task: time went from 8:00 down to 3:42 (about 2.2x faster), cost went from $3.06 down to $2.22 (27% cheaper), and the visual score rose from 0.936 to 0.970. For the past four months, Opus had held the default slot (first 4.7, then 4.8), and none of the models the team tested had beaten it — GPT-5.6 is the first one that did.
Colorful inflatable shapes connected by translucent tubes, a metaphor for an AI agent migrating between two model systems
Ploy official blog image: several colorful inflatable shapes connected by a single translucent tube, a metaphor for a production-grade agent migrating between two model systems. Source: Ploy Blog

Below is a sample from its brand-page-rebuild eval suite, where the agent has to redo a brand homepage to match a reference design:

Average per completed buildClaude Opus 4.8 (n=11)GPT-5.6 (n=10)
Cost$3.06$2.22
Time8:003:42
Input tokens2.60M1.70M
Output tokens33.0K17.1K
Visual score0.9360.970

GPT-5.6's code is leaner. In one paired sample, Opus generated a globals.css with 17,957 characters and 174 CSS variables (large chunks of the color scale went unused), while GPT-5.6 used only 2,508 characters and 45 variables to render a comparable — sometimes better — page.

Claude Opus 4.8
Top of the CreatEcards page as rendered by Claude Opus 4.8
GPT-5.6 Sol
Top of the CreatEcards page as rendered by GPT-5.6 Sol
The paired sample described above, as rendered by each model: Opus 4.8 on the left, GPT-5.6 Sol on the right. GPT-5.6 delivered a comparably finished page with roughly one-seventh the CSS. Only the top of each page is shown; click an image for the full-length screenshot. Source: Ploy Blog
Related coverage on this site
OpenAI Officially Launches GPT-5.6: Overall Intelligence Score Closing In on Claude Fable 5, Cost Cut in Half, ChatGPT and Codex Merged
That piece covers what GPT-5.6 itself shipped; this one is a production user's real-world migration record from the day it launched.
2Is the Eval Fair

Check Whether the Eval Framework Is Fair Before Judging Which Model Is Stronger

Before comparing two models' scores head-on, you have to ask one thing first: is the eval framework itself fair? Ploy ran the same eval suite across both model families, and the most surprising finding was that the framework had already been tuned around the old model — without the team realizing it.

On the first cross-model run, the bias showed up in a few specific places: