Harness engineering is not enough: why the software factory model fails
- The trending "lights-out factory" approach has AI write and review code while humans read nothing. HumanLayer founder Dex Horthy argues in a two-part post why this model fails—not due to poor workflow, but because of how models are trained.
- Faros data tracking 22,000 developers over two years shows a 66% rise in completed epics per developer, but production incidents per PR spiked 242.7%, and unreviewed PRs merged directly into main grew 31.3%.
- The root cause lies in training rewards: the grading harness only checks whether the target test passes and existing tests don't break. How the fix is implemented doesn't affect the score, so swallowing exceptions in try-catch blocks gets full credit.
- Bad architecture goes unpunished because it lacks a fast feedback loop: tests execute in seconds, whereas the bill for poor architecture arrives months later—far too late to backpropagate.
- The solution isn't using less AI, but shifting effort upfront: product review, system architecture, program design, and vertical slicing. He estimates an hour of planning cuts review time from six hours to 20 minutes, eliminating 80% of expected pain in the first ten minutes.
In July 2026, HumanLayer founder Dex Horthy delivered a keynote titled "Why Software Factories Fail" at the AI Engineer World's Fair. He expanded his talk into a long-form essay, split across two posts on July 24 and 25 due to X's media limits per post.
Part 1 is subtitled "Harness is Not Enough." The industry calls the environment surrounding AI coding models a harness—the tools provided, context fed, retry attempts, and automated review checks. Over the past year, engineering effort has focused heavily on refining this harness to achieve fully autonomous pipelines. His thesis: this approach cannot fix code quality because the issue lies in how models are trained, not in their harness.
Part 2, "Turning the Lights Back On," details the workflow his team adopted after learning these hard lessons. This breakdown synthesizes both parts.
The state of software factories and blind automation
Faros AI aggregated two years of engineering data across 22,000 developers and 4,000 teams. None of these numbers come from surveys; all were extracted directly from code repositories, CI pipelines, incident systems, issue trackers, and IDEs. They compared the same companies across periods of low vs. high AI usage.
While output metrics look impressive, downstream metrics show rising incidents and bugs.
per developer
per developer
per developer
per PR
(up from +9% prior year)
turnaround time
The fourth metric, "incidents per PR," normalizes for higher submission volume. Higher submission frequency alone cannot explain this rise—individual changes have become inherently riskier. Faros attributes this to reviewers failing to keep pace with AI code generation rather than intentional reviewer bypasses. Meanwhile, unreviewed PRs merged directly into main (without human or machine review) rose 31.3%.
Dex highlights these figures to underscore a core point: code output has increased, but the technical debt is being paid downstream.
What a software factory is—and what "lights-out" removes
A "software factory" is an assembly line where ideas enter a queue, get built, reviewed, deployed, and recycled back when incidents occur. This workflow predates AI.
With AI, only one stage shifted: "Human builds" became "Agent builds." Companies like Ramp, Stripe, WorkOS, and Brex have reported that 75% of their code is written by agents. However, while building dropped from days to minutes, code review stayed at hours or days. The review stage quickly became the primary bottleneck.
Teams tried layering on more automation: AI code reviews, browser regression tests, and auto-routing production incidents back to agents. Yet review remained slow. Eventually, some teams eliminated human review altogether.
2022 Model: Both building and reviewing are done by humans, taking hours to days.
Agent Factory: Building drops to minutes, but reviewing remains hours to days, making review the main bottleneck.
Lights-out Factory: Human review is eliminated entirely, redirecting effort to automated testing, sandboxes, auto-reviews, and canary deployments.
The term "lights-out" was coined by Dan Shapiro, drawing from fully automated manufacturing where no humans are needed on the floor. StrongDM is frequently cited as an example, operating pipelines where humans neither write nor read code. At this stage, the only remaining constraint is how many tasks can be queued.
