Building a Software Factory

My mornings start with three Linear views: Needs Merge, Needs Go, Needs Answer. If all three are empty, nothing needs me. In the first half of this week, 77 pull requests merged across four repos. As a software engineer who has been writing code since I was 12, with a deep love of the craft, I am proud to say I did not write or even edit a single line of application code.

In late June, I spent a week in San Francisco at the AI Engineer World's Fair and attended over 60 (of the 500+) talks, workshops, and keynotes. The pattern that kept surfacing across many of the talks was the "Software Factory." I spent the 6-hour flight home with Claude and my Obsidian vault condensing my notes, slide photos, and transcripts into my own canonical idea of a complete agentic software factory. With it all fresh in my mind, a 4-day weekend to spare, and a Claude Max 20x subscription with Fable 5, I put over 36 total hours into building my own factory covering several related applications.

As Dex Horthy puts it, the idea of a factory is nothing new in the world of software (he traces it back to 1968). We are just now starting to find where AI best fits, and where it needs to be constrained by either traditional CI/CD or human judgment and oversight.

This is what it looks like at a glance: a loop of AI agents that takes work from 'I should build this' or 'this should be fixed' to deployed code. I sit at just three points: nothing enters without my label, no spec gets built without my merge, nothing ships without my final judgment recorded. Everything between those points is agent executed, with rigid CI keeping them in line.

I run this solo across three products as well as a handful of related marketing sites.

The loop

Every piece of work moves through the same stages: signal, triage, plan, execute, validate, ship. A seventh stage, compound, runs alongside all of them and feeds lessons back into the machine.

• Signal
  Anything worth doing becomes an issue in Linear. Ideas, bugs, and alerts. I have a Hermes agent that reads my Wispr Flow or Pocket transcriptions and creates tickets on my behalf.
• Triage
  An agent is constantly watching my Linear backlog. It automatically labels each new issue with repo, size, and risk, picks a route (small things go straight to implementation, bigger things get a spec first, judgment calls come to me), and posts a ranked brief.
• Plan
  For spec-first work, an agent writes an RFC: the problem, the approach, and how we'll know it works. That last part is the contract everything downstream gets judged against. The RFC lands as a pull request and stays in the repo permanently.
• Execute
  A new agent implements against that contract on a branch, writes the tests, and "grinds at the gates until everything is green" (title of an AI workshop from an engineer at Reddit). Then it opens the PR.
• Validate
  CI runs a ladder of deterministic checks, plus an AI code reviewer that judges the PR for correctness and, more usefully, against the spec's validation contract.
• Ship
  I merge. Merge deploys, and a post-deploy check rolls back automatically on failure.
• Compound
  Every time I correct anything, the lesson gets routed somewhere permanent. More on this below, because it turned out to be the most important stage.

My three touchpoints map onto that loop as checkpoints. Checkpoint one: nothing proceeds until I apply a label to the issue. LLMs are not ready to determine their own work, and I've canceled or massively changed several tickets at this stage. Checkpoint two: I review and merge the spec. Corrections become decision logs and feed my system-wide Engineering Strategy. Checkpoint three: I merge PRs that are fully green. If I ever need to (or accidentally) do anything else, like manually dragging an issue between statuses, ask the system about something that seems stuck, or deal with a broken build, a sweep captures that as a bug in the factory itself, and it gets filed into the queue.

The primitives

A few design decisions do most of the work.

Every agent stage is a skill with two triggers
Each stage is written once, as an agent skill, and can fire two ways: me invoking it in a terminal session, or an autonomous runner on a schedule or webhook. It was a good amount of work to ensure each skill can work on my local harness, a GitHub runner, or a Claude Code cloud session, but keeping it skill-based means it's much easier to debug and maintain. The "factory" really started as me running skills in a Claude session one at a time as a mechanical Turk. After I could trust it on my machine, I attempted it in a Claude Code cloud session, then automated it with either Claude Code routines, GitHub Actions, or both. You can't debug an autonomous agent you've never driven manually.

CI runs the tests
I made heavy use of git pre-commit and Claude Code hooks. Deterministic checks surrounding unruly non-deterministic LLM output is the only way to stay sane and safe. Human engineers can get real grumpy if you introduce excessive linting, testing, and overall rigidity in your application code, but LLMs will happily refactor ad nauseam to get the build to green.

Separation of concerns
Implementation and validation are completely separate agent invocations: fresh context, different rules. I've seen countless instances where the current session tries to convince me a bug we are working on is from a previous session, or some unrelated issue. But an LLM reviewing "someone else's" code has no incentive to defend it, since its primary goal is actually to find issues. The validator also ensures no tests were changed in devious ways by the implementor to dishonestly get to green.

Autonomy graduates per class of work
Some PRs now merge without me: ledger entries or certain changes to marketing pages. I expect to see this grow over time as I add more granularity to my ticket and code classifications, as well as stricter tests. I have a weekly sweep that pays attention to accumulated weeks of "zero-rework" history, that sweep proposes additional "safe" classifications along with evidence and a plan. A single incident in a graduated class revokes it automatically until I'm happy with it again, kind of like Toyota's andon cord: any defect stops the line. One area that will never be autonomous is changes to the factory itself. That's how you end up with a robot uprising.

Every run appends to a ledger
I opted for ndjson here, every single agent run appends what stage, which issue, what outcome, and what it cost (or really what it would have cost without a subscription). Another weekly sweep compiles all of the ledger entries into a scorecard which shows items shipped, signal-to-merge time, cost per merged PR, and rework rate. These scorecards are also permanently saved in my repo and will eventually end up in a proper database so I can do more with it than a weekly dashboard.

Always compounding
Back in early 2025, an interview with Kieran Klassen gave me my first "Oh sh*t" moment around AI engineering as he talked about his view of compounding engineering. He had a workshop at the AI Engineer World's Fair where he showed some of his updated thinking and explained his primitives more clearly. He proposes that traditional codebases tend to degrade over time, but AI engineering done well can actually do the opposite. Anyone who has tried to have OpenClaw write your code (or just run a claw in general for more than a couple of months), will have a hard time believing this, but it comes down to your own design. LLMs will NOT produce compounding codebases by default by any stretch of the imagination. While I don't use Kieran's very well done compounding engineering plugin, I used his concepts to wire compounding throughout my entire factory and this is currently my favorite part of the whole system. Both an agent and I do regular sweeps across every ticket and PR looking for opportunities to learn and improve. I have constant proposals for pre-commit hooks, CI checks, skill files, templates, documentation, polish, tickets to be filed, etc. These come over conversationally first, and not all are accepted. It's amazing how many requests come over that cause me to initially groan because I can still feel like I have to be the one doing the work. Some petty change to a linter or prettifier that if a co-worker had asked of me a couple of years ago I would have fought hard. But AI gives me the freedom to have the pettiest, most particular standards since the entire implementation might only consume a minute of my total attention.

The tooling, roughly

I built this factory to match my own existing tooling and preferences. If you made it this far, you're probably interested in a recipe. I would highly recommend you take the concepts and build your own instead. The experience was worth it, and it feels like an extension of my engineering workflows rather than a new system to learn.

• Linear is the queue as well as the state machine. I use labels for everything. A "go" label is checkpoint one, and the "route/size/risk" labels drive pretty much everything else (e.g. route:spec-first, size:medium, risk:low). The system uses probably 6-8 total labels for a given work item, but I'm only concerned with a couple.
• Agent skills hold each stage's logic: instructions, rubrics, and templates in a dedicated skills repo.
• Claude Code routines are the autonomous trigger for triage, planning, and execution, scheduled sweeps and event-driven runs, each invoking the same skill I'd invoke by hand, and I keep the routine's prompts versioned in the factory repo.
• GitHub Actions runs the validation ladder and the AI code reviewing, plus relay workflows that trigger on CI/CD state: fire the execute workflow when a spec merges, move the Linear issue when an implementation merges, file a ticket when a build breaks.
• Git hooks are the cheapest guard, so they're the first one. Pre-commit runs lint on staged files, secrets scanning, a ledger-schema validator, and repo-specific guards, all in under five seconds. Pre-push runs the fast verify suite. Shift left.
• Branch protection everywhere. Even I can't merge until the build passes.

What bit me

A handful of things of note that caught me:

Identity. Some of this is still outstanding and causing problems: The runners act as me: my comments, my PRs. I cannot use PR reviews the way I want since I authored the PR. I'm also realizing I can't measure my own involvement anymore, because every Linear comment resolves to my name whether a human or a robot wrote it. Part of the delay is seat cost and the toil of getting it rewired, but I need to make the switch soon.

A guard that silently doesn't run is real bad. Two specific incidents. 1) GitHub Actions treats a skipped upstream job as success by default, so when I added a fastpath that skipped a formatting job, the AI review gate quietly inherited the skip and "passed" several times without actually running. 2) way scarier: I added an auto-merge classification for "docs-only" changes but didn't check the PR, and it lazily passed for any markdown content. But this particular repo was Astro and the site content is markdown. I must be more diligent with human code review on the factory itself.

LLMs do what the instructions say, not what you meant. I once had to revert a bulk state change an agent had made. The next 8 scheduled runs tried to redo it and I had a mountain of PRs to clean up. Read the skills. This is one area to be constantly refactoring by hand.

Concurrent agents can't share mutable files. My ledger started as one append-only file per repo. With multiple agent PRs in flight I quickly got into conflict hell. I switched to one file per agent with write timestamps vs. sequential numbering, so conflicts are isolated to individual PRs.

Squash merges break stacked branches. After a squash merge, a child branch never "contains" its parent's work, and you get add/add conflicts that oddly confuse agents even more than they confused me. I sacrificed a more succinct git log for the sake of conflict isolation. Honestly, with agents following conventional commits, it's better to have a more verbose log that shows the full history of a branch, so that's two wins.

It's expensive. I burned through 90% of my first Claude Max 20 plan's monthly usage in two weeks and had to switch to "extra usage", and burned up another ~$200. I didn't get the estimated token counts working in the ledger until several days in, and around a dozen total runs logged $176 in would-be API costs, so the full two weeks is likely in the thousands without a subscription. I will likely need to move some of these steps to opensource models when the generous subsidies run out.

The numbers so far

One caveat on the "36 hours": I did have some of these components in place already, such as the AI code review and much of the CI/CD pipeline.

Two "weekly scorecards" in:

The stat I care most about is what happened to the backlog. Issues that had sat for 50-80 days, real fixes I never got to, shipped the same day the loop picked them up. A backlog that isn't just a place where tickets go to die is something I've never truly had, personally or professionally.

What's next?

The factory is two weeks old and it works, but I don't fully trust it yet. Trust here looks like the graduation system: accumulated zero-rework history, one incident revokes it. The machine proposes its own improvements with evidence attached, so my job from here is mostly slowly saying yes.

If you're building something like this, have done it better, or have any questions, reach out.