During September, we started working closely with a mid-size tech team that wanted to build their software factory. We didn’t want to overcomplicate the challenge, so we decided on an extremely naive approach:
Every 5 minutes a script wakes up, checks Linear, GitHub, active ongoing tasks, and chooses a path forward for each available issue. That’s it.
No orchestration, no message queue, no custom context. All that lives in the systems of record that already exist in your organization. Our system just remembers the pointers. But, why?
The source of truth must be where humans work.
An update to a ticket, PR review, slack thread, notion document, or any system should reflect immediately across the system. No delay, no wait until we rebuild the embeddings, no sync’ing issues. And more importantly, no overhead controlling all systems instantly.
Just like a human would do: AI reads comments, it knows who’s the authority, and follows accordingly.
We’ve been running these types of project for almost 2 months now, today, we share how we started. The only caveat: we did leverage our existing infra in agent-swarm.dev heavily.
First: The setup
The company: We run this with a partner, a software company with 80+ employees, and a b2b software that has 4+ years. This was not a toy example.
The project: A feature that was meant to take a whole sprint (2 weeks), for 2 people. 14 defined non-trivial issues. The system touched their whole stack, including sensitive areas for data and operations.
The system: It has access to all codebases, ie. multiple repos, a few critical for the product and organization. Those were the ones changed. It had access to the db (obvs, only to change the dev environments). An advanced CI/CD with thousands of tests, using tools for static and dynamic code analysis, with feedback for security, compliance, code-health, and more.
The kick-off: We pointed the system to the project. We defined one human owner reviewing every key PR and/or decision. The cycle took 7 days, every issue was closed after 103 automated iterations. And several AI requested human interventions.
Second: A FSM for existing states
The workflow executed in each turn simply resolved states between systems (Github, Linear, our own tasks), if they weren’t already updated. After that it executed a state transition (think of turn based systems/games with dependencies) in each run.
For example:
-> From Backlog to Todo
-> From Todo to In-progress
-> In Progress to Review.
-> Any state to Human review required.
But it wasn’t only the transition that applied, it would always spawn multiple tasks to resolve the next steps required for each of the tasks that were now unlocked. So, implicitly, something that was marked as in-progress, or a task that was marked as continue, would spawn a new agent to address such a task.
Agent selection, context retrieval, model effort, etc, all are controlled for our lead agent.
Finally one hard rule: The human has the last word to mark something as done.
The factory heartbeat loop
This one is easy, it’s an old semaphore strategy in which you lock, you execute an overarching task, and once you update states, you give back the lock.
Choosing the timeout for the lock here was the only tricky part. For us, given that we were running 5 minute loops, 3 minutes seemed like the ok option. We need our system to execute the step in that window.
Three things to take into account:
Custom Dashboard: We built a simple UX to observe the state of this FSM, that helped us not only debug but also understand how the project was going.
Deterministic Validation: A lot of scripts, CI/CD automations, litmus gates, etc. to ensure as humans we were only bothered if something was ‘ok’. This gets better with each iteration.
Only selected projects: Choosing which project to flag as part of the factory was the main discussion within the team. You want something complex enough, but not extremely complicated for your first iteration.
Third: The budgeting trick.
We added limits to the activities, not just to control costs, but also to understand the behavior early on. A few of those:
1 action per tick per issue, ideally making failing ticks idempotent but with added context.
Loops have a maximum duration: 5 minutes. We don’t want the system/tasks spinning forever. If the step takes longer, it needs to be divided and retried.
Cap activities: our ticks allowed 5 active implementations, 5 CI fixes, 5 review fixes, 5 merge-queue fixes, 3 ticks with no CI result, 25 attempts in total. Past that, human intervention was required.
~2hrs: Overall stuck tasks will raise an alert if no progress was made in ~2hrs. That would set them as blocked and required human review.
The first round numbers
14 issues in 1.1k active issue-hours, with truly ~80 actively working hours. The rest was mostly waiting for humans to respond or CI/CD. They do sleep often.
That means, when active, we were usually working on ~14 issues in parallel (btw, the system did create smaller sub-issues).
For context, the human did 43 reviews: 14 approvals and 21 change requests. The feedback came in ~15 minutes when asked during working hours.
The-one-thing
Throughout the project, and now as a continues effort, the team is focused on:
Improving the infrastructure
Most task failures were due to some infrastructure issue. This is going to continue as you accelerate. Your team should have the leverage to scale their solution.Idempotency
Repeated tasks appeared as result of overloading the system. We also saw broken states, and unnecessary iterations. Using TLA+ to represent our tasks states took care of that. Look for formal verification methods for your FSMs.AI slop
AI ignored issues or would post messages with fake progress. In cases were AI was reluctant to implement a solution, the budgeting came into play. For those cases, you need to understand the challenge to built your skills & system around it.
The launch was a success.