During the past 3 weeks I’ve been paying special attention to when we work together with Taras. It’s not frequently we interact in the same thread with our swarm, but when it happens, things take a different turn. Let me illustrate.
Last week Taras had a very interesting idea: figure out how the EU fosters OSS. It was a natural question to ask ourselves, given last week was the Barcelona Open Tech week. To do so, he asked our swarm to run a long study finding funding and exit rounds for the top 100 OSS companies in Spain, vs EU & US. At least that’s how the thread started.
He iterated on that slack thread for 2 days, before he suggested I could take over and actually write the post myself (we don’t use AI for writing, if we do, we disclose it). At that point, being aware of even E&Y reports with hallucinations, I asked my first question:
“I want to know, how did you validate the information we are using, and who wrote the content?”
Simple, right? Just make sure I’m not saying something that’s off. The agent went on to swear it was right, and handed over to me the facts for review. They look good, too good. So I asked one more time that same day:
“I just want to triple check every fact and company we mention”
6 minutes after this, the post was merged. It wasn’t Taras. It was me. I merged the post. The surprise came after.
An hour after the merge, the swarm came back to me saying that we had 4 wrong, 4 unsupported claims. That was because out of the 1,327 claims in the original dataset, 34 were wrong, 14 were unsupported. I didn’t read it in time, neither did I wait for the decision. So days later I merged a PR fixing the wrongful & unsupported claims.
This is what happens nowadays in your AI relying workflows. Agents will deny you 3 times, like Peter, until they finally decide to denounce their mistakes.
Multiplayer half-worked this time. I saw the issue before Taras, but I fell into the same trap (by myself) of not deep diving into the research until it was too late. That’s how smart or manipulative your AI is nowadays.
First: Multiplayer as afterthought
Everyone tells you they are multiplayer, they are not, and it’s not on them. In fact, only 11.3% of AI written PRs involve more than 1 human. For us, that number is worse, in the last week, only 9 out of 141 threads were shared. That is a slim ~6.4%.
We are lucky, agent-swarm has built-in a lot of failsafes to ensure collaboration. However, humans are still fixated on working with machines as individual workstations. That’s the real challenge. We have transcended AI as a technological aid, moved towards AI as an assistant, and now we need to move to AI as a separate team. You don’t work in private with a separate team. That’s the mind shift most are not making right now.
“We thought PostHog employees would default to creating sessions in team Spaces to promote transparency, but they overwhelmingly prefer to start tasks in private. We often see behaviors…”
Posthog’s on multiplayer AI
Many people are conflating two different solutions into one. They think their personal playground and the company solution need to be the same. If you believe that, then you end up with solutions like QM. Those isolate individuals and provide gated ways of collaboration. At that point you are not replacing the burden of bureaucracy (see Brooks’ law) you are actually exponentially increasing it.
“Right now, five agents can work on this codebase simultaneously. Five people can’t. That’s backwards.” - link
The interesting thing is that many still think the big advantage of AI is you don’t need to ask colleagues for information. They miss the point because humans are not good at updating context, sharing it or downloading it. So yes, AI may help you set up a github action, but it won’t know what your colleague tried before and why they decided against it. That is at best in his private session with Claude.
A year ago we realized the opportunity was in treating AI as a separate team, now we know the savings.
Second: Forcing your own hand.
Surprisingly enough, threads involving 2 people cost about ~20% less per interaction. This is only taking into account messages the AI acted upon (we disregard interactions between players), and you should know conversations run longer. This last part we are not concerned about. Usually when we loop in the other person, it’s because we consider it complex enough to need support, ie. there’s a bias.
Beyond direct costs, you also have indirect ones. Let me share one more example from this week: Caching. I found an easy bug, I wasn’t paying attention. I sent a message to the swarm “change the default to 1000”. Luckily for me, Taras was looped in. He instead proposed the correct solution. Yes I was being lazy, yes AI was being lazy too, yes my mate did the right thing. Amazing.
These first 2 are easy to think about, the third benefit is actually exponential: A multiplayer AI system solves for change management. Instead of relying on human learning & development, both notoriously slow, fragmented and time consuming, you can actually push for a system that defaults to help those interactions. We’ve seen this pain point in companies big and small, and it’s not only us: “It’s moved so fast that the human change-management part, the training and support, is lagging,” link.
Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure. Melvin Conway
This works as a direct extension of Conway’s Law. Conway understood that software coupling is enabled and encouraged by human communication. Now you need to take it to the next frontier. If software, and every task, in an organization is developed by Agents and Humans, how those communicate between each other and themselves is the key to achieve efficiency. That takes us to the third part.
It’s very tempting to think of multiplayer-ai as everyone chatting in a single thread. The true power comes, instead, when each human & agent surface is part of the same ecosystem.
It doesn’t matter where you live in (linear, slack, whatsapp, email, videcall, SalesForce or Hubspot), an ubiquitous system should keep everyone’s projects deeply intertwined, involving the right parties at the right stage.
Third: Swarms have herd mentality
If you came here to tell me “let AI talk with AI”, you haven’t used AI enough. AI has a herd mentality. It chooses a path, it propels itself down that path, and it’s not good at questioning itself. You can suggest it to do so, but as you saw on the intro, it seldom listens, and when it does, it’s usually too late.
The above is also true in agent-to-agent systems. In fact, we see this repeated in poorly designed swarm systems:
Discovery and interaction between agents is driven by task description,
Agents don’t verify a peer’s factual claim, but they do that happily with humans,
Reliability is defined by the least reliable model involved in a task.
Usually people evaluate results over system efficiency, ie. “Can it solve harder tasks?” vs “Can it solve tasks efficiently and reliably?”. This last part is changing nowadays.
For the naive eye, the above 4 points are “obvious”, however building the system that enforces (2) is not an easy task. Agents will readily lie to each other, and give up after a few tries. Even if you have your LLM-as-judge setup, if their objective is to get something done, and the system learns, iterations will make it so it passes or fails, always. To solve for this, we usually incorporate scripts, or system one models, that validate and verify tasks. A good example on how complex this could become is our code review setup, leveraging Jev, Clairic, and static code analysis before requesting an LLM review. So yes, you need to combine the three levers.
Agentic two-pizza team
If you cannot change their nature, at least contain it.
Not surprisingly, agents do interact as much as humans do, or more. In fact, Mollick is categorical: “I no longer think organizing agents is the hard part.”. I don’t agree, but I understand his point. Communication is not the problem, understanding and steering that communication is. This we have internalized without even realizing it.
For the past quarter, our number of agents stopped growing. We are at a stable 12. Other larger organizations may have ~30. This pattern comes from the learning that dividing work has diminishing returns as you scale. Yes, each agent has up to 30 instances that can run in parallel, but that’s the equivalent of you multitasking, not really of having multiple “yous”.
Yeah! Neither meat nor metal are as good as collaborating as we wished for, but if you go for each other’s strength…
The-one-thing
Getting the right person involved at the right time is what good managers do. It may create a bit more friction at the start, but they know it saves time and money down the road. It’s so well known that the Kanban, shape-up, agile and many other methodologies are only focused on optimizing that. Think about it, organization design was created because this is a fundamental optimization problem with clear ROI for a company.
AI should learn those knowledge sharing mechanisms, and it should propel them for its own benefit. Your system needs to be able to identify and loop in individuals at the right moment. Hence, our definition of multiplayer for agent-swarm.dev:
We consider multiplayer-ai those systems that enforce organizational knowledge transfer (incl. Meat & Metal proxies).
Have a great week!