The trap: we automated coding, not quality

How observability, QA, monitoring, should look like if your team doesn’t know the code.

tl;dr: Software Control solutions need to evolve to master business level communication. Most of the nitty gritty details – events, debug logs, assertion, traces, etc — will be consumed by LLMs soon enough, anyways.

Who will be able to abstract complexity?

Before we start, let’s make sure we agree in the following statements:

1. Smaller tech teams are producing more complex software solutions,
   This has been true for the past decades. In the 90s you needed thousands of engineers to build a marketplace, nowadays with shopify and webflow, everyone can do it instantly.
   This is the biggest software promise: never doing the same thing twice.

2. The technical acumen required to develop software is trending downwards,
   Again, a corollary of the above, and something that simply got accelerated in the past couple of years. You can now build your own app without even knowing what coding is, isn’t that great?

3. Software solutions are evolving faster, and becoming less deterministic,
   This is new. Until recently, the nature of any no-code/low-code solution was extremely limiting. You could start a project with them, and as soon as you needed to take it beyond the ‘standard path’, you would need to call an expert.
   A great example of this is the whole DevOps/RevOps ecosystem around SalesForce(SFDC). For those companies using SFDC heavily, it’s usually a good idea to use providers or define a full software development life cycle (SDLC) around that supposedly no-code platform. In fact, they also develop custom plugins and abstraction layers on top of the existing solution.
   
   Tools like Replit, v0, lovable, base44, etc. promise you can actually create what you like, anyway you want, without that burden…

4. It’s a trap!
   The essential complexity of a running software solution is ensuring your software is reliable, accurate, secure, and performant. That to say, we know issues arise from ill conceived software, or users doing the unexpected, or providers failing, etc. Assuming that ‘generating a solution’ is equivalent to ‘generating the right solution’ or even worse ‘generating a production ready solution’ is the trap. And that is the challenge that every mature software faces.
   
   In this article, I’ll use software control, SC, in lieu of observability, monitoring, QA, etc.

If you agree on the above, then here I’ll discuss:

1. First: The trap, how generating “accurate” code is fixed to a point in time.

2. Second: Before, why we built solutions that now are intrinsically outdated,

3. Third: After, how to build towards a holistic approach.

If you don’t agree with my points above, challenge me in the comments. I love when proven wrong.

First: The Trap

“I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation.” - Frederick Brooks.

This is the trap, and we all fall into it so often, and in such unexpected ways, that we have built professions around it. We first believed translating requirements to code was the challenge, and we have been proven wrong every time we see those that choose the right problem succeed. We still believe Quality Control, Observability, and Monitoring (Software Control) to be challenges, when those are the only mechanisms at our disposal to verify the fidelity of our generated code. And those need to change.

By falling into this trap, we long ignore the inherent properties of the irreducible essence of software systems: complexity, conformity, changeability, and invisibility. Particularly, conformity and changeability force software to be like Proteus, condemned to cycle through endless versions eternally to please its masters. Specifically:

1. Conformity is defined by the human desire for such a solution, and how that changes through time and as we have more individuals using our software. The Kano Model being the ultimate example of this time affectation, we simply get used to features, and we usually experiment with software to give it new meaning and uses.

2. Changeability, to be honest, conformity wouldn’t be such a big deal, if changeability wasn’t there. Most human products evolve, however simply the investment required to change serves to dampen the whims of the changers. It’s because software allows for fast, seamless, iterations, that it creates an expectation of constant improvement. Forty (40) years ago, this was reserved for highly advanced software, nowadays it’s the only way to have a competing product that provides a solution in any industry.

Like a shark that is not moving, if your software is not evolving, then it’s dead.

Second: Building the train vs running the train

We have discussed two seemingly contradictory truths above:
(a) software changes constantly, hence validation is a key challenge, and
(b) testing the fidelity of a software representation is not an essential part of software.

On the surface, they seem to be dichotomous, however the explanation is simpler than expected: validation in (a) refers to the conceptual construct, whereas (b) is directly talking about the systems you need to have in place to verify a solution is worked as defined. Unfortunately we haven’t found a way to do (a) without (b), as humans or machines need to verify a conceptual construct by seeing it behave in the wild. Let me repeat it:

As humans/machines, we lack the power to ensure descriptions of conceptual constructs (ideas) are accurate enough to create & iterate software generated from them.

Observing is the only effective way we have found to partially address the problem. We believe something works well if we see the system operate under specific conditions for a period of time. That’s all that we know in this area, and it’s very little.

In fact, QA (testing, quality control, unit, integration, fuzzing, end-to-end, load, synthetic monitoring, exploratory, etc) is centered on artificially generating scenarios that we believe would happen in the wild, so that we can observe those to refine our conceptual constructs. The human struggle to prove ‘absence’ is so deeply ingrained, that we are forced to have systems surveilling our software in real-time.

As an example, when you go through a driver license exam they check you know how to drive. They don’t try to corroborate that you won’t ever get into traffic accidents. The first one is what vibe-coding apps are helping with, the latter is what we need to evolve towards, and its complexity is terrifying.

Third: QA/Observability for vibecoders

Software Control solutions are a mess.

Check your APM dashboard and ask yourself if you would ever use something like this in a team of 3. We can do better, we deserve it. They are fragmented, expensive, and terribly complex.

The good news is that with this latest fundamental shift we can re-invent them, due to:

1. Including extra code to observe your system is now ‘free’.
   With AI generating code you can be explicit about anything, e.g. with a simple command you could ensure all software includes multiple calls to different Software Control (SC - ie. QA, Monitoring, Observability, etc) SDKs or services with high reliance and autonomy. This means, you can now invest heavily in an area that traditionally was an afterthought, and a privilege of large organizations.

2. You can ‘safely’ reduce the noise.
   Your agents should be able to understand the first layer of output from any SC solution, and make a first attempt to reflect on the meaning of such results. That’s why so many people believe it’s the end of dashboards. Agents can tell you what’s a real problem, and where to look into first.

3. Computing is sufficiently cheap => cheaper than human labor
   Now you can run swarms of agents to do all your Quality Control, they can try to reach a confidence level that would be close to that ‘absence’ above, in a time/cost efficient manner.

The real challenge here will be the UX.

The mus would be a UX for non-tech users that will allow them to understand what’s happening without getting into traces, resources, and YAMLs. They won’t have the team to understand those either, and if you were to facilitate such information, it would be simply copied & pasted into Claude. We need solutions to feel closer to an ERP or a CRM, a place where you define what matters for your business, and you take small controlled actions to course correct or enable for improvement.

I’m going to build it. You can do it with me, or watch me do it, but this is happening, it needs to be done, and it’s a lot of fun.

the one thing

While companies reduce in size, CTO & C(x)O roles continue to merge. Individuals wearing the hat of Tech together with Marketing, Product, People, Operations, etc, are not an exception anymore. This was happening even before LLMs, and it has accelerated. Software has become a core competency.

Software Control solutions need to leverage this, they need to get closer to a Company mission control center, than to a Tech solution. They need to start speaking business so that your company sees, understands and decides where to invest their time and resources next. That’s where the ROI will be clear.

It’s time to start considering how your QA/Observability/SC solution can be understood & used by any CEO.

References

1. Domain-Oriented Observability - Martin Fowler

2. Continuous Delivery - Martin Fowler

3. Software Delivery Guide - Martin Fowler

4. Testing Guide - Martin Fowler

5. Continuous Delivery Tag - Martin Fowler

6. Do Things that Don’t Scale - Paul Graham

7. Tweet Thread: Automating Software Engineering - Andrej Karpathy

8. Latent Space: Software 3.0 Deep Dive - Latent Space - Andrej Karpathy

9. Kyle Howells’ Summary - Kyle Howells

10. charity.wtf Blog - Charity Majors

11. The Pragmatic Engineer Podcast - Charity Majors

12. Refactoring Podcast - Charity Majors

13. The New Stack: Testing in Production - Charity Majors

14. Observability Engineering - Charity Majors, Liz Fong-Jones, George Miranda

15. Two Person Teams - Jason Fried (REWORK Podcast)

16. “Just the two of us” - Jason Fried

17. Episode 132: Simplicity by Design - Jason Fried (Product Thinking Podcast)

18. The Three Ways: Principles Underpinning DevOps - Gene Kim

19. Moonshots Podcast: Best Startup Ideas from Paul Graham - Moonshots