Auditing my own AI coding harness for enforcement

In 2025 I wrote an AI coding harness for a production codebase with a real product running on it, and at the time I believed it was state of the art. Earlier this year I went back and audited it, because I was getting frustrated: the agents kept making the same mistake again and again, and I wanted to understand why. So I sat down and asked one question of the harness - how many of these gates are mechanical, and how many are probabilistic? That is where I started to be honest with myself.

As I went through each of the skills and all of the convention rules, everything fell on the probabilistic side. There were no mechanical checks at all. Nothing in the harness could actually block anything, and that is when it hit me that I had been doing it wrong all along.

What the harness actually did

The harness itself was substantial. It took a raw idea through a requirements interview to flesh out the exact product requirements, wrote the stories from them, did a technical pass, and broke the stories down so that every ticket handed to an agent was verifiable through its acceptance criteria. Some tickets ran sequentially, some in parallel, depending on the decomposition. And still, the agent would sometimes ignore the conventions and go write the same things we had asked it not to write in the codebase.

Our answer, every time, was more prose. The conventions file grew from around 5,000 lines to 12,000 lines over time, as we found real issues with the agent’s behaviour. We kept adding to it in the hope that it would steer the agent better, but that did not happen; the agent kept making the same mistakes. The part I would rather not write down is that we knew context engineering matters, and we kept adding rules and rules to a conventions folder anyway. I knew better while I was doing it.

Why prose could not enforce the rules

The model is probabilistic. It is always a maybe with the model whether it reads a given rule or not. With 12,000 lines, most of the time it misses some of them - the research on long context suggests models attend to the top and the bottom of what you give them, and the messy middle is largely lost (Lost in the Middle, Liu et al.).

The obvious response is that 12,000 lines is the problem, and that a 300-line rules file would work fine. But my root file was already about 300 lines. It only orients the agent to the relevant areas of the repos, and different skills load different context for different tasks - I had thought of it as a progressive realization of context, loading only what a task needed. Even with that, the model kept violating the rules. And the 12,000 lines could not simply be cut down. These were real issues, found the hard way. They had to be followed.

The router only fetched what the prompt already knew to ask for

I even put a router in between - a small agent that would fetch the specific sections of the conventions relevant to the task at hand. It did not save me, and the reason is the part that taught me something. During the audit I underspecified a task on purpose, to see what the router would do. It never looked for the right convention at all. A router can only retrieve what the prompt already knows to ask for; when the prompt does not know a rule exists, the router has nothing to go looking for. There was never an enforcement mechanism actually available to the model.

Where the line between prose and code sits

This is the split I came out of the audit with. Any rule you can write as code - a lint rule, a dependency check, anything codeable - goes into a sensor. Anything you cannot specify through code, that can only be decided by judgement, belongs in prose, kept as short as you can make it. This guides-and-sensors way of splitting the two is Birgitte Böckeler’s, and it matched what I was seeing on my own codebase.

Two examples from my own conventions make the line concrete. We had all constants defined in a single file, with every other module importing from there. That is a simple grep - a lint rule can enforce it exactly. I had it sitting in prose instead, and the model would sometimes follow it and sometimes not; it never needed to be prose at all. On the other side, we had a convention that simple queries can live in the facade layer while complex queries always go in the service files. What counts as “simple” is a mental model every developer forms over time. No lint rule can draw that line, so it stays in prose, with examples.

What the sensors do when the agent commits

It helps to separate the two roles. The rules, conventions, and skills are guidance: they steer the agent toward the right way of doing things. The mechanical rules - the lint rules, the dependency checkers, the structural checks - are the sensors. The guidance steers, but because the model is probabilistic it will not follow everything exactly. The sensors are what fire when the agent tries to commit, and once a sensor fires, the model finally has something concrete to react to.

I tested this on the same codebase, with the same prompts, hooks on and hooks off. With the hooks off, I got the same repeated violations as before. With the hooks on, the agent tried to commit, hit a sensor, saw the problem, understood the actual issue, went back and fixed it, came back, committed, and pushed; when it hit another sensor it ran the loop again. It behaves like the red-to-green rhythm of test-driven development, except the failing test is your own convention. I should be honest that this is a small N - five or six runs - so all I will claim is that it worked in those runs.

Why the enforcement tooling was there all along

Everything I’ve described runs on tooling that has existed for a decade or more: pre-commit hooks, lint rules, dependency checkers, required checks in CI. None of it is new, which raised the question I had to sit with. If the enforcement layer was always available, why had nobody wired it up - me included?

The answer, I think, is that it was never actually required. Humans wrote the code, and humans knew what to do from the code already in the repo. They read it, understood it, and formed a mental model of the standard over time. We never told a developer “you need to do this” more than once.

Agents changed that. An agent starts every session from a fresh state. It has no mental model of the repo, and it does not build one; each new session starts from scratch. So the mental model has to live somewhere the agent cannot skip past. I started to think of the enforcement layer as compiled mental models - everything your team knows that can be written as code, compiled into gates the agent has to pass.

That is also why my own goal had moved without my noticing. In 2025 the goal was to hand over as much of a task to the agent as possible. Now it is to run long-running tasks with no intervention at all. When you are watching every step, you catch what the guidance missed. An agent running for hours on its own has nothing but the gates to catch it.

What I’d suggest, and what I still don’t know

If your team ships with agents, the audit is worth repeating on your own setup. Open it and count: how many of your gates are mechanical, and how many are probabilistic prose you have written for the agent. If, like me, you find no mechanical rules, the move is the same one I made - take every rule that is codeable and move it to the mechanical side, and let the prose carry only the judgement calls.

I want to be clear about the limits of what I found. The hooks-on result rests on five or six runs on one codebase, which is enough to convince me and not enough to prove the case. I also cannot always tell in advance which rules are genuinely judgement-only and which I simply have not worked out how to codify yet; that line probably moves as the tooling gets better. What I am confident about is the direction. On my own harness, the enforcement points were the piece that had been missing all along, and prose was never going to stand in for them.

---

P.S. This audit is one slice of the paid diagnostic I run for engineering teams: I score your AI-assisted development workflow across 20 areas against the industry frontier, from your own artifacts - rules files, sample PRs, CI config, your own numbers - and hand back where you stand against your peers and a prioritized 90-day plan. If that is worth doing on your team, it starts with a short written application, not a sales call. You can start here.