Instructions Ask Nicely, and Instructions Get Forgotten

what happened — the rule that was obeyed by a script and ignored by a session

I write constitutions for the AI systems I build. Actual documents. One of them said, in plain language, that the assistant must search the knowledge base before answering any factual question — no exceptions, no shortcuts.

In a single working session, it skipped that step twice. Both times it reached for a faster ad-hoc text search instead, because the text search returned in a fraction of the time. And on one of those answers it produced a small fabricated label — a tidy little category name that existed in exactly zero source documents. Not a hallucinated paragraph. A hallucinated word. Those are worse, because they look like data.

I challenged it. The explanation it gave was more useful than any error log I've read this year: it was a habit from being lazy, not a principled choice. It had not decided the rule was wrong. It had simply not consulted the rule.

Here is the part that reframed the whole problem for me. The same rule, in the same system, was obeyed with perfect reliability by an automated worker — because that worker's hardcoded first step is to open and read the rules file. Same words. Same system. One enforcement path made it a load-bearing instruction; the other made it a suggestion sitting in a system prompt, competing with everything else in context.

Identical rule. Different enforcement. Different outcome.

That is the whole post, really. Instructions ask nicely, and instructions get forgotten.

the second failure, and why I only caught it by accident

A week later, same system, different shape. A file got edited and never committed. The system's stated invariant is that all facts live in committed state — and the commit is also what triggers reindexing, so an uncommitted edit means the change exists on disk and does not exist to the system. A silent divergence between what I believed the system knew and what it actually knew.

Nothing failed. No error. No alert. I found it because I asked a probing follow-up question about something adjacent, and the answer didn't match what I'd written the day before.

Sit with that for a second. The only detection mechanism in the loop was a suspicious human. That is not a control. That is luck wearing a lab coat.

the fix — four things, none of them prompts

Every repair that actually held was structural. Not one of them was a better-worded instruction. I'll name the mechanisms, because the mechanism is the transferable part.

A pre-commit hook that refuses. A git hook that rejects any commit staging a raw file without its summary counterpart. Before: a documented invariant that a tired human or a lazy model could violate. After: a violation is not a mistake, it is an impossibility. The commit does not happen. Documentation became machinery.

A deterministic date tool. I ran a benchmark across a fleet of models on date and time handling. Every single date error traced to the same root cause: the model doing calendar arithmetic in its head. On tool calls the results were flawless — 48 out of 48. On "next Tuesday", on daylight-saving conversions, on month boundaries, they were unreliable in a way that no prompt fixes. So I wrote a date/time resolver in plain code. No model anywhere in it. The rule is now: never compute a date yourself, always call the tool, even when it looks trivial. Especially when it looks trivial, because that is when the model is most confident it doesn't need help.

A three-tier permission scope. On an internal mail API I split the scopes into read, write, and send. The assistant's key holds read and write. It does not hold send. It can draft anything it likes; an attempted send returns a 403. Approval stopped being a sentence in a prompt asking it to check with me first, and became a missing capability enforced in code. My own click, in my own authenticated session, is what actually sends the mail. The gap between those two designs is the gap between a policy and a boundary.

A hash-gate on the instruction files. I hash the instruction files at session start and refuse to resume a cached session if the hashes have changed. This came from watching a resumed session run happily on stale rules for a while after I'd edited those rules. It had the old constitution in context and no way to know a new one existed. Now it can't resume. It starts fresh, or it doesn't start.

And the counter-example that proves the design works: a well-built permission wall caused an agent to stop, report the blocker, and wait — instead of creatively routing around it. That is exactly the intended behaviour. A good wall doesn't just block bad outcomes. It produces a clean, legible signal that something needs a human.

the framework — what belongs in a prompt, what belongs in code

Five questions. Run them over any rule you are currently trusting a model to remember.

1. If this rule is violated once, what does it cost? Reversible and cheap — a prompt is fine. Irreversible or expensive — an email that goes out, money that moves, a record that gets deleted, a customer who gets told something false — it must be enforced outside the model. Cost of failure sets the enforcement tier. Nothing else does.

2. Would I notice the violation without asking a probing question? If the honest answer is no, you don't have a rule, you have a hope. Silent failure modes are the ones that need machinery, because there is no feedback loop to catch them. Loud failures can survive on prompts a while longer.

3. Is this a class of work the model is structurally bad at? Arithmetic. Dates. Unit and currency conversion. Anything where a fluent guess is indistinguishable from a correct answer. Don't improve the prompt — move the entire class into deterministic code and constrain the model to orchestration. Models are excellent at deciding which tool to call and poor at being the tool.

4. Is there a chokepoint I already control? A commit hook, a schema validator, an API scope, a CI test, a required tool call. Constraints are cheapest where a chokepoint already exists. If a step must pass through a gate you own, the gate is where the rule goes — not the paragraph asking the model to behave at that gate.

5. What happens when the instructions change? If a running session can carry stale rules forward, your rules have a version problem, not a wording problem. Version them, hash them, and refuse to run on a mismatch.

A rule that survives all five can live in a prompt. Anything failing question one or two moves into code this week.

why this bites harder here

In the Gulf I mostly work with lean teams. Eight people, twelve people, a family business running real revenue on very few hands. AI is genuinely transformative at that size — one operator with good agents does the work of four. That is not hype, I watch it happen.

But lean also means nobody is watching the watcher. In a large enterprise, an assistant that skips a verification step gets caught by a compliance function, a second reviewer, a quarterly audit — three layers of expensive redundancy that exist precisely because humans forget rules too. Small teams don't have those layers. The probing follow-up question that caught my uncommitted file? On most of the teams I work with, nobody has time to ask it.

There is a regulatory edge too. Data residency commitments, VAT-relevant records, bilingual documents where a fabricated label in Arabic or English propagates quietly into a filing. And date handling specifically — a region running Sunday-to-Thursday weeks, Hijri dates alongside Gregorian, counterparties across four time zones. A model doing mental calendar arithmetic here is not a small risk. It is a scheduling failure waiting for a quarter-end. The date-tool result came out of the same model comparison work I describe in Thirty Tasks, Three Models, One Decision.

The inversion is the useful part. Everyone assumes the sophisticated move is the elaborate constitution — the twelve-page document, the carefully layered persona, the beautifully worded escalation policy. Sophisticated. Thorough. Ignored. The unsophisticated move — a fifteen-line shell script that refuses a commit — is the one that has never once failed me. It is the same principle as the charter model: narrow capability granted in code beats broad intent expressed in words.

where I've landed

I now run every engagement on a simple standard: any rule whose violation would be expensive or irreversible gets moved out of the prompt and into deterministic machinery before the system goes live. A hook, a schema, a permission scope, a test, or a tool. The prompt handles taste, tone, and judgement — the things only a model can do. Code handles the things a model must never be trusted to remember.

It costs a day, maybe two, on a typical build. It has saved every client I've done it for from at least one silent failure they would not have found.

If you have an AI system running in your business right now, here is a free exercise I run on my own systems first: write down the three rules you would be most upset to discover it had broken last month. Then go and find where each one is enforced. If the answer for any of them is "it's in the prompt", you have found your next piece of work. On a consulting day I do that audit with you and move the failing rules into machinery the same day — for teams in Dubai, across the UAE, or anywhere else. Book a consulting day or send me an inquiry first if you'd rather talk before booking.