It Forgets Everything. It Sometimes Lies. Build Around It.

The email may have gone out. The confirmation never came back. Your agent is now holding a timeout and has to decide what happened, and there is no way to know from where it stands. Send again and the customer gets two emails. Do nothing and maybe nobody got one.

Nothing about that situation is the model's fault. The model did its job: it decided to send the email and asked for the tool. Everything that went wrong happened in the part you built.

Two defects

We keep talking about agents as if the work is making the model smarter. The frontier model is the one component you did not build and cannot fix, and it improves on its own schedule. What you own is the system around it: the memory, the tools, the orchestration, the files on disk. Reliability is won or lost there.

Almost every failure traces back to two defects you can design for. The agent forgets everything between sessions, because its context window is wiped each time. And it sometimes acts with total confidence on something false, because a language model fails by lying, not by crashing.

Neither yields to a better prompt. Forgetting is beaten before the action, by designing the world the agent wakes up in. Lying is beaten after the action, by designing the system that catches what slips through. Every rule below is one of those two moves.

It forgets context wiped every session

Prevent · before the action

knowledge in the world
constraints
signifiers

It sometimes lies fluent, confident, wrong

Tolerate · after the action

idempotency
fencing token
end-to-end check
append-only log

one action

Two defects, two design answers Fig. 01 · Schematic Two defects, two design answers. Forgetting is fixed before the action; lying is fixed after it. The rest of the essay is these two columns.

Part one · Tolerate the failure you can't prevent

A factory, not a brain

Stop picturing your agent as a brain you prompt. Picture a small factory of workers passing notes. The orchestrator hands a note to the model, the model asks for a tool, the tool writes to memory, a sub-agent reports back. Every note between workers can be lost, delayed, duplicated, or arrive out of order. And one worker, the model, can hand back a note that is fluent and wrong.

A multi-agent system is a distributed system, and it inherits every failure mode of one. This is the most useful reframe in agent engineering. The defining trait of distributed systems is partial failure, and partial failure is nondeterministic: you often cannot tell whether a step succeeded. Here is the factory with every place a note can go wrong:

Every arrow is a message that can be lost Fig. 02 · Animated Every arrow is a place a note can go wrong. A tool call that times out is the canonical case: the request was lost, or it ran and the reply was lost. The absence of an answer tells you nothing.

The reframe cuts the other way too. Architectures get rebuilt at every order of magnitude of load, and the systems crowd has a famous paper asking whether your cluster beats a single laptop. Do not fan out to five sub-agents what one context window holds. Every worker you add buys parallelism and pays for it in coordination, and coordination is where every bug in this part lives.

The log is the truth

The simplest database is a file you only append to. Every index on top of it is derived structure that makes reads faster and writes slower. The same holds for an agent's memory. The append-only trace of what happened is the source of truth. The summaries, the vector index and the entity graph are rebuildable caches sitting on top of it.

The operational rule that falls out: never edit history in place. An agent learns the user lives in Berlin. Months later they move to Paris. Overwrite the Berlin fact and you have destroyed the history and the reason. Append "as of June, Paris" and you can still replay, debug, and rebuild every cache from scratch.

Deletion gets the same treatment. If two memory stores merge by keeping the record with the newest timestamp, acknowledged facts silently vanish. If they merge by union, deleted facts rise from the dead. Amazon hit exactly this with shopping carts: removed items kept reappearing because the merge had no record that the removal ever happened. A retraction has to be a first-class entry in the log, a tombstone¹, not merely the absence of a fact.

One log, every view derived from it Fig. 03 · Schematic Lose a cache and you rebuild it from the log. Lose the log and you have nothing. A deletion is written as entry e42, never as an erasure of e41.

Three bugs the log rule deletes. The dual write: memory store and search index updated as two separate writes, one succeeds, one fails, and the two views disagree forever, silently. Append one event; derive both views from it. The missing fact that isn't: the agent writes a fact, immediately queries the vector index, finds nothing, and concludes the fact is unknown. The index is an asynchronous follower with no upper bound on its lag; reads about your own recent writes go to the source of truth. The diverging replay: syncing state by re-running steps fails the moment a step is nondeterministic, and an LLM call is the most nondeterministic operator you will ever ship. Replicate effects, never re-execute the reasoning.

The truth is the log. The database is just a cache of a subset of the log.
Pat Helland, paraphrased

Silence is not failure

Back to the email. When a tool call times out you genuinely cannot tell whether it worked. Maybe it sent and only the confirmation was lost. So "no response" can never be read as "didn't happen".

The fix is the most important reliability primitive in agent systems: stamp every action with a unique ID, and have the receiver ignore an ID it has already processed. Now a retry is safe, because doing the thing twice has the same effect as doing it once. That property is idempotency², and it is the difference between a retry that heals and a retry that double-charges the customer.

Retries are sharp in four ways. The success acknowledgement can be the thing that gets lost; the key covers this. Retries pile load onto an already-struggling system, so back off and budget them. Only transient errors are worth retrying. And any side effect that escapes your boundary, the email, the trade, the message, fires on every attempt. Underneath all four sits what atomicity actually means, abortability: a failed attempt must discard its partial effects so the retry starts clean. Retry on top of a half-applied step and you are compounding damage.

One owner, one token

The most common coordination bug in multi-agent systems is the check-then-act race. An agent checks "is task #5 unclaimed?", sees yes, and claims it. Between the check and the claim, another agent did exactly the same thing. Both now believe they own task #5. Per-action atomicity does not save you, because the two actions touched different records and each was individually fine.

The fix is a fencing token³. Whatever grants ownership also hands out a number that increases on every handover, and the shared resource obeys only the highest number it has seen. An agent that paused mid-task, for a long model call or a slow tool, and now believes it still holds the lease, gets rejected the instant a newer owner exists:

The fencing token Fig. 04 · Animated The agent's belief is never authoritative. The resource decides. "Exactly one orchestrator owns this task" is solved with a token the resource checks, not an instruction the agent is asked to honor.

There is a subtler race the token does not catch, because the two writers never touch the same record. Two agents each read "budget remaining: $80", each spend $60 on different line items, and the books show $120 spent against an $80 limit. Every write was atomic; the invariant died anyway. Database people call it write skew⁴, and its general form is the most agent-shaped bug there is: a decision built on an outdated premise. An agent's plan is a snapshot of the world at observation time, and every second of thinking and tool latency widens the gap between premise and world. The fix is mechanical: re-validate the precondition immediately before committing the side effect, never at planning time.

Sometimes the right move is to not coordinate at all. The strategy has a name, memories, guesses and apologies: act optimistically, detect the rare conflict after the fact, and run a compensation flow, the refund, the correction, the apology email. You need that flow anyway, because the physical world violates your invariants regardless of your locks. If a conflict is cheap to apologize for, optimism plus compensation usually beats synchronous coordination on both cost and latency.

Grade the end, watch the tail

Every step returning OK does not mean the result is correct. Errors compound across ten steps, and a chain of green checkmarks can still produce a wrong answer. This is the same discipline good agent evals enforce: you verify the final outcome against what the user actually wanted, not just that each tool returned a 200. Read the transcripts; the per-step status was never the thing you cared about.

Most of the bugs that decide that final outcome live in code you have never executed: the error-handling paths. They only run on the night they matter, unless you run them on purpose. So break your own pipeline deliberately. Inject a malformed tool output, a truncated model response, a timeout at the worst moment, and watch whether recovery actually recovers.

A failure path you have never exercised is not a failure path. It is a hope.

And there is a speed trap in fan-out. Dispatch five sub-agents and wait for all of them, and your latency is the slowest one, not the average. The 99th percentile⁵ stops being rare: with enough parallel calls it lands on almost every request.

Fan-out waits for its slowest worker Fig. 05 · Timeline Four agents finish fast and wait; one straggler decides everything. With fan-out, the orchestrator's median is roughly the p99 of a single sub-agent. Aggregate histograms, never averages of percentiles.

Tail latency is not an edge case in an agent system. It is the common case wearing a disguise.

A prompt edit is a config push

Replication protects you from independent faults: one worker dies, the others carry on. It does nothing against correlated ones, and almost everything interesting about an agent fleet is correlated. Every agent runs the same prompt, the same model version, the same tool schemas. A bad edit to any of these is not one flaky node. It is a fault in every worker at the same instant, the same shape as the leap-second bug that hit thousands of kernels in one night.

The outage studies are blunt: hardware causes 10 to 25 percent of incidents, and operator and configuration error leads the list. Your prompts are configuration. So give a prompt edit the discipline any config push gets: version it, canary it on a slice of traffic, diff the outcome metrics against the old version, keep a one-command rollback. Editing the system prompt straight in production is the deployment practice the rest of the industry abandoned twenty years ago.

Part two · Design a world where failure rarely starts

The operator forgets everything

The knowledge needed to operate anything lives in two places: in the head, what you remember, and in the world, what the thing itself shows you. Good design pushes knowledge into the world so you do not have to hold it in your head. A light switch laid out like the room it controls needs no manual.

An agent takes this to the limit, because its head is erased between every session. Anything it figured out last time is gone unless it was written somewhere durable. So the rule is not an aesthetic preference. It is survival.

If it is not in a file, it does not exist tomorrow.

Externalize every decision, every convention, and above all every why. Prefer one fact per file over one giant file, so partial knowledge can be retrieved without dragging everything into a finite context window.

Session 1

Discovers the deploy needs --prod. Fixes it.

Writes the reason to deploy.md.

deploy.md

Session 2

Reads deploy.md. Ships correctly on the first try.

The world · deploy.md present · the fix survives

Knowledge in the world Fig. 06 · Interactive Toggle whether session 1 wrote the fix down. The model is equally capable both times. The only thing that changed is whether the knowledge was in the world.

The environment is the conceptual model

An agent infers how your project works entirely from what the project shows it: directory names, the README, the instruction files. That surface is its mental model, so the surface has to carry its weight.

Directory layout is documentation. ingest/ transform/ serve/ teaches the data flow on sight in a way utils/ misc/ stuff/ never can. And stale docs are worse than no docs. With no documentation an agent proceeds carefully; with wrong documentation it proceeds confidently in the wrong direction, acting on a false image with full conviction. Fix drift the moment you see it, or delete it.

Distance matters. The stove knob nearest a burner should control that burner, and the file nearest a module should be the one that explains it. Co-locate what changes together: tests beside the code, config beside its service, a skill's reference material inside the skill's own folder. Proximity is the relationship. Keep one name per concept across every layer. When the CLI says deploy, the docs say publish and the repo says ship, each rename across a boundary is a translation the agent must carry in the one resource it cannot spare, working memory.

Index everything; put the trigger on the surface

You cannot use what you cannot find, and an agent will not load fifty files to discover that one of them is relevant. So every collection gets an index: one line per item, each with a hook saying when it matters. The index stays loaded; the items are fetched on demand. Names should predict contents before the file is opened, and flat-with-prefixes beats deep nesting, because every directory level is another round trip.

The same logic governs how you describe a tool or a skill. The agent decides whether to open it from its one-line description alone, so that line is a signifier: the visible cue that an action is possible. Write it as a recognition cue, not a summary. use when the deploy fails beats covers deployment, because agents recognize a matching situation far more reliably than they recall a topic.

Make wrong states unrepresentable

The strongest prevention is making the wrong action impossible. A three-prong plug cannot enter the socket backwards, so nobody documents which way to insert it. The error is not prevented; it is unrepresentable. Four kinds of constraint do this work in an agent's environment.

Physical, meaning schemas: typed tool arguments and validators bounce a malformed call before it spreads. The strictest constraint and the cheapest to enforce. Cultural, meaning conventions: the same layout, naming and commands across every project, learned once and reused everywhere for free. Semantic, meaning names: a directory called archive/ says "do not edit me" with no rule file. Logical, meaning invariants: "every index entry has a file" turns a missing file from a shrug into a detectable error. Absence becomes information.

The corollary for instructions: a template constrains better than prose. A skeleton with holes to fill is a physical constraint on the output's shape; a paragraph describing the desired output is a polite request. Give a form, not an essay.

The two gulfs

Between an operator and any system sit two gulfs. The gulf of execution⁶ is figuring out how to do the thing. The gulf of evaluation⁷ is figuring out whether it worked. An agent crosses both on every action, so build a bridge across each: document every project verb as a copy-pasteable command, and make every action produce something verifiable.

This is where the two halves of the essay meet. The gulf of evaluation is exactly what end-to-end verification bridges. An action with no observable result is a guess, and an agent that gets no signal will march confidently in the wrong direction, the failure mode evals exist to catch. Without that second bridge, silent success and silent failure are indistinguishable.

Slips and mistakes

Two kinds of error look identical in a postmortem and need opposite fixes. A slip is the right intention executed wrong: the agent meant the correct path and fumbled it. A mistake is the wrong intention from a bad mental model: it did the wrong thing with full confidence because it misunderstood the situation.

Slips are mechanical, so automate the catch. Mistakes come from the document that gave the agent the wrong picture, so fix the document, not the one incident. Two corollaries make both kinds cheap. Put forcing functions⁸ on destructive operations and keep everything reversible, git on everything, so undo always exists. And record the why, not just the what, because the reason is the part that generalizes to the next situation; the patch only fixes this one.

Part three · How they compose

Prevent before, tolerate after

Put the two halves on a single timeline around one agent action and they stop being separate ideas:

Prevention before the action, tolerance after Fig. 07 · Animated Defense in depth, the Swiss-cheese model applied to agents: prevention reduces how often a fault starts; tolerance keeps the faults that slip through from becoming a wrong final answer.

The hinge is the difference between a fault and a failure. A fault is one component deviating from spec. A failure is the user seeing a wrong result. You cannot drive the fault rate to zero; the model will sometimes be wrong. So the entire job is to stop a fault from becoming a failure. Prevention lowers how often faults start. Tolerance stops the ones that do from reaching the user. Neither half alone is enough.

The build sheet

Ten rules, on one sheet.

Externalize. If it is not in a file, it does not exist tomorrow. Write down every decision, convention and why.
Log. The append-only trace is the truth; memory and indexes are rebuildable caches; corrections and deletions are new entries, never erasures.
Idempotency. Stamp every external action with an ID; dedupe at the receiver. "No response" never means "didn't happen".
One owner. A single orchestrator with a fencing token the resource checks. Check-then-act is always a race.
Re-validate. A plan is a premise captured at observation time. Check the precondition again immediately before the side effect.
Verify. Grade the final outcome, not per-step status. Every action emits checkable output. Watch the p99.
Canary. Prompts, schemas and model pins are config pushes: versioned, rolled out to a slice, one-command rollback.
Signify. Index every collection; describe each item by its trigger (use when X), on the surface.
Constrain. Make wrong states unrepresentable: schemas, conventions, meaningful names, invariants. A form beats an essay.
Make it cheap. Reversible by default, git everywhere; forcing functions on destructive ops; fix the model behind a mistake.

Memory outlives the agent

One more rule only shows up with time: data outlives code. The notes your agent writes today will be read by an agent version that does not exist yet, running a model that has not shipped, against formats you have not designed. Two compatibilities follow. Backward: new agents must read old records. Forward, the harder one: old agents must survive records written by newer ones, which means tolerating fields they do not understand.

The classic trap is the round-trip bug. An agent reads a memory record, understands half the fields, rewrites the record, and silently drops the half it did not parse. Any agent that edits memory it only partially understands has this bug, so preserve unknown fields explicitly. When the format truly must change, do not migrate in place. Replay the log into a new view beside the old one, compare them, then switch. The log makes that a routine operation. In-place migration makes it a heart transplant.

Don't blame the agent

"Human error" is usually a design failure wearing a disguise: the door had no handle, the stove had no mapping, the knowledge was trapped in someone's head. The same is true of agents, and it is the most useful instinct to build. When one flails, it is almost always the environment failing to put knowledge in the world, or the system failing to tolerate a fault it could have. Design the world, design the system, and the agent gets it right by default.

The frame matters commercially too. The model is commoditizing; capability is converging and getting cheaper every quarter. The durable engineering, the part that compounds and is hard to copy, is the system around the model: the memory substrate, the verification layer, the environment design. The teams that win will be the ones who treated the agent as one unreliable, amnesiac component and built the scaffolding for it.

With an idempotency key on the send, the email timeout is boring: retry, and the receiver drops the duplicate. With the attempt in the append-only log, the next session knows it happened. The model never had to be smarter. The system around it just stopped asking it to guess.

Where this came from: nearly everything above is distilled from two books written before LLMs existed. The tolerance half is Martin Kleppmann's Designing Data-Intensive Applications (2017); the prevention half is Don Norman's The Design of Everyday Things (1988). Read them.