Chapter 16 · Mechanism
Part 3 · Teaching it to remember17 of 35
Why our AI agent got stuck in a loop, and the context fix
Asked to move a workout plan by a week, our assistant said 'let me do it now' hundreds of times and never did it. The model was fine. The chat engine kept feeding it its own dithering.
The Margin4 min read
Our founder asked the assistant for something a person would do in two minutes: "the workout plan on this board is overdue, shift it to start this coming Monday."
What happened next belongs in a museum of AI failure. The assistant read the board. Then it read every column. Then every card. Then it read the same 1,700-character card again, in pages, roughly forty times. And then it began to talk to itself:
Let me update the card now. Let me also check the Programming card. Actually, let me stop overthinking and just do it. OK, I'll make the calls now. Let me proceed. For real this time. Let me do it now.
It said some version of "let me do it now" hundreds of times. We counted. It never did it. It burned its entire turn budget deliberating, asked to continue, and deliberated again.
One overdue workout plan
~40
1,700
0
The founder killed the conversation and dropped one line in our channel.
This all just feels from the stone age.
The wrong first suspect#
The obvious read was that the conversational assistant ran on a small, fast model, and a bigger brain wouldn't dither like this. Upgrade the model, move on.
That was wrong, and the founder caught it. The same small model writes the most impressive prose in our product: the dossiers the Mind builds about the people in your life, the ones that correctly narrate a marriage, flag a roleplay scene as fiction, and admit the limits of their own evidence. Same model. Excellent output. So what's different?
What we feed it.
When the Mind writes a dossier, the model gets a curated evidence bundle: every layer of knowledge, labeled, deduplicated, budgeted. We built that machinery because we'd already learned the lesson once. The fix is never a smarter prompt or a bigger model. The fix is better evidence flow.
Why it looped#
Chat never got that lesson. It did what nearly every chat engine does: it replayed the raw, accumulating transcript back to the model every round. So every "let me do it now" the model produced was appended to its own context and fed straight back to it. Within a few rounds the model's entire world was its own dithering.
Language models continue patterns. That's the whole job. Feed a model a context dominated by careful evidence and it continues the evidence. Feed it forty near-identical page-reads and a wall of "let me proceed," and the most statistically ordinary next sentence in the universe is "let me proceed." The engine didn't fail to stop the loop. The engine built the loop, one appended round at a time, and sat the model down inside it.
A bigger model resists that trap longer. It doesn't deserve credit for surviving bad plumbing, and we'd rather fix the plumbing than rent a stronger swimmer.
Conversation gets the same spine as everything else#
The rest of the Mind had already grown up. Narratives trace to evidence. Facts carry tense. Sealed content stays sealed. Chat was the last component still living in the era before all of that, so it got the same overhaul, in four moves.
Four moves
- A working state instead of a transcript
- the engine keeps a compact running state across a turn: the goal, the facts gathered so far, the decisions made, the next intended action. Each round the model sees the state plus the last few exchanges, never the whole raw history, so its own past deliberations never re-enter its context.
- Reads become a ledger
- once a card's content is captured in the state, older reads shrink to a one-line receipt, and identical reads in a row collapse into one. The model can no longer build its context out of forty copies of the same card.
- Guards
- a model that keeps planning without acting gets one nudge, act or answer. One that starts repeating itself gets stopped. Either way the turn ends honestly, with whatever it actually did summarized for the user.
- Turns that outlive your tab
- each turn runs as a durable background job with its own event log. Close the laptop, lock the phone, lose the connection: the work continues, and when you come back the thread replays what happened while you were gone.
The part that surprised us#
Durable conversations raised a fair question, from the founder himself: if every thread is now a permanent, resumable record, does the Mind's episodic memory (its memory of past conversations) become useless?
It's the opposite. Durable threads make episodic memory more useful, because its job was never "resume where we left off." Its job is to be the searchable memory of your conversations. The distillate that turns a hundred threads into a handful of durable facts. The place where "we decided to move the launch" gets caught the moment it's said in chat, weeks before it's written on any card. The thread is the transcript; episodic memory is what the transcript meant.
We nearly deleted a feature because a better one arrived, the same week we nearly blamed the model for the chat engine's plumbing.
The founder's line stuck, because it was right about more than chat: the weakest engine in a product sets the impression of the whole thing. "Stone age" was accurate, and it only stung because everything around it had already moved on. The fix that week didn't touch the model. We just stopped pouring its own confusion back into it.