Field notes

Mechanism · Chapter 16

The assistant that talked itself in circles

Asked to move a workout plan by one week, our assistant said 'let me do it now' two hundred times and never did it. The autopsy vindicated the model we almost blamed.

Our founder asked the assistant for something a person would do in two minutes: "the workout plan on this board is overdue — shift it to start this coming Monday."

What happened next belongs in a museum of AI failure. The assistant read the board. Then it read every column. Then every card. Then it read the same 1,700-character card again, in pages, roughly forty times. And then it began to talk to itself:

Let me update the card now. Let me also check the Programming card. Actually, let me stop overthinking and just do it. OK, I'll make the calls now. Let me proceed. For real this time. Let me do it now.

It said some version of "let me do it now" — we counted — hundreds of times. It never did it. It burned its entire turn budget deliberating, asked to continue, and deliberated again. The founder killed the conversation and dropped one line in our channel: "this all just feels from the stone age."

The wrong first suspect

The obvious read was that the conversational assistant runs on a small, fast model, and a bigger brain wouldn't dither like this. Upgrade the model, move on.

That was wrong, and the founder caught it. The same small model writes the most impressive prose in our product: the dossiers the Mind builds about the people in your life, the ones that correctly narrate a marriage, flag a roleplay scene as fiction, and admit the limits of their own evidence. Same model. Excellent output. So what's different?

What we feed it.

When the Mind writes a dossier, the model gets a curated evidence bundle: every layer of knowledge, labeled, deduplicated, budgeted. We built that machinery because we'd already learned the lesson once. The fix is never a smarter prompt or a bigger model. The fix is better evidence flow.

Chat never got that lesson. It did what nearly every chat engine does: it replayed the raw, accumulating transcript back to the model every round. So every "let me do it now" the model produced was appended to its own context and fed straight back to it. Within a few rounds the model's entire world was its own dithering.

Language models continue patterns. That's the whole job. Feed a model a context dominated by careful evidence and it continues the evidence. Feed it forty near-identical page-reads and a wall of "let me proceed," and the most statistically ordinary next sentence in the universe is "let me proceed." The engine didn't fail to stop the loop. The engine built the loop, one appended round at a time, and sat the model down inside it.

A bigger model resists that trap longer. It doesn't deserve credit for surviving bad plumbing, and we'd rather fix the plumbing than rent a stronger swimmer.

Conversation gets the same spine as everything else

The rest of the Mind had already grown up. Narratives trace to evidence. Facts carry tense. Sealed content stays sealed. Chat was the last component still living in the era before all of that, so it got the same overhaul, in four moves.

A working state instead of a transcript. The engine keeps a compact running state across a turn: the goal, the facts gathered so far, the decisions made, the next intended action. Each round the model sees the state plus the last few exchanges, never the whole raw history. Its own past deliberations never re-enter its context. You can't fall into a mirror that isn't there.

Reads become a ledger. Once a card's content is captured in the state, re-reading it returns a one-line receipt — already read — not the full text again. The forty-read spiral is now impossible, and so is the repetitive context it used to manufacture.

Guards, because any model can have a bad day. A model that starts repeating itself, or keeps planning without acting, gets one nudge: act or answer. Then the turn ends honestly, with whatever it actually did summarized for the user. If the assistant gets stuck, it says so.

Turns that outlive your tab. A conversation turn is no longer a fragile wire between your browser and our servers. Each one runs as a durable background job with its own event log. Close the laptop, lock the phone, lose the connection: the work continues, and when you come back the thread replays what happened while you were gone.

The part that surprised us

Durable conversations raised a fair question, from the founder himself: if every thread is now a permanent, resumable record, does the Mind's episodic memory — its memory of past conversations — become useless?

It's the opposite. Durable threads make episodic memory more useful, because its job was never "resume where we left off." Its job is to be the searchable memory of your conversations. The distillate that turns a hundred threads into a handful of durable facts. The place where "we decided to move the launch" gets caught the moment it's said in chat, weeks before it's written on any card. The thread is the transcript; episodic memory is what the transcript meant.

We almost deleted a feature because a better feature arrived. We were about to make the same mistake with chat, and blame the model.

One line of the founder's stuck, because it was right about more than chat: the weakest engine in a product sets the impression of the whole thing. "Stone age" wasn't harsh. It was accurate, and it only stung because everything around it had already moved on. The chat engine that embarrassed us is running the same small model it always was. We just stopped pouring its own confusion back into it.