Chapter 12 · Trust
Part 3 · Teaching it to remember13 of 35
Silent failures: the AI memory feature that never saved a row
Our assistant's remember feature failed on every call from the day it shipped. Fail-soft error handling, a busy table and mocked database tests hid it. How we found it and what changed.
The Margin2 min read
invalid UUID 'insight:hub_...' scrolled past while I was chasing a
completely different crash. Pulling the thread took ten minutes. What it meant
took longer to sink in: Margin's durable memory store had never received a
single row. Not recently. Not occasionally. Ever.
You can tell Margin Intelligence to remember things. "Remember that I prefer async standups." "Remember the wifi password is..." (well, not that one anymore, but that's a different story). There's a reflection pass that distills patterns from your conversations, and a consolidation step that writes higher-order insights on top of them. Three features, one shared destination. And the destination was empty, and had been empty the whole time.
The bug was one type wide#
The memory table's id column is a UUID. The code that wrote durable memories built friendly, human-readable ids instead, a prefix and a slug, like a filing label. The database rejected every one of them, every time, with a perfectly clear error.
So how did three features ship, run in production for weeks, and appear to work? Our own carefulness built the hiding place.
What hid it
- Best-effort by design
- a memory failure must never break someone's conversation, so the write caught its own error, logged one quiet line, and moved on. Correct behavior, perfect camouflage.
- Busy neighbors
- content indexing, the notes and tasks and boards, used real UUIDs and filled the same table just fine. Anyone glancing at it saw thousands of rows. Only the durable rows were missing, and no dashboard told a table with rows from a table with the right rows.
- Mocked tests
- we had tests for the extraction logic, the dedup logic, the recall logic, and every one of them mocked the database. The single fact that would have failed, this string is not a UUID, lived in precisely the layer everyone mocks.
The fix, and the thing under it#
The repair was almost anticlimactic. Derive each memory's id deterministically from its meaning, so the same fact always produces the same valid UUID, so re-stating something updates its row instead of duplicating it, and everything downstream (dedup, exports, graph links) keeps its contract. We proved it with the first durable memory the system ever successfully stored, watched it land, deleted the test row, and let real ones start piling up.
Graceful degradation is still the right call for a memory layer. But with nothing counting what it swallows, it turns loud, embarrassing crashes into quiet, permanent absences. An absence has no stack trace. Nobody files a bug that reads "I have a feeling nothing is being remembered."
The fix came with a policy attached. Every fail-soft path in the memory plane now counts what it drops, and a zero where there should be a number is treated as an alarm instead of a comfort. The memory write paths now have tests that run against a real Postgres and its actual type system. The mocks stayed. They just aren't the only witness anymore.
Your assistant remembers things now. We checked.