The app is TypeScript. The assistant is Python, in a different service, on a different host, behind an internal key.
Every time I explain that, someone asks why it is not just a route in the Next app. It would be fewer moving parts. It would be one deploy. It would be one language.
What a model call actually does to a web app
A request that renders a page finishes in tens of milliseconds. A request that asks a model to do something takes fifteen seconds if it goes well and two minutes if the task is real, because a turn is a loop rather than a single call. Read, think, call a tool, read the result, think again.
Those two things do not belong in the same process. Putting them there is possible; it is just that everything you tune for one is wrong for the other. Connection pools sized for fast requests, memory limits set for rendering, timeouts that make sense for a page load: put a two-minute agent loop through them and you do not get a slow assistant, you get an app that falls over while someone is loading their dashboard.
Separating them means the assistant can be slow, or busy, or briefly broken, and the product stays a product.
What the runtime is actually holding
The deeper reason is that the agent runtime does a genuinely different KIND of work, and mixing it in hides that.
It holds conversation state across turns. It decides which tools exist for this particular user, on this tier, in this workspace. It passes every prompt through a guard before it reaches a model. It meters what a turn cost and writes that down. It falls back to another provider when one is down.
Written as routes inside the web app, those become scattered middleware nobody can see the shape of. Written as a service, they are the service. One place you can point at and say: this is what happens when someone asks our product a question.
The bill for the boundary
It is not free, and I would rather be honest about the parts that have hurt.
Two languages means two of everything. Schemas, validation, test harnesses, CI. Change the shape of a card and you change it in TypeScript and again in Python, and there is no compiler that will tell you when you have only done one.
The wire between them is a place data goes wrong. We shipped a bug where JSON was encoded twice, once by our own call site and once by the database driver, and the data landed as a string containing a string. Everything "worked". Nothing was readable. That entire class of failure does not exist inside one process.
And the service is reachable from the internet, which is a sentence with real consequences. Two routes once shipped with no authentication on them, one of which let an outside caller hand work to our agents. They were found and closed, but the lesson stuck: a service on its own host does not inherit your app's session. Every route has to say out loud that it requires the internal key, and forgetting is silent until someone else notices.
Why it stays
Because the alternative fails in a worse way.
An assistant embedded in the web app is an assistant that can only ever be reached from the web app. Ours is reached from three places: the product, an MCP server that lets external clients drive it, and background jobs that run with nobody watching. That is only possible because the runtime was never a feature of one frontend.
And because it can be replaced. The model provider is a setting. The conversation loop is a file. A better model arrives every few months, and the change stays inside a service whose entire job is being the part of the system that changes.
The product does not have opinions about which model is good this quarter. It has an assistant, and the assistant lives somewhere else.