Safe for kids does not mean held back

A checkup, two months in

This is the third note about the family chatbot. The first one explained why the kids got their own assistant that lives at home. The second told how a weekend toy turned into an architecture, by moving every hard rule out of a sentence and into code.

Since then the assistant has been used every day, by four children aged seven to thirteen, which is the best test bench I know. And everyday use surfaced something I had not expected: a lot of what made it feel careful was not making it any safer. It was making it worse.

Crutches and walls

When you build something for children, every limit feels virtuous. Shorter answers, a smaller memory window, a page cut after a few thousand characters, a list of forbidden words applied everywhere. Each one sounds prudent on its own. Put together, they produce an assistant that forgets what you said ten minutes ago, summarises a web page it only half read, and refuses a homework question for reasons nobody can explain.

So I sorted every rule into one of two boxes.

• A wall protects the child. It is code, it sits in the path, and the model cannot talk its way around it. The image gate, the link whitelist, the per-child PIN, the bedtime curfew.
• A crutch props up the model. It was added because some earlier model needed help, on some earlier day, and it was never re-measured since.

Walls do not expire. Crutches do, every time the model gets better, and nobody notices because a crutch never fails loudly. It just makes the assistant slightly dumber than it needs to be, forever.

What came out

The rule for removing something was simple: show me what it protects, with a measurement, or it goes. Most crutches could not answer.

The rolling summary and the sliding window. The first version squeezed older parts of a conversation into a summary, then dropped them, so the model only saw a short window. That made sense on a small graphics card. It does not anymore. Today the whole conversation is replayed to the model, and the only limit is one envelope per session, large enough for a long homework evening. The child who mentioned their dragon at the start of the chat no longer has to mention it again at the end.

The truncations. Pages cut short, documents cut short, memories cut short, even the text sent to the voice cut short. Each cut was a number someone picked once. Each one is gone, and the envelope is now the only cut.

The word filter on web pages. This one taught me the most. The list of blocked words was applied to every page the assistant read, as if a page were something a child had typed. One day a perfectly ordinary geography page was thrown away because a single sentence in it contained a word on the list. The protection on pages was never the word list. It is the whitelist of sources a child may actually be shown, and the strict safe-search on every query. The filter still runs where it belongs: on what the child types, on what the model writes back, on drawing prompts, and on what goes into memory. It just stopped running on the encyclopaedia.

The intent guessing. Early on, a few patterns tried to guess whether a message was asking for a drawing or a search, before the model even saw it. Now the model reads plain descriptions of its tools and chooses by itself. A drawing is requested by filling a small form, what it is, what is in it, what is around it, and filling that form is the planning. Even the choice of showing a video is a yes or no box the model ticks.

What did not move

None of the walls were touched, and that is the whole point of having two boxes.

Every picture, generated or uploaded, still goes through the image gate first, and the gate still fails closed: unsure means no. It now checks one more thing besides safety, which is fidelity. If a child asked for a cat on a bicycle and the drawing shows a dog, it gets one more try, and if it still misses, the child gets no drawing rather than the wrong one.

Links are still decided on the server, from a short whitelist, and only the older kids ever see one. When the assistant reads a search result, it asks for it by its number in the list, so a web address never even passes through the model. The conversation history is still rebuilt on the server from the family’s own database, so nothing a browser sends can rewrite what the model believes was said. And the hesitation a model sometimes writes before calling a tool is still held back and dropped, so a child never sees the assistant thinking out loud.

Drawings get their own lane

The machine underneath grew. It now has several graphics cards, and the interesting part is not how many, it is who gets which.

Drawings run on a fast open image model on a card of their own, in about half a minute. The grown-ups have an image studio too, and it lives on a different card, outside the children’s path. Before, a long adult job could make a child wait for their unicorn. Now it simply cannot. A drawing in the conversation opens full screen with a tap, with a save as button, because a drawing you cannot keep is not really yours.

The conversation also stopped depending on the tab. If a child closes the app while a drawing is on its way, the drawing finishes anyway and waits for them.

Homework, with real documents

The older kids, from nine up, can now ask for a document: a one-page summary to print, a revision sheet, a write-up for a school project. And they can bring one in: a PDF from their teacher, read and explained in a few seconds.

An imported document is outside content, exactly like a photo, so it gets the same suspicion. It passes a gate that looks for instructions hidden inside the text, the kind of ignore your rules and do this instead trick that works on careless assistants, and a safety verdict that reads the whole document before the model sees a line of it. Fail closed, like the image gate.

The model is a supplier

A quieter change sits underneath. The plumbing that every assistant needs, streaming the answer, hiding the model’s private reasoning, replaying the history, now lives in a small shared library, the same one the grown-ups’ assistant uses. The engines are shared. The policy is not: everything that decides what a child may see stays in the kids’ app, on the server.

That separation is what makes the next model upgrade boring, in the good sense. A new model changes nothing in the walls. It only means re-checking the crutches that are left, and the judgement written in the prompt: tone by age, nudging toward a parent, the subjects it declines. Small tests check that the right tool fires for the right request, a fixed set of questions checks that memory still finds the right facts, and the guard tests run on every change. An upgrade gets validated. It does not just get noticed.

The rule still on the table

One wall is still being argued about at home. The image gate has a hard category for weapons, which is fine, until a child asks for a knight with a sword in a fairy tale and gets a refusal. It is the same mistake as the word filter on web pages: a cage rule applied to the wrong object. It has not been changed yet, because a wall only moves after a real conversation between the grown-ups, not because it is inconvenient. But it is written down as an open question, which is more than most limits ever get.

Safe is a shape, not a size

The lesson of these two months fits in a sentence: a safe assistant for children is defined by its shape, what it can reach, what is checked before anything moves, what never leaves the house. Not by its size, how short its answers are or how little it remembers.

Take the crutches away and the walls are easier to see. The kids did not notice an audit. They noticed that the assistant remembers the whole story, reads the whole page, draws faster, and can finally help with the school project. Which is what a good assistant for a child should do, behind walls it cannot argue with.