Building a Personal Assistant: What I Got Wrong
A couple of weeks ago I wrote that I am the only user of my personal assistant app, so every issue in its tracker is one I filed after watching something fail on my own phone. That sentence is no longer true. My daughter started using it, and she found a limitation I had built in so deeply that I could not see it.
Before that, though, some numbers. The app is at version 6.5.0, build 191, with 115 release notes behind it. It is 245 Dart files in lib and 234 test files. The lib/services directory alone holds 124 services. I bring this up because people who ask me about building their own assistant usually think of it as a weekend project - a chat box with an API key behind it. The chat box was the weekend. Everything after that is the reason the thing is actually useful.
Three things moved it from a toy to something I open every day: it learns from its own mistakes, it accepts the way people actually talk, and it can look at a photo. Everything else is plumbing, and I will be honest about the plumbing at the end.
Why Not Just Use One of the Big Ones
A fair question with several answers. Here is one concrete one.
I needed AI access my daughter could actually use. She is a preteen, so most of what an adult would reach for is closed to her. ChatGPT is age restricted. I could not get Gemini working on her phone at all, and still do not know why. We tried the assistant that comes with WhatsApp, which means Meta, and it was not what she needed. Every path led to a policy wall or a product built for someone else.
So I built the thing instead, against an API endpoint. Today the app runs on Gemini, with entry points for Anthropic and OpenAI, and roles pinned to a local gateway when the data should not leave the device at all. Same models everyone else is using. Different terms of access, because the terms are mine.
The part I did not anticipate was transparency. The app has a screen that logs the prompts it sends, verbatim, so you can see exactly what left the device. I built it for myself, to debug what the app was actually sending. It turns out to be the thing that makes a kid using AI reasonable rather than nerve-racking. I can see the request and the response.
I want to be careful about how I say this, because I am not reading her conversations under a microscope, and a parent who does that will find their kid stops using the tool honestly. But the alternative on offer is a chat box run by a company, aimed at a preteen, where I get to see nothing at all. Being able to look is different from looking. I would rather have the option and rarely use it.
The Loop That Closes Itself
When the assistant gets something wrong - a bad answer, a card that renders badly, a log that lands in the wrong place - I paste that straight back into the chat and it files a GitHub issue with the context attached.
That sounds like a small convenience. It is the most important thing in the app. Small projects do not die because the bugs are hard. They die because noticing a bug and writing a good report about it are two different jobs, and the second one lands at the exact moment you are least willing to do it: standing in a kitchen, one hand busy, annoyed. So you tell yourself you will remember. You do not. The bug survives another month.
The app also watches for its own mistakes - you restating the same request, you undoing what it just wrote, a question that almost matched something specific and got a vague answer instead. That last one matters because you never see it as an error.
What I got wrong at first was asking on a schedule. When it prompted "did I get that right?" at a fixed cadence, I said yes to make the prompt go away. The friction stayed and the signal disappeared. Now it asks only when something actually looks wrong, and it works that out on the phone without sending anything anywhere.
Messy Input Is the Real Feature
Nobody logs data in a form. People say "had a bowl of soup and a coffee around two." So the parser does the work, and most of what makes a parser good is knowing when not to act.
Two bugs taught me that, and they failed in opposite directions. "remove a cup of water" logged a cup of water, because the parser saw a quantity and never looked at the verb in front of it. Going the other way, everything hydration-related matched the literal word water, so seltzer and club soda reached no logging path at all. Nothing failed. Nothing was logged either.
One is the app acting when it should not have. The other is the app not acting when it should have. Both come from reading the surface of a sentence instead of what the sentence is doing.
The third rule is the one I would keep if I threw out the rest: when you state your macros, they are stored exactly as you said them. The model does not get to re-estimate a number you already gave it. If you tell the assistant something factual, re-deriving it with a model is not helpfulness. It is the app deciding it knows better than the person holding the phone.
Logging a Meal From a Photo
On August 17 I photographed a bowl of cold beetroot soup - svekolnik - and asked the app to log it. It came back with 110 calories, 3g protein, 22g carbs, 1g fat, and I confirmed the card.
The subtle part is intent. A camera button used to be the intent; with one attachment control, the words have to carry it. An attached picture next to an ordinary question is a question, and misreading it as a log buries the answer you were waiting for under a food card. The opposite mistake costs one retyped sentence, so the matcher under-claims. The model will happily guess every time if you let it.
The failure modes are honest ones. A plated meal goes well: chicken, rice, something green, three things to name and add up. A pot on the stove goes badly, because it is one volume with everything dissolved into it. Portion size is the worst of it, since the model is judging volume from a photo with no reference object and no depth. It cannot tell a teacup from a mixing bowl. Neither could I, from the same photo.
So nothing is written until I confirm the card, and the card takes a correction in plain words: "about two cups," or "half of what is in the picture." The model is good at naming what it sees and bad at knowing how much of it there is, and the app is built around that split instead of pretending otherwise.
One reason I keep working on this part: I have been using it to track calories and fasting, and I am down more than ten pounds. It has taken over from MyFitnessPal for me. Not because the estimate is more precise - it is not. Because logging a meal became cheap enough that I actually do it.
Then My Daughter Started Using It
She asked for one thing I had not thought of: let it take files and images for analysis, not just food. Her first test was photographing a math equation from her homework and asking the assistant to check her work. A good request, and it earned its keep on something that was neither homework nor food. She has track and field events this season, and the schedule reached me the way schedules actually reach people: as a picture of a calendar, sent from her phone to mine. I uploaded it and the assistant read the dates off the image correctly and put them on my schedule, since I am the one doing drop off and pick up. Nobody typed anything.
That one has quietly changed how I run the week. School just started, and the schedules arrive in every shape a school can produce: a PDF from one program, a month grid from another, plain text in an email, a photo of a paper handout that came home in a backpack. Different activity, different format, same deadline. The old version of this was me squinting at a screen and retyping dates into a calendar, which really meant half of them never got entered at all. Now I point the app at whatever is in front of me and the dates come back filled in, with the names and the times attached. It saves me a real amount of time each week, and it has nothing to do with the chat box.
It also shows where the app still falls short. Those events landed on my schedule and stopped there. Two-way Google Calendar sync is not working yet, so nothing flows back to hers, and we are still coordinating around one device instead of one calendar. An open issue, not a solved problem, and the same shape as everything else here: the model part was the easy half.
The problem she exposed is bigger than any of it.
The app has profiles - personal, work, health, finance, travel - and each one carries its own memory categories, its own enabled tools, and its own routing preference for sensitive data. Health and finance are local-only. That design has held up well, and I was quietly proud of it.
Then someone else picked up the phone, and I saw what those profiles actually are. They switch context. They do not switch person. Every default under all five of them is mine: hydration nudges, fasting timers, focus blocks, the particular set of habits a software person decides to fix about himself. None of that is what a preteen wants from an assistant, and a fasting timer offered to a kid is not a neutral default. It is wrong.
So the work in front of me is a harness around a profile that carries whose assistant it is: which suggestions exist at all, which memory categories are collected, which tools are reachable. The rules a preteen profile needs are mostly subtractive - things the model does not get to raise on its own, no matter what it infers from the data in front of it. Health is the obvious case. I would rather the assistant be boring than have it decide, on its own, to have an opinion about a kid's eating.
I want to be clear that this part is not built yet. It is what I am building next, and I am writing it down here because the lesson is the transferable part: I designed for one person without ever deciding to, because the one person was me and the app agreed with me about everything.
What I Would Tell Someone Starting
First, the promise I made at the top: almost none of those 124 services are AI. The parts that took the most time were calendar and contact intake, notification scheduling, encrypted local storage, backup and export, and being able to change model providers without rewriting the app around whichever one is currently cheapest. That last one matters more than it sounds, for reasons I wrote about in the July cost post. Build around one vendor's client and a price change is an outage. Keep model roles in a registry and it is a settings change.
The rest is short.
Build the correction loop before the features. The app that files its own bug reports gets fixed; the one that requires you to open a laptop does not.
Keep the data local and own the schema. Every provider swap I have done was painless because the data never lived at the provider.
And decide early whether the assistant serves one person or a persona. I did not decide. I defaulted to me. Retrofitting that is the expensive kind of work, and it took someone else about a week of using the app to expose it.
Comments