Architecture deep dive
How memory works.
A loop with three stages — signals, memory, reflection — running on your device. Every fact has provenance, confidence, and a decay curve. Nothing about you exists in a place you can't inspect or delete.
The loop in one diagram
Three stages, three transitions. Signals arrive constantly. Most disappear. The salient ones promote to memory. Memory gets revisited during reflection and either strengthens, decays, or merges with a related fact.
A signal becomes a candidate
A signal is anything bimi observes — a calendar event firing, a sentence you spoke, a location change, a track that just started playing. Most signals die in milliseconds: they're not interesting enough to remember. The promotion gate scores each one and only the salient ones move on.
The scoring is local. The gate considers:
- Recency. A signal earlier than the last reflection cycle is weaker.
- Salience. Did you flag it (Save this), or is it semantically novel against existing memory?
- Identity overlap. Does the signal mention a person, place, or topic bimi already tracks?
- Source confidence. Calendar API ≫ push-to-talk transcript ≫ passing mention.
Why a gate, not a sink
Without the gate, every passing sentence becomes a permanent memory. Three days in, the constellation is unusable — buried in noise. The gate is what makes bimi's memory feel like memory instead of a log.
What a memory record looks like
Every promoted signal becomes a typed record. The shape is the same regardless of source — what changes is the category. Here's an identity-category fact, abbreviated:
The five fields you can always count on:
- statement — the human-readable fact. This is what shows up in the constellation.
- provenance — where did this come from? voice, calendar, manual edit, reflection.
- confidence — how sure are we, on a 0–1 scale. Edits set this to 1.0.
- sensitivity — public / personal / sensitive (with finer-grained categories for PHI, credentials, and financial data). Affects retrieval policy and what gets shipped to cloud providers.
- links — graph edges to related memories. Powers retrieval and the constellation lines.
Multi-vector retrieval
When you ask bimi a question that needs memory, it doesn't do a single vector search. It does four scoring passes in parallel and merges them.
| Pass | What it scores | Why |
|---|---|---|
| Semantic | Embedding cosine against the question | The classic "what's this about" match. |
| Temporal | Recency-weighted decay (newer memories rank higher unless the question says otherwise) | "Last semester" and "this week" actually behave like time anchors. |
| Category boost | If the question is about a person, identity + episode memories rank above preferences | "Tell me about Maya" shouldn't return a song preference. |
| Entity linkage | Graph hops along the links field |
Asking about TBC Corp also surfaces colleagues, projects, recent meetings. |
The four scores are combined into a single rank, with the weights tunable per query type. The top N enter the prompt; everything else stays in the constellation, untouched.
Where the model lives
The model that decides "is this worth remembering?" runs on your device.
Not as a promise — as a type. The memory subsystem can only call into
models that conform to a Swift protocol called
LocalOnlyLanguageModel. Apple's on-device Foundation Models
conform. The optional On-Device Brain lane (a larger MLX-Swift model you
download yourself) conforms. Cloud providers don't. A future config bug
or accidental refactor can't accidentally route a memory verdict
through a hosted model — the compiler refuses to build it.
| Job | Default model | Optional upgrade |
|---|---|---|
| Decide if a candidate is durable | Apple Foundation Models (on-device) | On-Device Brain (MLX-Swift, your choice of open model) |
| Split a multi-fact dump into atomic rows | Apple Foundation Models (on-device) | On-Device Brain (MLX-Swift) |
| Background curator (Phase 1) | Apple Foundation Models (on-device) | On-Device Brain (MLX-Swift) |
| Embedding for retrieval | System NaturalLanguage framework (on-device) | — |
On-Device Brain is opt-in and visible: Settings → Memory → On-Device Brain lists the models, what they need (disk, RAM), and whether they're installed. Picking one is a single click; switching back is too. The bimi sidebar always shows which model handled the last memory verdict.
Why a separate lane for memory work
Your chat with bimi might bounce off a cloud model if you've configured one — that's a choice you can name and see. The memory subsystem doesn't get that choice. Anything that becomes a permanent record about you stays on the device. The split lane is what makes the privacy promise checkable instead of trusted.
Confidence, decay, half-life
Human memory decays. Useful artificial memory should too — otherwise stale facts pile up and start corrupting answers. Every record has a half-life, chosen by category:
- Identity — long half-life. "Steven's email" doesn't decay quickly.
- Preference — medium. Tastes shift; restate one to refresh.
- Commitment — sharp. "Email Maya by Tuesday" should decay the moment Tuesday passes — and graduate into an episode or a follow-up.
- Episode — long but indexed by time. The 2024 trip to Tokyo doesn't decay; it just settles into "old."
Restating a fact resets its confidence to 1.0 and pushes the half-life clock back. Contradicting one (you, in conversation, or an edit in the Inspector) drops the old fact into review.
"In review" — the target behaviour
When two memories disagree, bimi shouldn't pick a winner silently — it should mark both for review and surface the disagreement next time it's natural. The schema supports it; the surfacing UX is on the roadmap. That's the restraint pillar in code: the agent doesn't decide things about you on your behalf.
Reflection between turns
Reflection is what runs while you're not asking. It's not magic — it's a scheduled job that walks recently-touched memories, looks for patterns, and either:
- Distills a new rule ("Steven uses 'TBC-' tickets to mean infrastructure work at TBC Corp")
- Merges duplicates ("M-series" and "M1/M2/M3" appearing in three different memories about the same conversation)
- Decays a stale episode
- Surfaces a proactive opportunity (only if the surface policy permits)
Reflection is the quiet half of the agent. Without it, bimi is just a chat UI over a memory store. With it, bimi gets to know you between turns — which is what a memory you can trust really means.
The Memory Inspector
The constellation is the cinematic view. The Inspector is the operational one: a sortable, searchable list of every memory bimi holds about you, with edit, forget, and audit affordances.
- Filter by category, sensitivity, source, confidence.
- See the full provenance chain for any fact.
- Edit the statement — confidence resets to 1.0, source becomes "manual."
- Forget — the record is removed, and a tombstone is kept so future signals don't silently re-create it.
- Hold — keep a memory in review while you decide.
The audit promise
You should always be able to answer the question "Why does bimi think that about me?" — with a single tap. The Inspector exists so the answer is never "I don't know."
The same loop, applied to your machines
Everything above is memory about you. The same machinery works on your infrastructure. When bimi does something on your behalf — connects to a server over SSH — the result is just another signal, and it promotes into durable memory: a machine it now knows, with the login that worked and a credential it can reuse. First contact needs one hint. After that, bimi resolves the host, the user, and the key on its own.
The machine shows up in your Memory Constellation like any other entity, and
in Settings as you@host with a credential badge. Ask “what's
on that server?” weeks later and bimi answers from what it recorded.
It's the audit promise, extended to your infrastructure: you can always see
which machines bimi knows, which login it uses, and revoke either.
The key stays in the Keychain
When bimi imports a key it learned, the bytes go into the encrypted, per-user Vault (Keychain-backed). Everything else — the machine record, the memory, the entity graph — carries only a handle. The secret is resolved at the moment of connection and never written to a prompt, a log, or an audit record.
What bimi won't do
Architectural negative space matters as much as the positive shape. A short list:
- No silent cloud upload. Memories marked
sensitive(or the finer-grained PHI / credential / financial categories) are restricted from hosted-provider context windows by policy. The conversational-permissions surface that asks you in-the-moment is on the roadmap (see trust, in public). - No off-device audio. Voice transitions to text + signal locally. Push-to-talk audio is RAM-only. Meeting Mode audio is cached locally with your-set retention. No audio is ever transmitted off-device.
- No third-party telemetry SDK. Crash diagnostics use MetricKit, attached to your bug report only when you submit one.
- No personality dial. The voice character is consistent. There's no "make bimi flirty" slider; that's a different product.