What was measured, and how

Summary

Two more results are pass or fail. A server killed after confirming a write restarted with that write, and after a purge the forgotten text was in no file of the data directory. Both are recorded below.

The weakest results are open-domain questions, at 0.608 recall at best and 0.375 of answers accepted in the June run, where their recall was 0.550, and fused ranking on raw turns, where the gain over embeddings alone is under 0.01 and does not hold on the held-out conversations.

The retrieval and timing runs were made on the night of 7 October 2026, between the 0.2.1 and 0.2.2 releases. The current release is 0.2.2.

Retrieval on LoCoMo

LoCoMo is a public benchmark of 10 long conversations between two people, 5,882 turns in all, each spread over 19 to 32 dated sessions. Every question names the turns that hold its answer. The dataset has 1,986 questions in five categories. The published convention leaves out the 446 adversarial ones and scores the other 1,540: single-hop (841), temporal (321), multi-hop (282) and open-domain (96).

Each conversation is stored as one customer's memory, one event per turn. Evidence recall@20 is the share of a question's evidence turns that come back when every query returns 20 events, averaged over the 1,536 questions that name evidence (four open-domain questions name none). The score is computed from turn ids, and no model judges it.

Two kinds of client

memspine does not extract facts or rewrite queries. A client either does that work or leaves it out, and the benchmark harness plays both.

with client-side work

After each session the harness asks gpt-4o-mini for dated facts and stores each as a fact event linked to its source turns. With each question it also sends up to three shorter queries written by the same model. Every query returns 20 events and the union is scored. That is 3.9 queries per question, and the union holds about 43 distinct turns under keyword-only ranking, 32 under embedding-only and 34 under fused, so the three rankings are not scored on the same number of turns. A retrieved fact counts as the turns it cites.

raw turns

Only the turns are stored, and the question is the only query. Exactly 20 turns are scored.

Each client was run three ways: keyword only (BM25 over event text, no embedding sent), embedding only (exact cosine similarity, text-embedding-3-small at 384 dimensions), and both fused by reciprocal rank.

One question, scored six ways

One line of each of the six result files. Turns scored is the number of distinct turns in the scored union. Recall is the retrieved share of the three evidence turns.

With client-side work, the fused ranking keeps the turn that keyword ranking found and the one that embedding ranking reached through a fact, and recall rises from 0.333 to 0.667. On raw turns the keyword term displaces a turn that embeddings alone had returned, and recall falls from 0.667 to 0.333. One evidence turn comes back in no run. The question was picked because it shows both movements, and the next section counts them.

All 1,536 questions

Evidence recall@20 on LoCoMo, 10 conversations. The result files were written on the night of 7 October 2026, after the 0.2.1 release and before 0.2.2. The embedding-only and fused rows with client-side work score the same on every question as runs made on the 0.1.0 engine earlier that day. Embeddings and model responses were replayed from a cache recorded in June. Marked: the largest gain from fusing, keyword-only recall on multi-hop questions from raw turns, and the loss from fusing on multi-hop for raw turns.

Open-domain questions are the weakest category in every row but one, and no cause has been established. The exception is keyword-only recall on raw turns, where multi-hop is lower (0.279 against 0.310) because multi-hop questions share too few words with their evidence. A caller with no embedding model needs the client-side work: keyword-only recall is 0.812 with it and 0.587 without.

What each part of the client's work buys

Evidence recall@20, embedding-only ranking. The two middle rows were run in June on the engine before 0.1.0, whose vector index was approximate. The first row scored the same on both engines. Under each label is the mean number of distinct turns scored per question. No keyword-only or fused run has one part of the work without the other.

Taking the facts away from the full client costs temporal questions 7.5 points and single-hop 3.7. Taking the rewriting away costs multi-hop 8.6 and open-domain 3.6.

Facts alone reach 0.840 while scoring fewer turns than the raw client, about 18 against 20: a fact takes one of the 20 places, and some facts cite a turn already counted. Their gain therefore does not come from a larger budget. Rewriting alone reaches 0.830 on about 38 turns, and no run scored raw turns at that budget.

The price is one language-model call per session at ingest and one per question at recall, plus an embedding for every fact and every extra query, all made by the client.

Fusing helps when the client does the fact and query work

The fused ranking adds a half-weight keyword term to the embedding's reciprocal rank, after dropping keyword candidates that score under half of the best keyword match. A mean can rise because a few questions moved a lot, so fused and embedding-only ranking were compared one question at a time.

Questions for which fused ranking retrieved more, or less, of the evidence than embedding-only ranking. An exact sign test on the questions that changed gives p below 0.00001 with client-side work and 0.67 on raw turns. The test treats questions as independent, which questions about one conversation are not.

With client-side work, fusing retrieved more evidence for 70 questions and less for 24. Questions with all their evidence retrieved rose from 1,238 to 1,270, and questions with none fell from 117 to 88. On the five held-out conversations the count is 35 to 14, and fused recall is higher in each of the ten.

On raw turns the mean rises from 0.787 to 0.794, the count is 71 to 65, and multi-hop falls from 0.609 to 0.590. Fused recall is higher in five conversations of ten, four of them among the five the settings were tuned on, and on the held-out half it is 0.785 against 0.790 for embeddings alone. For a client that sends the question as asked, these runs do not show that fusing helps.

How the weight and the floor were chosen

The weight of the keyword term and the floor under it were chosen on the first five conversations and checked on the other five, in runs with client-side work.

By conversation

Evidence recall@20 by conversation, under the dataset's own ids. Marked: fused below embedding only. For conv-48 the difference is in the fourth decimal place.

End-to-end answers

J scores the answer built from the retrieved events: an answer model reads them, a judge model marks the answer against the gold answer, and J is the share marked correct.

J has been scored once, on 11 June 2026, on the engine before 0.1.0, with embedding-only retrieval and client-side facts and rewriting. The answer model and the judge were both gpt-4o-mini, at about 1,150 tokens per question on the answer path. The judge prompt was a published rubric, used verbatim.

The June run. Recall is averaged over the 1,536 questions that name evidence, and F1 is word overlap between the answer and the gold answer.

The 0.786 is one pass with no error bar. It depends on client code that is not part of memspine: the harness's answer prompt, and its layout of the retrieved events, facts first and then turns in date order under session headers. It also predates the current engine, which replaced an approximate vector index with exact search.

In embedding-only mode the engine since 0.1.0 reproduces June's evidence recall on every question and returns a byte-identical list for 1,416 of the 1,529 distinct questions (11 are asked twice). Of the other 113, 34 differ only in order and 79 in turns that are not evidence, so J should carry over. It was not re-run, and fused retrieval has no J. A full pass costs about 1.8 million tokens on the answer path, with the judge's calls on top.

Where the rejected answers came from

The answers the judge rejected in the June run. All, part and none say how much of the question's evidence had been retrieved. A refusal is the answer "No information available". Two rejected open-domain answers belong to questions that name no evidence.

Of the 329 rejected answers, 195 scored full evidence recall: every evidence turn was in the context the answer model read, as the turn itself or as a fact derived from it. Raising evidence recall would not change them. Refusals account for 172, and 102 of those had full evidence recall.

In two categories most rejected answers lacked some of their evidence: multi-hop, 46 of 67 (42 had part, 4 none), and open-domain, 31 of the 58 that name evidence (10 part, 21 none).

Timings at 10,000 and 100,000 events

The scale run fills one customer's memory with synthetic events: a random 384-dimension unit vector, eight words of text and two short properties each. It then times 50 single appends, 100 searches, 100 fused recalls, three forgets with a purge after each, and a cold open. The engine is called in process, so no figure includes HTTP or JSON.

memspine-bench --mode scale on the 0.2.1 engine, 7 October 2026, on an Apple M-series laptop that was busy with other work. Sizes between the two were not run. Peak memory covers the whole run, purges included. The 100,000-event run on 0.1.0, before forget and purge were part of it, peaked at 1.2 GiB.

• Search compares the query with every embedding the customer has stored, so its time grows in proportion to that customer's event count.
• A single append costs one fsync and a forget two, so unbatched writes are bounded by the disk's fsync rate. The batch route stores up to 1,000 events per fsync.
• A purge rewrites the customer's whole log and blocks that customer while it runs: a median of 25 s and at most 39 s at 100,000 events in this run.
• An open customer holds its embeddings and indexes in memory. After an hour idle, by default, it is closed, and its next request pays the cold open.

Crashes and deletion

An append replies only after its batch is in the customer's log and the log has been fsynced. The direct check is to confirm a write, kill the server without warning, restart it and read the event back.

$ curl -s localhost:7777/v1/memory/event -H "$K" -H "$J" -d @last.json; echo
{"event_id":"1eb38657-2649-47e6-b082-bf90e6da5c72"}

$ kill -9 $(lsof -tiTCP:7777 -sTCP:LISTEN); sleep 0.3; curl -s -m 2 localhost:7777/health; echo "curl exit $?"
curl exit 7

$ MEMSPINE_BIND=127.0.0.1:7777 MEMSPINE_EMBEDDING_DIM=4 MEMSPINE_DATA_DIR=./data \
    memspine-server > memspine.log 2>&1 &

$ curl -s localhost:7777/v1/memory/event/$E11 -H "$K"; echo
{"event_id":"1eb38657-2649-47e6-b082-bf90e6da5c72","kind":"turn","ts":1791366150000000,"text":"Ana: thanks, that is all for today","properties":{}}

Recorded against the released 0.2.2 binaries on macOS arm64. $K and $J hold the authorization and content-type headers, and $E11 the id the first command returned. The whole session is in session.txt. This is the only check that kills the process, and it reads one event back.

Forgetting has two steps. DELETE makes the event unreadable and removes its links in one batch, with two fsyncs: first a marker file that records the purge owed, then the log. A purge then rewrites the log so the bytes leave the disk. A purge pass starts within 60 seconds by default and takes the customers that owe one in turn. With ?purge=true the purge runs before the reply, and the reply says whether it succeeded ("purged": true).

$ grep -rl "912 555 019" data
data/00000000-0000-0000-0000-000000000042/events.ffs.qlog

$ curl -s -X DELETE "localhost:7777/v1/memory/event/$E9?purge=true" -H "$K"; echo
{"deleted":true,"purged":true}

$ grep -rl "912 555 019" data; echo "grep exit $?"
grep exit 1

From the same session. A phone number is found in the customer's log file, its event is deleted with a purge, and the same search of the data directory finds nothing.

What each claim rests on

What has broken

All four tagged releases came out on two consecutive days. After 0.2.0, two independent reviews went looking for defects, and a finding counted once a second reviewer had reproduced it. 0.2.1 and 0.2.2 are the fixes. The last part of the review finished after 0.2.1 had gone out, and it found a defect that 0.2.1 itself had introduced.

1. 0.1.07 October 2026

   The engine is rebuilt on FFS. Keyword and fused recall, batch append, reading an event back.

2. 0.2.07 October 2026

   Forget and purge, listing, and closing customers that sit idle.

3. 0.2.17 October 2026

   26 fixes for what the review found in 0.1.0 and 0.2.0.

4. 0.2.28 October 2026

   9 fixes from the part of the review that finished late, one of them for a defect of 0.2.1.

The defects a caller could have met, newest fix first:

Every entry is in changelog.txt. One gap remains from this list: events forgotten under 0.2.0 left no record of their id, so a retry of one of their keys after upgrading still stores the event.

The code is covered by 162 Rust tests, 52 of which drive the HTTP server end to end, plus 29 tests in the Python client's suite and 21 in the TypeScript one. Every change runs formatting, lints with warnings denied, the tests, and a container image that is started and written through.

How the runs were made

The harness is memspine-bench, a crate in the source repository. That repository is private and the installer ships only the server and the admin tool, so these commands show how the runs are invoked. Running them takes the repository; talk to ERP.AI.

cargo run -p memspine-bench --release -- \
  --mode recall --retrieval hybrid

cargo run -p memspine-bench --release -- \
  --mode recall --retrieval hybrid \
  --no-facts --no-expand

cargo run -p memspine-bench --release -- \
  --mode scale --events 100000

The recorded runs also passed --workdir bench-runs/v0.2.0, and the scale runs were wrapped in /usr/bin/time -l, which is where the peak-memory row comes from.

--retrieval

vector (the default), hybrid or text: the embedding-only, fused and keyword-only rows. --text-weight and --text-floor set the fusion numbers.

--no-facts --no-expand

Together, the raw-turn rows. Either flag alone, with --retrieval vector, gives a middle row of the client-work table.

the dataset

locomo10.json from the benchmark's public repository, saved under bench-data/.

the model cache

Embeddings and model responses are cached in llm-cache.jsonl in the working directory. The retrieval runs replayed the cache recorded in June with the model API key set to a dummy value, so a missing entry would have failed the run instead of calling a model.

--mode full

Adds the answer model and the judge, and needs a real key.

--mode scale

Needs neither the dataset nor a key. --events sets the size.

Every run writes one line per question: the queries sent, the turn ids retrieved, the recall, and in a full run the answer and the verdict. The worked question, the counts of questions that changed, the turns scored, the per-conversation figures and the breakdown of rejected answers on this page were computed from those files. They sit in the private repository with the harness and are not published here.

Not measured, or known to be weak