# Changelog ## 0.2.2 (2026-10-08) Fixes from the last part of the review behind 0.2.1, which finished after that release went out. The first item is a defect 0.2.1 introduced. ### Fixed - Starting a second server on a data directory that another server was using damaged tenants, even if the second server never received a request. 0.2.1's purge pass opens every tenant that owes a purge and compacts its log; the first server still had that log open, so its later appends went to a file that had been replaced. A server now takes an exclusive lock on `memspine.lock` in the data directory and refuses to start while another holds it, and every open tenant holds a lock on `/lock`. Run one server per data directory. - Several requests arriving together for a tenant that was not open each closed a different tenant to make room, and at a small cap could be refused with a 503 while other tenants sat idle. One request now makes room and the others wait for the tenant it opens. - A server restarted while its control-plane database was missing came up accepting tenant UUIDs as keys; 0.2.1 only covered the database disappearing while the server ran. The data directory now records that it has had a control plane (`control-plane.seen`), and a server that finds the record without the database answers 503. To run without keys on purpose again, delete that file. - An event stored without an embedding by 0.1.0 or 0.2.0 still could not be returned by fused recall after an upgrade, because 0.2.1 marked only the events it rewrote. For events with no mark the vector index is now asked, once per open tenant. - Shutdown waited for a purge pass to work through every tenant it had left, which could outlast a container's stop timeout. The pass stops after the tenant it is on; the rest stay marked on disk and run at the next start. - With `RUST_LOG=warn`, the line that reports the cause of a 500 had no trace id, because the request's span was created at INFO level. - The limiter's idle sweep could drop a tenant's bound on calls in flight while those calls were still running, after which the tenant got a full new set. - A purge interval longer than the idle check was rounded up to the idle check's tick. Each job now sleeps until its own next due time. - `GET /v1/admin/keys`, `GET /v1/admin/audit` and `DELETE /v1/admin/keys/` answered bad input with a plain-text body, before checking the admin key. They check the key first and answer with the JSON error body. ### Known limits - When this version first opens a tenant written by 0.1.0 or 0.2.0, each event it rewrites costs time in proportion to the size of the tenant, because of how FFS journals a property rewrite. Only events holding a string over 4,096 bytes are rewritten, so a tenant with many of those opens slowly once. - The limits listed under 0.2.1 still hold. ## 0.2.1 (2026-10-07) Fixes for defects that an independent review found in 0.1.0 and 0.2.0. Upgrade from either. The one that loses access to data: a tenant holding any string longer than about 8,100 bytes could not be opened after its next purge. FFS's property index rejects a key that long and reports the failure only on stderr, so the index file kept its earlier state while the purge compacted the log underneath it, and the next open failed with `property-index image load: NotFound`. Event `text`, every string property and the JSON of nested properties were each stored as one string, so one long message was enough. The same failure left a forgotten event's text in `events.ffs.propidx` after a purge that reported success. 0.2.1 repairs a tenant that is already in that state when it first opens it: it deletes the tenant's checkpoint files and lets FFS rebuild from the log, which holds every event. The server logs `rebuilding the tenant from its log` when it does. This was checked against a data directory written and broken by the released 0.2.0 binary. To do the same by hand, stop the server and delete every `events.ffs.*` file in the tenant's directory except `events.ffs.qlog`. ### Fixed - Long values are stored where FFS does not index them. `text` over 4,096 bytes and the JSON of nested properties are kept in full and read back intact, and are no longer index keys. `meta.json` records storage format 2. When 0.2.1 first opens a tenant written by 0.1.0 or 0.2.0 it rewrites the events that hold a string over 4,096 bytes and leaves the rest as they are. - FFS keeps an in-memory index entry for every pair of indexed properties on a node, so memory grew with the square of how many properties an event had. At most 16 of an event's properties are now indexed, taken in key order, and only booleans, numbers and strings up to 256 bytes; the others are stored with the nested properties. All of them read back as sent. Only the indexed ones can be filtered on in Cypher, and `n.text` is null there for text over 4,096 bytes; return the node to get it. - A tenant whose log cannot be opened for writing no longer opens. FFS would run such a database in memory, so appends were acknowledged and then lost on restart. Running out of file descriptors was enough to cause it. Opening a tenant now writes a record and checks that the log grew. When that fails the request gets a 500, the log says why, and the next request tries the open again. - The server caps how many tenants are open at once (`MEMSPINE_MAX_OPEN_TENANTS`, by default derived from the file-descriptor limit, which the server raises to the hard limit at startup) and closes the least recently used tenant that has no request in flight. - A Cypher query could abort the server process. FFS's parser, evaluator and syntax-tree destructor recurse without a limit, so about a thousand nested parentheses overflowed the stack, and so did a few thousand links of `1+1+1+...`, `n.a.a.a...` or `x[0][0][0]...`, which nest nothing. A query is now cut into tokens by FFS's own lexer before it is parsed and refused if it is over 16 KB or 512 tokens, nests brackets or `CASE` more than 64 deep, or chains more than 32 prefix operators. What passes runs on a thread with a 16 MB stack; the deepest query the limits allow was measured at 0.2 MB in a release build and 2.5 MB in a debug build. - `UNWIND`, a pattern that shares no variable with the ones before it, and a variable-length path longer than 8 hops are refused, and a result is cut off at 10,000 rows with an error. Each of the refused shapes let a 1 KB query grow the process by gigabytes. - One tenant could fill the blocking pool that every tenant's engine calls share. Each tenant now gets `MEMSPINE_TENANT_MAX_IN_FLIGHT` engine calls running or queued (default 32); the next one gets a 429. - Replaying an append under an `Idempotency-Key` whose event had been forgotten stored the event again, including after a purge. The id of a forgotten event that came from a key is now kept, and the replay returns that id without storing anything. Events forgotten under 0.2.0 left no such record, so a replay of one of their keys after the upgrade still stores the event. - Replaying a batch under the same `Idempotency-Key` with more events than the first time returned 200 and stored the extra ones. It now returns 400. A replay with the same number of events or fewer still returns the first batch's ids without comparing bodies. - Fused recall could not return a keyword match that had no embedding once enough embedded events existed, because the event was scored as if its embedding had ranked last. Such an event now scores what an embedding-only match at the same rank does, so the two kinds interleave. - Keyword recall returned a different set of events on each call when more events tied on score than `k` allowed. Ties now go to the event stored first; the fix is in FFS as of the pinned commit. - Search and recall could return fewer than `k` events after a forget, until the tenant was reopened, because the forgotten event stayed in FFS's vector index. FFS removes it on delete as of the pinned commit, and the workaround that overwrote the embedding with zeros is gone. That workaround also made a tenant with no embeddings refuse to open after a change of `MEMSPINE_EMBEDDING_DIM`. - A purge owed by a tenant that was not open never ran, so after a crash the forgotten bytes stayed on disk until that tenant's next request. The server now scans the data directory for `purge.pending` at startup and on every purge pass. A tenant whose purge fails while it is being closed stays open and is retried. - After a write to FFS fails, or a call into it panics, the tenant's engine is dropped without a checkpoint and the tenant is opened again from its log. FFS does not undo everything an aborted batch did in memory, and a panic can leave one of its locks poisoned, so the engine used to keep serving from state that the log did not hold. Requests get a 503 for the moment it takes in-flight calls on the old engine to finish. - `recall` with `hops` did work proportional to `k` times the fan-out of a shared source when a time filter rejected what the walk reached. The walk now leaves every hit at once, one hop at a time, examines an event once, and stops after `8 * k` events. Additions are ordered by distance from the nearest hit. - `DELETE ...?purge=true` answered 500 when the delete succeeded and the purge after it failed. It now answers 200 with `"purged": false`; the purge stays owed and is retried. - A tenant that owes a purge and cannot be opened (a changed `MEMSPINE_EMBEDDING_DIM`, a newer storage format) was replayed in full on every purge pass. It is now retried every 15 minutes, and the checks that can refuse a tenant run before anything is written to its log. - An embedding whose length overflows or underflows `f32` was accepted and produced `NaN` scores. It is now rejected with a 400. - A 500 told the caller to look up `x-memspine-trace-id` in the server log, and no log line carried it. The trace id is now on a span that covers the request, including the engine call on the blocking pool. - If the control-plane database existed at startup and was missing at request time, the server fell back to treating a tenant UUID as a key. It now returns 503 until the file is back. - `MEMSPINE_PURGE_INTERVAL_SECS` and the idle-tenant check ran on one shared tick, so a purge interval longer than a quarter of the idle TTL was not honoured. They have separate schedules. - 429 responses carry `Retry-After` in seconds, which is what the SDK reads; `retry-after-ms` is kept. - Admin endpoints return the documented JSON error body. Their 500s no longer include the underlying error text, which could name the control database's path. - Rust SDK: `forget_event` is retried only after a 429. It used to be retried after a timeout, a dropped connection or a 502 to 504 as well, and a retry that followed a delete which had gone through reported the event as never having existed. `append_events` splits a batch at 8 MiB as well as at 1,000 events, so large embeddings no longer hit the server's body limit. - Docker: the image's non-root user owns `/var/lib/memspine`. The documented `docker run` with a named volume could not write. - Documentation: Rust 1.85 builds `memspine-server` and `memspine-admin` with the tracked lockfile; the SDK, the benchmark and the test suites need 1.86, and the TypeScript binding 1.88. `GET /v1/whoami` accepts any valid key. The server has no CORS layer, and the README and site no longer say it does. The TypeScript and Python READMEs describe how errors are reported. The Caddyfile example in `docs/DEPLOY.md` no longer blocks the scraper it allows. The site's walkthrough starts one server and can be pasted as shown. ### Changed - FFS is vendored under `vendor/ffs` at a pinned commit (`scripts/vendor-ffs.sh` updates it), so a clone builds without access to the FFS repository and CI needs no token. CI is green again. Its docker job starts the image, waits for the health check, writes an event through it and checks that it stops on SIGTERM. - `installer.yml` installs from memspine.com on Ubuntu, an older-glibc image (AlmaLinux 8, glibc 2.28) and macOS after each deploy and runs the binaries. - `memspine-bench` reads `OPENAI_MODEL_PREFIX`, for scoring through a gateway that namespaces model ids, and takes `--run-tag` for independent scoring passes. ### Known limits - Cypher still has no row or time budget inside FFS's executor, and the screen only refuses the shapes found to be explosive. Give Cypher access only to callers you would let run a full scan. - A forgotten event that was appended under an `Idempotency-Key` leaves a `Forgotten` node holding its id and nothing else. Patterns without a label, such as `MATCH (n)`, see and count those nodes; match `(n:Event)`. - Keyword recall indexes ASCII letters and digits only. Text in other scripts is stored and returned but matches no keyword query; use embeddings for it. - FFS reports some persistence failures on stderr instead of returning them. The check at open and keeping long values out of the index cover the two cases found; watch stderr for lines from FFS. - Purge still rewrites the tenant's whole log and blocks that tenant while it runs. - Vector search is still exact and linear in tenant size. ## 0.2.0 (2026-10-07) Closes three of the limits 0.1.0 shipped with: events could not be deleted, could only be listed through Cypher, and a tenant's indexes stayed in memory until the process exited. ### Added - `DELETE /v1/memory/event/` forgets an event. When it responds, get, list, search, recall and Cypher no longer return the event, its lineage edges are gone, and that is durable. The bytes of its text and embedding leave the tenant's log at the next purge. - Purge. The server compacts every tenant that has a forgotten event each `MEMSPINE_PURGE_INTERVAL_SECS` (default 60), when it closes an idle tenant, and on shutdown; `DELETE ...?purge=true` does it before responding. A `purge.pending` file in the tenant's directory, written before the delete, carries the obligation across a crash. Tests grep the data directory to confirm the text and the embedding bytes are gone afterwards. - `GET /v1/memory/events` lists a tenant's events newest first, with `kind`, `since`, `until`, `limit` and an opaque `cursor` for the next page. - Idle tenants are closed. A tenant with no request for `MEMSPINE_TENANT_IDLE_TTL_SECS` (default 3600, `0` to disable) is checkpointed and its database closed, which releases its vector, text and property indexes. A tenant with a request in flight is never closed. Its next request reopens it. - Rust SDK: `list_events`, `forget_event`. Python: `list_events`, `forget_event`. TypeScript: `listEvents`, `forgetEvent`. - `memspine-bench --retrieval text` scores keyword-only recall, and `--mode scale` times forget and purge. - `deploy.sh` publishes the site and the installer's tarballs together. ### Changed - Events appended in one batch get consecutive timestamps, one microsecond apart, when they carry no `ts` of their own. They used to share one timestamp, which left their order undefined when listed. - Graceful shutdown purges tenants with a forget pending before it checkpoints. - The installer sends the tarball's checksum as a query string, so a CDN cannot serve a tarball cached from an earlier release against a new checksum. ### Fixed - The Python and TypeScript binding suites had not been run against 0.1.0, and two tests in each would have failed: they sent an all-zero embedding, which 0.1.0 rejects. The test data is fixed and both suites were run for this release (29 and 21 tests). - memspine.com: the one-line installer returned 404 because an earlier site deploy had removed the tarball directory, and the hero image and social card still named the pre-0.1.0 files and HNSW. ### Known limits - A purge rewrites the tenant's whole log and blocks that tenant while it runs: 0.6 to 2.5 s at 10,000 events and 4 to 12 s at 100,000 across runs on an M-series laptop that was not otherwise idle. The delete itself is one fsync. - FFS still keeps a deleted node's entry in its vector index until the log is compacted. memspine overwrites the entry with zeros before deleting and filters it from results, so nothing is exposed, but a tenant with many forgets and no purge does extra work per search. - Forgetting an event does not rewrite other events that name it in `source_event_ids`; they keep the id, not the content. - Backups taken before a purge still contain the forgotten bytes. - Vector search is still exact and linear in tenant size. - CI is still failing at checkout on the expired `FFS_READ_TOKEN`; this release was checked locally (fmt, clippy `-D warnings`, 126 Rust tests, both binding suites). ## 0.1.0 (2026-10-07) First tagged release. The storage engine was rebuilt on FFS's `Database` API, and the HTTP API grew the read side it was missing. Before this release the engine drove FFS's pager, WAL and HNSW types directly. It wrote each event's properties into a WAL record that no code path read, so search hits came back with `properties: null` and Cypher returned internal node ids. It also kept its own snapshot of the vector index and edge table, written every 100th embedded event or on ctrl-c, so a crash or a `docker stop` could drop up to 99 recent events from the index, and an event without an embedding never triggered a snapshot at all. ### Added - `GET /v1/memory/event/` returns a stored event: kind, time, text, properties. - `POST /v1/memory/recall` ranks by keyword (BM25 over event `text`), by embedding, or by both fused with reciprocal rank. `kind`, `since` and `until` filter the result; `hops` follows `SOURCE_FROM` edges out from the hits. On LoCoMo the fused ranking raises evidence recall@20 from 0.869 to 0.889 over embedding-only retrieval (0.878 to 0.896 on the five conversations held out from tuning); `BENCH.md` has the tables. - `POST /v1/memory/events` appends up to 1,000 events as one commit and one fsync. The batch is stored whole or not at all. - Events take `text` (indexed for keyword recall) and `ts` (event time in microseconds; defaults to the append time). - `kind`, `since` and `until` filters on `POST /v1/memory/search`. - `MEMSPINE_METRICS_BIND` is implemented: `/metrics` is served on that address and removed from the data-plane listener. The variable used to be read and ignored. - `memspine-bench --mode scale` measures ingest, search and open time on synthetic events; `--retrieval hybrid`, `--hops`, `--text-weight` and `--text-floor` for LoCoMo runs. - Rust SDK: `get_event`, `recall`; `append_events` uses the batch endpoint. Python and TypeScript bindings: `get_event` / `getEvent`, `recall`. - `Cargo.lock` is tracked. ### Changed - Search hits and Cypher rows carry the event's text and properties. Top-level string, number and boolean properties are stored as native node properties, so Cypher can filter and sort on them (`WHERE n.speaker = 'ana'`). - `POST /v1/memory/cypher` returns `{"columns": [...], "rows": [[...], ...]}` and accepts `params`. It was `{"rows": [, ...]}` or `{"rows": {"count": n}}`. Queries are read-only; `CREATE`, `MERGE`, `SET` and `DELETE` get a 400. - Vector search is exact cosine similarity. Hits carry `score` (cosine similarity) and `distance` (`1 - score`); `distance` used to be squared L2. For unit-length embeddings the ranking is the same. Search time is linear in the number of events in the tenant: 2.4 ms at 10,000 events and 31 ms at 100,000 (384 dimensions, M-series laptop). - An append is durable when it returns: node, embedding, text and edges go to FFS as one batch and the log is fsynced once. A restart replays the log. - `Idempotency-Key` works in both auth modes. The event id is derived from `(tenant, key)` and the engine does not write an id twice, so a retry returns the first id and stores nothing. The SQLite `idempotency_keys` table is no longer used. - Requests the caller can fix get a 400 with `{"error": "invalid_request", "detail": ...}`: a wrong embedding dimension (was a 500), an all-zero or non-finite embedding, malformed JSON (was axum's plain-text reply), a bad `Idempotency-Key` (was a 401). - A 500 response no longer includes the underlying error text, which could name files on the host. The cause is logged with the response's trace id. - Engine calls run on tokio's blocking pool, and reads within a tenant run concurrently. Opening one tenant no longer blocks requests for the others. - `GET /v1/whoami` also returns the key's `scopes`. - Minimum Rust version is 1.82, which is what the FFS dependency already required. ### Fixed - The admin key was written to the log at startup: `run` logged its config with `{:?}` and the config held `MEMSPINE_ADMIN_KEY` in plain text. The config's `Debug` output now redacts it. Rotate the key if those logs left the host. - Requests authenticated with an `smk_live_` key were never rate limited. The limiter parsed the bearer token as a tenant UUID, which only matches v0 mode. It now keys on the tenant the key resolves to. - The server only shut down cleanly on ctrl-c. It now also handles SIGTERM, which is what `docker stop` and Kubernetes send. - The rate limiter's idle-tenant sweep was written but never started, so its map grew without bound. - The Docker `HEALTHCHECK` called `wget`, which `debian:bookworm-slim` does not ship, so containers reported unhealthy. The runtime image installs it. - The README's `cargo build --release` fails to link the Python binding on macOS; the documented build command now names the two server binaries. ### Removed - `predicate` on `POST /v1/memory/search`. It always returned 501. A request that sets it now gets a 400 pointing at `kind` / `since` / `until`. - The per-tenant `.hnsw` and `.rels` snapshot files, and the HNSW index with them. ### Upgrading from an untagged build - **Data is not migrated.** Tenants are now `//events.ffs.*`. The old flat `.ffs / .wal / .hnsw / .rels` files are left in place and not read; the server logs a warning when it finds them and those tenants start empty. Re-ingest from your source of record. - An all-zero embedding is rejected. Omit `embedding` for events that have none. - Clients that read `rows` from the Cypher endpoint need the new shape. ### Known limits - No delete endpoint. FFS keeps a deleted node's embedding in its vector index. - Vector search is exact, so latency and memory grow linearly with tenant size; a 100,000-event tenant peaked at 1.2 GiB resident. - An open tenant stays in memory until the process exits. - CI has failed at checkout since the `FFS_READ_TOKEN` secret expired. This release was checked locally: `cargo fmt --check`, `cargo clippy -- -D warnings`, and 116 tests. The Python and TypeScript binding crates were type-checked; their test suites were not run.