Measured. Same judge. Reproducible.
Every number here comes from our own harness on the public LoCoMo and LongMemEval datasets — including the results that went against us. Our rules:
- One judge across every arm of a comparison — judge choice alone moves scores ~10 points.
- Rankings only from identical-harness runs; vendor self-reports don't count as rankings.
- Recall denominators printed; abstentions never count as retrieval hits.
- Negative results published. Every headline ships its re-judgeable per-question dump.
The field
The basis column is the one that matters: ours is measured and re-judgeable; everyone else's is self-reported on their own harness (one 84% claim was later corrected to 58.44% by outside review).
| system | LoCoMo answer accuracy | LongMemEval | basis |
|---|---|---|---|
| 👑 Memoception | 70.2% overall · 62.7% excl-adv · 96.2% abstention | 96.4% retrieval@10 | ✅ measured — re-judgeable |
| Mem0 | 66.9% excl-adv (paper) · 92.5 composite (marketing) | 94.4 (marketing) | self-reported |
| Supermemory | — | 95% Recall@15 (aggregated) | self-reported |
| Zep | 84% → corrected to 58.44% | — | self-reported, corrected |
| Mastra | — | 95% (research page) | self-reported |
Head-to-head — identical harness, identical judge
The only comparison we publish as a ranking: both systems ingest the same conversation, same prompts, same judge (gpt-4o-mini both sides; Mem0 OSS v2.0.12, conversation 0, 199 questions).
Also: search p50 ~28ms local vs 442ms remote · zero-cost ingestion tier. Caveats we publish: one conversation; Mem0's FAISS mode disables their hybrid keyword search.
LoCoMo by category
Full 1,986 questions, claude-haiku-4.5 answerer + judge, zero dropped. Adversarial questions count as correct only when the system abstains.
Same engine under three judges — internally comparable, full runs each: 70.2% haiku-4.5 · 67.7% mistral-small · 63.3% gpt-4o-mini¹
¹ 1,740/1,986 — 246 dropped on rate limits, counted and excluded, never silent.
LongMemEval & cost
- Retrieval essentially saturated: 96.4% answer-session@10 across 470 answerable questions.
- Answer accuracy split published, not hidden: single-session 77–89% · knowledge-update 66.7% · abstention probes 100%.
- Retrieved context ≈ 500 tokens at k=20 — dense memories, not raw transcript dumps.
- Ingestion: 272 sessions → 5,485 dated memories in 6.7 min on a zero-cost tier.
What did NOT work
- A verbatim-details extraction prompt — produced 41% more memories and diluted retrieval. Reverted.
- Wider-retry on abstention — small overall gain, degraded open-domain. Not adopted.
- Biologically-inspired spreading activation over the memory graph — elegant, subsumed by iterative retrieval, moved nothing.
Negative results stay in the record — that's what makes the positive ones believable.
Reproduce it
- Protocol: ingest → retrieve top-k → LLM answers from retrieved memories only → LLM judge grades vs gold.
- Every headline number ships its per-question dump (question, gold, our raw prediction, verdict) — re-grade with any judge model.
- Head-to-head adapters run other systems through the identical pipeline; want yours added? Write us.
- Datasets: public releases of LoCoMo (snap-research) and LongMemEval.
Questions or corrections: hello@memoception.com