Research
LoCoMo self-run
notes
What we measured, how we measured it, and what we are not claiming. Numbers verified against the raw result JSON from 2026-07-10.
Headline
v0
31.7%
488/1540
v0.2
77.9%
1200/1540
Same dataset and question set both runs. Jump is methodology parity with the open eval stack plus extractor fixes — not a different dataset.
Method
- Dataset
- LoCoMo-10 (non-adversarial)
- Questions
- 1540 (adversarial category excluded)
- Harness
- mem0 open eval harness (memory-benchmarks-style)
- Extract
- gpt-4o-mini
- Answer
- gpt-4o-mini
- Judge
- gpt-4o-mini
- Embeddings
- text-embedding-3-small
- Recall budget
- 6,500 tokens · k=150
- Store
- InMemoryBeliefStore (accuracy focus; latency not claimed)
By category
Grey = v0 · Red = v0.2. Fractions are correct / total in that category.
Product suite (our wedge)
On a separate, small harness aimed at staleness and conflict (location update, preference flip, job change, multi-fact, scope isolation): mem01 5/5. Same harness, mem0 OSS scored 2/5.
Internal staleness/conflict product suite (location flip, preference flip, job change, multi-fact, scope isolation). This is not LoCoMo; it is the failure mode we optimize for.
What we are not claiming
- mem0’s published ~92.5% LoCoMo figure uses a stronger gpt-4o-class answer/judge stack. Our self-run uses gpt-4o-mini throughout for cost and reproducibility.
- Single full run; LLM judge noise is roughly ±2 points on 1,540 questions.
- In-memory store for this accuracy run — we do not claim production latency numbers from it.
- Per-query token/latency telemetry was not recorded for this run (backlog).
Bottom line
Under a fixed, disclosed stack (gpt-4o-mini everywhere), mem01 moved from 31.7% → 77.9% on LoCoMo-10. Gains show up across multi-hop, temporal, open-domain, and single-hop. Headline vendor numbers with stronger models are a different comparison — we keep our model choice public.
Back to site