Written by
Aliasgar Khimani
At
Thu Jun 25 2026
82% Accuracy on BEAM: First Benchmark Results
Engrammic achieves 82% overall accuracy on BEAM-1M with 10-15x token reduction. How we retrofitted a structured memory system to work with standard benchmarks.
You're spending 10-15x too much on AI context.
Opus 4.8 pricing: $15/1M input tokens. A 1M token conversation costs $15 per query. With Engrammic, ~$1.20. An agent doing 100 long conversations per month saves $1,380/month, $16k/year.
And better accuracy. 82% on BEAM vs 31% for vanilla RAG.
Results
82% overall on BEAM-1M (LLM-judge eval). Best RAG baselines in the paper: ~31%.
BEAM-1M accuracy (LLM-judge evaluation)
| Category | Score |
|---|---|
| Abstention | 88% |
| Knowledge Update | 82% |
| Contradiction Resolution | 85% |
| Temporal Reasoning | 74% |
| Scale | Engrammic | Full Context | Reduction |
|---|---|---|---|
| 100K | ~7,500 | ~490,000 | 65x |
| 500K | ~35,000 | ~500,000 | 14x |
| 1M | ~80,000 | ~1,000,000 | 12x |
Abstention (88%): Don't know? Say so. Most systems hallucinate.
Knowledge updates (82%): "I'm vegetarian" turn 3, "I started eating fish" turn 47. We track the update. RAG retrieves both.
Temporal reasoning (74%): "What was X before Y?" Graph traversal, not retrieval lottery.
Contradiction resolution (85%): Conflicting claims get flagged. Confidence scoring resolves them.
Facts, not chunks
10k token conversation → 3-5 facts, ~200 tokens. We retrieve facts: epistemic units with confidence and evidence: not raw chunks.
mem0 and Hindsight compress similarly. We're more accurate because we maintain epistemic state, not just store text.
How we got there
Standard benchmarks assume bag-of-chunks retrieval. They weren't built for structured memory, so we had to retrofit. Here's what we hit:
1. Role confusion
The harness seeds raw conversation. Agent sees "USER: I graduated from MIT" and doesn't realize this is about the person it's talking to. It's just text. We had to reframe everything as third-person facts during ingestion: "The user graduated from MIT": so the agent understands who it's describing.
2. No update tracking
User says "I'm vegetarian" in turn 3, then "I started eating fish" in turn 47. RAG retrieves both with equal confidence. The agent has no way to know one supersedes the other. We added supersession detection during seeding: new facts explicitly link to what they replace. Now the agent knows fish came after vegetarian.
3. Seeding speed
BEAM 500K is 560,000 conversation turns. Running each through MCP takes hours. We bypassed MCP entirely for seeding: pre-embed offline, bulk upsert to Qdrant, batch write to Memgraph. 4 hours → 20 minutes.
On reproducibility
Vendor-claimed vs reproduced scores differ wildly across the industry. Our numbers use LLM-judge eval. Raw BEAM paper (strict matching): ~30-35%. The gap is methodology, not magic.
Publishing full methodology so others can reproduce.
Stack
| Component | Choice |
|---|---|
| Embeddings | BGE-M3, TEI, T4 GPU (~8ms) |
| LLM | Gemini 2.5 Flash |
| Graph | Memgraph |
| Vector | Qdrant |
Related
- BEAM: the benchmark
- mem0: memory layer
- Hindsight: opinion networks
- Zep: long-term memory
- LongMemEval: related benchmark
[1] Gemini Flash for bulk eval. Pro/Sonnet: 3-7% higher.
[2] First public run. Still tuning.
[3] BEAM paper uses strict matching (~30-35%). Memory systems use LLM-judge: more lenient, more realistic.
GCP us-west1-b, Gemini 2.5 Flash, BGE-M3 on T4, Engrammic v0.6.0.