Latest offline results · October 24, 2026
Rerun on an Apple M4 with Node 25.3 and o200k_base token counts. No paid model calls. The engine measured here, Fast, is a pre-release local engine; it is not in the published supercompress-proxy package yet. Each suite measures something different, so read cut and quality together. We don’t publish latency from this run because other jobs shared the machine.
Single-fact tasks
99.2–99.7%
Cut with the planted answer kept in all 540 tasks. One fact per task, so this is the easy case.
Every-fact corpus
78/84
At a 72.9% cut. Every fact and valid structure required. Down from 84/84; see known issue.
Real-world dumps
88.6%
Cut on 12 CI logs, traces, diffs and agent chats, with every gold string kept.
Agents answering
24/24
Correct from Fast-compressed context at a 99.46% cut. Oct 22, 72 isolated sessions.
Compression against other tools
The single-fact column plants one answer per task. The every-fact column requires every fact in the case to survive, and fails a case whose CSV, JSON or code fence breaks.
| Tool |
Single-fact (108) kept · cut |
Every-fact (84) passed · cut |
Broken structure |
| SuperCompress Fast (pre-release) |
108/108 · 99.7% |
78/84 · 72.9% |
0 |
| Caveman engine bin-v4.0.1 |
102/108 · 50.6% |
78/84 · 18.6% |
6 (multiline CSV) |
| Headroom, local profile |
90/108 · 46.4% |
84/84 · 10.5% |
0 |
| LiteLLM, lexical compression |
69/108 · 33.0% |
66/84 · 43.3% |
3 |
| Caveman Cloud (Oct 21 run) |
— |
77/84 · 18.6% |
6, plus 1 timeout |
Single-fact run: Headroom 0.37.0, LiteLLM 1.103.2. On the separate 180-task needle suite, Fast kept 100% at a 99.7% cut and the Caveman engine kept 94.4% at 50.9%.
Known issue: Fast drops numbered procedure steps. All six of its every-fact failures are one case family. When a markdown heading matches the question, Fast keeps the heading and drops the numbered steps under it, keeping 0 of 4 facts while reporting a 99.6% cut. The October 21 engine passed the same corpus 84/84 at a 59.65% cut. We’ll publish a rerun once it’s fixed.
Paraphrase and language. On 24 semantic cases (paraphrase, negation, two-hop, aggregation, multilingual), Fast passes 17 at a 37.8% cut and fails all three multilingual questions. Headroom passes 22 at a 3.8% cut. LiteLLM and Caveman Cloud pass all 24 by returning the input unchanged.
Agents answering from compressed context. On October 22, 72 isolated agent sessions each answered one task from original, Fast-compressed or Caveman-compressed context. Original and Fast both scored 24/24; Caveman scored 23/24 at a 42.32% cut. No answer was wrong: Caveman’s miss came back UNKNOWN because the answer had been cut. These are single-fact tasks, and the engine has changed since that run.
Hosted API and Neural Keep. Not re-benchmarked. An October 17 run of the hosted path had timeouts and over-cap rejections, so we make no current hosted quality or latency claim. The September figures under Historical experiments are not current.
Router · not launched
Offline tier classification and synthetic selector costs. No requests were sent to models, so these numbers show which tier the router picks and what that costs at list price, not whether the task succeeds.
| Test |
Result |
Notes |
| AutoRouter’s 18 published example turns |
18/18 tiers matched |
$0.81 at list price vs $2.17 for AutoRouter’s picks and $4.59 for the top model on every turn |
| 59 held-out coding-agent prompts |
56/59 |
All three misses routed up a tier, none down. Prompts were written and labeled by us before the run. |
| Selector vs LiteLLM 1.106.0, 2,000 synthetic requests |
$24.36 vs $24.92 |
Same quality and capacity gates for both, 0 violations each. Prices and outcomes are synthetic. |
Every figure above is in offline-benchmarks-2026-10-24.json. None of it comes from the release protocol below.
What ships with every result
Frozen corpus and configuration hashes, exact returned engine/model identity, source revisions, hardware and region, warmup exclusions, cache policy, raw per-request rows, failures, cost coverage and confidence intervals. Short or capped runs do not qualify as completed release trials.
Latency is client-observed, non-streaming end-to-end time. Concurrent paired workers are not a claim about isolated per-arm service capacity. Token reduction is not answer quality, and a subscription allocation is not an invoice.