Benchmarks · Offline results Oct 24, 2026 · Release suite not run

Compression and routing.
Measured separately.

One reproducible suite. Two separate experiments. We measure answer quality, cost and speed—not just how much smaller a prompt gets.

Prepared. Not measured yet. The paid release suite has not run, so its four charts stay empty. Below them are offline results from October 24, 2026, for a pre-release local engine and router, known issues included. The live product remains compression-only. Routing and control plane have not launched.

Latest offline results · October 24, 2026

Rerun on an Apple M4 with Node 25.3 and o200k_base token counts. No paid model calls. The engine measured here, Fast, is a pre-release local engine; it is not in the published supercompress-proxy package yet. Each suite measures something different, so read cut and quality together. We don’t publish latency from this run because other jobs shared the machine.

Single-fact tasks 99.2–99.7% Cut with the planted answer kept in all 540 tasks. One fact per task, so this is the easy case.
Every-fact corpus 78/84 At a 72.9% cut. Every fact and valid structure required. Down from 84/84; see known issue.
Real-world dumps 88.6% Cut on 12 CI logs, traces, diffs and agent chats, with every gold string kept.
Agents answering 24/24 Correct from Fast-compressed context at a 99.46% cut. Oct 22, 72 isolated sessions.

Compression against other tools

The single-fact column plants one answer per task. The every-fact column requires every fact in the case to survive, and fails a case whose CSV, JSON or code fence breaks.

Tool Single-fact (108)
kept · cut
Every-fact (84)
passed · cut
Broken structure
SuperCompress Fast (pre-release) 108/108 · 99.7% 78/84 · 72.9% 0
Caveman engine bin-v4.0.1 102/108 · 50.6% 78/84 · 18.6% 6 (multiline CSV)
Headroom, local profile 90/108 · 46.4% 84/84 · 10.5% 0
LiteLLM, lexical compression 69/108 · 33.0% 66/84 · 43.3% 3
Caveman Cloud (Oct 21 run) — 77/84 · 18.6% 6, plus 1 timeout

Single-fact run: Headroom 0.37.0, LiteLLM 1.103.2. On the separate 180-task needle suite, Fast kept 100% at a 99.7% cut and the Caveman engine kept 94.4% at 50.9%.

Known issue: Fast drops numbered procedure steps. All six of its every-fact failures are one case family. When a markdown heading matches the question, Fast keeps the heading and drops the numbered steps under it, keeping 0 of 4 facts while reporting a 99.6% cut. The October 21 engine passed the same corpus 84/84 at a 59.65% cut. We’ll publish a rerun once it’s fixed.

Paraphrase and language. On 24 semantic cases (paraphrase, negation, two-hop, aggregation, multilingual), Fast passes 17 at a 37.8% cut and fails all three multilingual questions. Headroom passes 22 at a 3.8% cut. LiteLLM and Caveman Cloud pass all 24 by returning the input unchanged.

Agents answering from compressed context. On October 22, 72 isolated agent sessions each answered one task from original, Fast-compressed or Caveman-compressed context. Original and Fast both scored 24/24; Caveman scored 23/24 at a 42.32% cut. No answer was wrong: Caveman’s miss came back UNKNOWN because the answer had been cut. These are single-fact tasks, and the engine has changed since that run.

Hosted API and Neural Keep. Not re-benchmarked. An October 17 run of the hosted path had timeouts and over-cap rejections, so we make no current hosted quality or latency claim. The September figures under Historical experiments are not current.

Router · not launched

Offline tier classification and synthetic selector costs. No requests were sent to models, so these numbers show which tier the router picks and what that costs at list price, not whether the task succeeds.

Test Result Notes
AutoRouter’s 18 published example turns 18/18 tiers matched $0.81 at list price vs $2.17 for AutoRouter’s picks and $4.59 for the top model on every turn
59 held-out coding-agent prompts 56/59 All three misses routed up a tier, none down. Prompts were written and labeled by us before the run.
Selector vs LiteLLM 1.106.0, 2,000 synthetic requests $24.36 vs $24.92 Same quality and capacity gates for both, 0 violations each. Prices and outcomes are synthetic.

Every figure above is in offline-benchmarks-2026-10-24.json. None of it comes from the release protocol below.

Two questions. Two experiments.

01 / Compression

Less context. Same answer quality?

Compare uncompressed context, local v1, hosted v2 and pinned comparison compressors. Keep the answering model, task and decoding settings fixed.

Measure Answer quality · token reduction · full cost · end-to-end latency

02 / Routing · future evaluation

The right model. At what cost?

Compare routers against every fixed model in the same allowed pool. Keep the original request unchanged; no compression in this experiment.

Measure Answer quality · cost per task · latency · failures

Combined pipeline: supported as a separate follow-up, not part of these two release experiments. Its results will never be presented as compression-only or routing-only gains.

Evaluation protocol Corpus, long runs, quality gates and cost accounting
  1. Freeze the corpus. At least 600 heldout tasks across 300 independent source groups and six domains. Keep tuning documents out of the test split.
  2. Measure quality downstream. Score the answering model, retain failures, and report paired source-group bootstrap intervals. Evidence retention is a separate diagnostic.
  3. Run long, then stress. Three independently seeded trials of at least one measured hour per experiment, plus six-hour soaks at 1, 4, 16 and 32 paired workers. That is a planned minimum of 54 measured stage-hours across compression and routing — not elapsed time or time already run.
  4. Show the whole bill. Include input, output, cache usage, compressor and gateway fees. Missing cost stays unknown; cached and uncached scenarios are disclosed separately.

Results, after the final runs.

No measurements yet. These four panels will show reviewed results—not simulated data or numbers carried over from older experiments.

Compression and downstream qualityv1 / v2 / uncompressed / comparison arms

Context reduction (%) × downstream task score

Routing quality–cost frontierFuture routing evaluation · not launched

Total USD per completed task × downstream task score

Latency under sustained loadp50 / p95 / p99, with failures alongside

Concurrent paired workers × end-to-end latency (ms)

Six-hour stabilityLatency / errors / throughput through time

Elapsed measurement time × latency, failure rate and throughput

What ships with every result

Frozen corpus and configuration hashes, exact returned engine/model identity, source revisions, hardware and region, warmup exclusions, cache policy, raw per-request rows, failures, cost coverage and confidence intervals. Short or capped runs do not qualify as completed release trials.

Latency is client-observed, non-streaming end-to-end time. Concurrent paired workers are not a claim about isolated per-arm service capacity. Token reduction is not answer quality, and a subscription allocation is not an invoice.

Historical experiments · September 2026 and earlier

Archived context only. The numbers and engine descriptions below belong to older experiments, not the pending release suite or a promise of current hosted performance.

v2 launch · Sep 2026 · 390 public cases · evidence containment

Historical B5 experiment: 64.1% context reduction.

SuperCompress v2 is a ~400M-parameter Neural Keep engine on the hosted API (Fly CPU today). We measure whether required evidence survives compression — not whether a downstream chat model finishes the task. Downstream LLM eval has not been run yet.

Public cases 390 Launch suite across needles, LongBench slices, multihop, coding agents, scaling.
B5 mean cut 64.1% Coding-agent dumps: 16,647 → 5,148 tokens. Max 96.6% at ≥99% retention.
Evidence held 24/24 Required evidence lines kept on every coding-agent case.
B5 latency p50 ~5.8s Hosted Fly CPU Neural Keep path — not the millisecond compiler path.

Two engines — do not mix numbers

Recommended · hosted Neural v2

~400M query-aware cross-encoder. Hosted Fly CPU inference. Highest-quality context selection. These launch numbers.

Fast / local Compiler

Lightweight local path. CPU. Millisecond-class preprocessing. Legacy results live in the labeled section below — separate product.

Metric honesty Evidence, not answers

We score containment of required lines. Prefer “preserves required evidence” over “same answers” until downstream eval ships.

Coding-agent suite (B5) — headline

Held-out coding-agent dumps. Same scorer for every method. Tokenizer: tiktoken cl100k_base.

Method Mean cut Evidence pass Tokens in → out p50 latency
SuperCompress v2 (Neural Keep) 64.1% 24 / 24 16,647 → 5,148 ~5.8 s
Headroom 0.36.4 14.8% 18 / 24 16,647 → 14,009 ~0.3 s
LLMLingua-2 52.7% 12 / 24 16,647 → 8,030 ~0.7 s
Truncation (keep last ~35%) 64.3% 11 / 24 16,647 → 5,851 <1 ms
SuperCompress v1 (compiler) 60.7% 21 / 24 16,647 → 5,918 ~10 ms

Raw JSON: launch-benchmark.json · writeup: v2 launch.

Aggregate honesty (all 390 cases)

Mean cut across the full 390-case mix is ~3.6%. That is intentional, not a failure. On needle / LongBench / multihop slices the engine often keeps nearly everything when every line could matter — it knows when not to compress. The 64.1% figure is the coding-agent (B5) suite, where dumps are noisy and query-aware keep shines.

Max reduction at ≥99% evidence retention: 96.6%. Aggregate mean quality retention (containment scorer): ~85%. Downstream LLM completion eval: not run.

How we measured

  • Quality metric: answer / needle / evidence containment in the compressed text (original wording).
  • Not measured yet: downstream LLM task success after compression.
  • v2 inference: query-conditioned cross-encoder (~400M), hosted Fly CPU path for API.
  • Comparators: Headroom 0.36.4, LLMLingua-2, truncation, and SuperCompress v1 compiler on B5.

Limitations

  • Latency: Neural v2 B5 p50 is ~5.8s on the hosted Fly CPU path — not millisecond-class.
  • Downstream eval: evidence containment ≠ proven answer quality from a chat model.
  • Workload fit: strongest on coding-agent / tool-dump noise; conservative on dense needle/QA mixes.
Legacy · compiler engine benchmarks

Compiler / v1 results (not Neural v2)

These numbers are from the lightweight compiler path (CPU, millisecond-class). They are not SuperCompress v2 Neural Keep results. Kept here for history and for teams still on the local compiler.

Legacy metric Result Notes
Held-out gold containment 188/189 (99.5%) Checked-in compiler real / fresh4 / fresh5 snapshots; scoreable answers only. Fresh6 cold: 64/72 (88.9%), failed gate.
Mean cut (compiler) ~58–66% Maximize drop with ≥98% answer-keep gate
Fixed 35% budget oracle recall 100% Public 8-seed suite vs ~25% truncation
Typical latency ~10–47 ms CPU Local compiler — not the hosted ~400M Fly CPU path

Older JSON snapshots remain under /assets/data/*-benchmark-latest.json. Prefer launch-benchmark.json for v2.

FAQ

Why isn’t aggregate mean cut ~64%?

64.1% is the coding-agent (B5) suite. Across all 390 cases many slices are already dense; v2 refuses to over-cut when evidence is everywhere, so the aggregate mean sits near ~3.6%.

Does “24/24” mean same answers from GPT/Claude?

No. It means required evidence lines were present after compression. Downstream completion quality is a separate eval we have not published yet.

GPU or CPU?

Hosted Neural v2 → Fly CPU today (seconds-scale on long dumps). Compiler → CPU, milliseconds. Pick the path that matches the product you ship.

Ship with less context

Historical experiments are not a pricing source. See the dashboard for current plans.

Get API key   Historical v2 notes →   Playground →