SuperCompress Architecture

1. System Overview

SuperCompress is a query-aware context compression engine. It sits between your application and the LLM, scores lines of the prompt against the user's question and removes content that scores low on relevance.

The key design choices:

• Neural Keep, hosted query-aware cross-encoder on Fly CPU today (recommended). Separate compiler path: tiny feed-forward on CPU for local preprocessing.
• Query-aware, scores each passage against the user's specific question, not just the content type.
• No model download, the policy is bundled in the pip package (~200KB total).
• Deterministic given inputs, same context + query always produces the same compression. Predictable and auditable.

The system is designed for serverless deployment, the local compiler path is a small CPU preprocess suitable for AWS Lambda, Cloudflare Workers, and Vercel Edge Functions. Do not treat a single latency figure as product-wide.

2. The Compression Pipeline

Step 1: Segmentation

The input context is split into semantic segments. Each segment is a coherent unit, a paragraph, a code block, a list item, or a section of a log file. The segmenter uses heuristics (blank lines, indentation, markdown headers, code fence boundaries) to identify natural breaks.

Step 2: Feature Extraction

Each segment is converted into a feature vector that captures:

• Lexical overlap, n-gram similarity between the segment and the query
• Positional features, where the segment appears in the context (beginning, middle, end)
• Structural features, whether it's code, prose, JSON, markdown, etc.
• Novelty score, information overlap with previously seen segments

Step 3: Scoring

The learned policy takes the feature vector and outputs a relevance score (0–1) for each segment. The threshold is adaptive, the policy adjusts its cutoff based on the distribution of scores and the target compression ratio.

Step 4: Assembly

Segments with relevance scores above the threshold are assembled back into compressed context, preserving their original order. The assembly step also handles edge cases: keeping context boundaries legible, re-inserting necessary transition text, and ensuring the output is valid for the downstream LLM.

3. The Learned Policy

The policy is a small feed-forward neural network with a learned query-aware neural policy. To put that in context:

The small size is intentional. It means:

• The policy fits in L1 cache on modern CPUs, no memory bottleneck
• Scoring runs as a lightweight CPU preprocess before the LLM call
• The entire package (policy + code) is ~200KB, fits in a single network packet
• Runs as preprocess before the expensive LLM call
• Edge deployment is straightforward, the policy compiles to WASM for Cloudflare Workers

The policy was trained on a dataset of ~100K (context, query, keep/drop) examples generated from real production prompts across customer support, code generation, data extraction, and Q&A tasks. Training used a contrastive loss that penalizes dropping segments that contain information needed to answer the query.

4. Compiler Mode

In addition to the basic scoring policy, SuperCompress offers a compiler mode that applies a second-pass optimization. The compiler:

1. Deduplicates, removes repeated information across segments, keeping the most complete version
2. Strips tool noise, removes boilerplate from tool outputs, stack traces, and logging statements that don't contribute to the answer
3. Preserves dependencies, if a later segment references an earlier definition, both are kept even if one scores low individually
4. Reports verification risk, the compiler flags segments where dropping them carries uncertainty, producing a risk score alongside the compressed output

Compiler mode adds a second pass (dedupe, tool-noise strip, dependency keep) and reports verification risk alongside the compressed output. Measure original_tokens / kept_tokens on your traffic. Published suites live on /benchmarks.

5. The Verifier

The verifier is a lightweight post-processing step that checks the compressed output for potential information loss. It answers two questions:

• Entity coverage: Are all named entities from the original context that are relevant to the query preserved in the compressed output?
• Dependency integrity: If the context defines a term or concept that the query references, is that definition still present?

The verifier produces a risk score (low, medium, high) for each compression. Applications can use this score to decide whether to use the compressed output, fall back to the original, or keep the full context.

In practice, most compressions across the benchmark suite receive a low-risk score, and the high-risk cases are typically very short prompts where compression has little room to operate.

6. Deployment Architecture

SuperCompress ships in these forms:

The hosted API runs the Neural Keep model and adds automatic scaling, rate limiting, and usage analytics through the dashboard.

For framework-specific setup, see the integrations docs.

7. Comparison with Alternatives

SuperCompress's architecture differs fundamentally from other prompt compression approaches. Here's how:

For a comprehensive comparison, see the alternatives hub and the full guide.