SuperCompress

RAG · Updated 2026-11-15

How to compress RAG context before the LLM call

By

Retrieval picks chunks because part of each one matches the question, and the rest of the chunk rides along into the prompt. To shrink RAG context, retrieve and rerank as usual, then send the retrieved text and the user's question to a query-aware compressor and pass only the kept lines to the model. Compress per document when you need citations, fail open when the call errors, and judge it by the answers, not just the token count.

Why do RAG prompts get so big?

Top-k retrieval returns whole chunks. A chunk can be selected for one sentence and still carry paragraphs that have nothing to do with the question. Overlapping chunks repeat the same text, and in a chat app the retrieved text often stays in the history and is sent again on the next turn. The model is billed for all of it as input.

Where does compression fit in a RAG pipeline?

Put it after retrieval and reranking and before generation: retrieve, rerank, compress, generate. Reranking decides which chunks to keep. Compression decides which lines inside those chunks to keep. Fix recall first: compression can only remove text, so it cannot rescue an answer that retrieval never found.

Filter, extract or summarize?

ApproachWhat it doesGood fitWatch out for
Filter chunks (reranker or score threshold)Keeps or drops whole chunksClearly off-topic chunksA long chunk with one useful line still goes in whole
Extractive compressionKeeps original lines or spans that match the questionAnswers that must be quoted or citedCan drop a line the answer needed, so test on your own questions
Abstractive compression (summarize)Rewrites chunks into shorter textGist-style answersCan drop caveats, numbers or conditions, and the text can no longer be quoted

Many pipelines filter first and then extract from what is left. For the trade-off in more depth, see extractive versus abstractive compression.

How do I compress RAG context with the SuperCompress API?

SuperCompress is extractive and query-aware. You send the retrieved text as context and the user's question as query, and it selects original lines against that question rather than writing a summary, so text in the kept lines stays as written. Preprocessing may rewrite JSON or collapse repeated stack frames before lines are chosen. For the mechanism, see how prompt compression works.

curl -X POST https://www.supercompress.dev/api/v1/compress \
  -H "X-API-Key: $SUPERCOMPRESS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"context": "<retrieved text>", "query": "<the user question>"}'

Put compressed_text in the prompt in place of the raw chunks. Each response also reports original_tokens and kept_tokens. They are estimates, not provider-billed totals, but they show what each call removed. When compression would not save enough, the response sets passthrough to true and returns your original text unchanged.

Send the question the user actually asked. Without a specific question the API falls back to a broader digest (query_mode is "dump") instead of a targeted cut. The full request and response reference is in the API reference.

How do I keep citations working?

Compress each retrieved document, or each group of chunks from one source, in its own call, and keep the source ID and URL in your application next to the result. That way every kept line maps back to its source without relying on an ID line surviving inside the text. Keep the original documents for display, so a citation can link to the full passage.

import os
import time
import requests

API_URL = "https://www.supercompress.dev/api/v1/compress"

def compress_for_question(text, question, timeout):
    r = requests.post(
        API_URL,
        headers={"X-API-Key": os.environ["SUPERCOMPRESS_API_KEY"]},
        json={"context": text, "query": question},
        timeout=timeout,
    )
    r.raise_for_status()
    return r.json()

deadline = time.monotonic() + 10  # one time budget for the whole compression step
parts = []
for doc in reranked_docs:  # output of your retriever and reranker
    kept = doc.text  # fail open: default to the original chunk
    remaining = deadline - time.monotonic()
    if remaining > 0:
        try:
            kept = compress_for_question(doc.text, question, remaining)["compressed_text"] or doc.text
        except requests.RequestException:
            pass
    parts.append(f"[source: {doc.id}]\n{kept}")

context_for_model = "\n\n".join(parts)

Fail open. Give the whole compression step one time budget, and if a call errors or the budget runs out, send the uncompressed chunks for the remaining documents rather than an empty context, so the model never answers from nothing.

When should I skip RAG compression?

For a fuller decision guide, see when to use prompt compression.

How do I know it helped?

Frequently asked questions

Is RAG compression the same as reranking?

No. Reranking chooses which chunks go into the prompt. Compression trims the lines inside the chunks you kept. They work well together: rerank first, then compress what is left.

Will compression change the text of my sources?

SuperCompress selects original lines rather than writing a summary, so text in the kept lines stays as written. Preprocessing may rewrite JSON or collapse repeated stack frames before lines are chosen, so keep the original documents for display and citation.

Do I need to send the user's question?

Yes, for a targeted result. Without a specific question the API returns a broader digest and labels the response query_mode "dump".

How do I measure whether RAG compression is worth it?

Compare answers to the same questions with and without compression, and log original_tokens and kept_tokens per call. Those counts are estimates, so confirm savings on your provider bill.

Get API key API reference Context compression guide