Token compression guide
LLM token compression before inference
LLM token compression removes waste before the model spends compute on it. The goal is to preserve the evidence needed for the answer while cutting everything else.
What is LLM token compression?
LLM token compression reduces the number of tokens sent to a language model during inference while preserving the information needed to answer the current query.
In a typical RAG query, 60-80% of retrieved context may be irrelevant to the specific question. Those tokens still cost money and add latency.
Token compression methods compared
| Method | Oracle Recall | Latency | Extra Cost |
|---|---|---|---|
| SuperCompress | 100% | ~60ms CPU | None |
| Summarization | 61% | ~63ms + LLM call | Full LLM cost |
| Truncation (FIFO) | 25% | ~57ms | None |
Integrating with your LLM pipeline
pip install supercompress
from supercompress import Compressor
comp = Compressor()
result = comp.compress(context, query)Frequently asked questions
Does token compression require a model download?
No. SuperCompress uses a tiny CPU policy and does not require a model download or GPU.
Does it work with local models?
Yes. Compression is model-agnostic and works before any model call.
Build with less context
Put compression in front of your next LLM call.
Use the hosted API or run SuperCompress locally. Keep the evidence, drop the token waste, and measure the savings before it reaches your model.