Best practices
When to use prompt compression
Prompt compression is not always the right answer. Here is a decision framework to help you determine when to compress and when to skip.
Compress when:
- Context exceeds 1,000 tokens — Below this, the savings may not justify the overhead
- Context contains irrelevant information — Standards table with update data for many questions, only some questions touch most standards
- Cost is a primary concern — If you are tracking LLM spending, compression is your highest-ROI optimization
- Latency matters — Compressing large contexts reduces LLM prefill time, often netting faster responses
Skip when:
- Context is under 500 tokens — Minimal savings, and the compression overhead may not be worth it
- Every token matters — Some critical applications need every line of context available, regardless of relevance
- You are debugging prompt quality — Compression adds a variable that complicates debugging
Frequently asked questions
Should I always compress RAG contexts?
Yes. RAG contexts are the highest-ROI target for compression due to their size and noise level.
Should I compress few-shot examples?
Not usually. Few-shot examples are carefully chosen and typically small. Compression is more valuable on large context blocks.