Verbatim lines vs token pruning: how to compress coding-agent context without breaking edits

What is token pruning?

Token pruning methods score individual tokens or words and delete the ones that score low. Microsoft's LLMLingua is the usual example: a small language model estimates how surprising each token is, and low-information tokens are removed. The leftover string is shorter, but it is no longer identical to the source. Words can sit next to neighbors they never sat next to in the original file. A path, an identifier or an error message can lose characters even when most of the surrounding line remains.

That can be acceptable for prose a model only needs to understand. It is a different bet for text an agent will quote or edit. If a compressor deletes a space, a comma or part of a symbol, an apply-patch tool looking for the original line will not find it.

What is line-level selection?

Line-level selection makes a keep-or-drop decision on whole original lines. Each kept line is byte-identical to the corresponding line in the input. Nothing is rewritten, paraphrased or summarized. The shortened result is a subset of the source, which makes it possible to check what the model actually saw against the input you sent.

SWE-Pruner (arXiv 2601.16746) is a research example of this idea for coding agents: it selects relevant lines given a goal hint from the agent. SuperCompress is a hosted, query-aware, extractive example of the same granularity. You send the bulky context and the current question. The hosted model keeps original lines that answer that question and drops the rest. It does not rewrite or summarize. The question itself is not compressed.

Query-aware here means the keep-or-drop choice depends on the question. The same test log can yield a different kept subset when the question changes from "why did test_auth fail?" to "which fixture loaded the user?" Two methods can both be extractive and still disagree on grain: one drops tokens inside a line, the other drops or keeps the line as a unit. See extractive vs abstractive context compression for the rewrite-versus-select distinction.

Why does granularity matter for coding agents?

Coding-agent context is mostly line-structured: stack traces, log lines, diffs, file reads, JSON lines and shell output. An agent has to quote exact paths, identifiers and error text. An edit only applies if the text it quotes still exists in the file or the tool output.

Skip the Compressor (arXiv 2610.21073) studies this directly. The authors report that compressors keep the files to edit but alter their text, so fewer patches apply, and that showing chosen files whole and verbatim did better at equal budget. That result is about how the kept text is shaped, not about any one vendor product.

The same pattern shows up in ordinary agent work. A long pytest run can bury the assertion, the traceback frames and the file path under captured stdout from other tests. Line-level selection can drop unused lines and still hand the agent strings it can copy into a patch. Token pruning can shrink the same log and leave a traceback that no longer matches the file on disk.

Extractive line keeping is also a different design from abstractive compaction such as Claude Code's /compact, which rewrites history into a shorter recap. That can be useful for old conversation. It is a poor fit for the file the agent is about to change. For where SuperCompress sits among command filters, MCP proxies and built-in compaction, see context compression for coding agents.

When should you keep a file whole?

Files the agent is about to edit should stay whole and verbatim. That is consistent with Skip the Compressor and with the advice on the coding-agents guide: query-aware extraction is for bulky output where only a few lines answer the current question, not for the buffer under the cursor.

Keep the file whole when the agent is about to apply a patch, a search-and-replace or a formatted edit to that file; when it needs neighboring lines as anchors, not only the one line that mentions the symbol; or when the file is already small enough that shrinking it does not change the request.

Use line-level selection when the input is a log, a test run, search results or an MCP tool result; when many lines are unrelated to the current question; and when the agent needs a few exact strings (an error, a path, a JSON field) and can ignore the rest. Claude Code PostToolUse hooks are one place this split shows up: compress the bulky tool dump, leave the file being edited alone. See compress Claude Code tool output.

Do not compress the question. SuperCompress leaves the query untouched and only scores the context against it.

How should you evaluate any method?

Measure cost per solved task and success, not token counts alone. Beyond Token Savings (arXiv 2609.32961) and Token Reduction Is Not Cost Reduction (arXiv 2607.12161) both argue that a smaller prompt is not the same thing as a cheaper or more successful run. Extra retries, broken edits and cache misses can erase a token saving.

None of these papers tested SuperCompress. Treat them as mechanism papers, not as a ranking of products.

If you try a compressor on your own work, compare sessions that finish the same task. Check that patches still apply. Compare the provider bill per finished task. Every SuperCompress response reports original_tokens and kept_tokens, so you can inspect each call without taking a vendor headline at face value.

A fair test also separates shrinking bulky tool output from preparing the file the agent will edit. A method that helps with the first can still hurt the second if it rewrites the file. For the broader method list, see what is context compression for AI agents.

npm install -g supercompress-proxy && supercompress setup

Frequently asked questions

Does SuperCompress change my code or error messages?

No. It keeps original lines verbatim. It does not rewrite or summarize them.

Is SWE-Pruner the same as SuperCompress?

No. SWE-Pruner is a research method for agents. SuperCompress is a hosted API and MCP tool. Both select lines rather than prune tokens.

Should I compress the file I am editing?

No. Keep that file whole and verbatim. Use line-level selection for bulky output such as logs, test results, search hits and MCP tool results.

How do I see what SuperCompress kept?

Each response reports original_tokens and kept_tokens. You can check those fields on your own calls.

Sources

• SWE-Pruner (arXiv 2601.16746)
• Skip the Compressor (arXiv 2610.21073)
• Beyond Token Savings (arXiv 2609.32961)
• Token Reduction Is Not Cost Reduction (arXiv 2607.12161)
• LLMLingua on GitHub, as a token-pruning example

Get API key Coding agents guide All alternatives