I don't like any of current solutions when it comes to compaction. I'd love to have a way to say what exactly should be summarized, because most of the time I just need to compact some noisy MCP tool calls, test runs and things like that. Just let me pick what should be summarized and keep the rest as is.
Just make an extension (or ask Pi to write an extension for itself) that intercepts compaction and leaves only what you want, or rewrites it in any other way. Should be just a few lines.
Compaction is painful if you run just one local LLM, the best way to avoid it is to keep context as small as possible.
One trick I find useful is to have one model with two KV caches running and while first cache has produced tokens, second cache immediately summarizes them during input tokens are being generated (tools time), then harness switches to the second KV cache which takes newly produced input tokens while KV in first cache is getting replaced with compacted summary tokens. This is a kind of ping pong, so we trade more space for less time. Still experimenting but it looks it works, and nice bonus it improves GPU utilization. Btw I have my own harness and model serving code, but it can be easily implemented in any other harness and model server.
Say, what happens when chain of summaries grows so long, that it still overflows context window. Is summarization runned over the summaries in the context window?
What more depth is there to go to? Compaction is a single LLM call (practically) which can have some deterministic diffing/extraction baked in, or multiple LLM calls (generally wasteful). There's only 1 summary in the context window at one time. Every prompt goes [CONVERSATION_HISTORY] + input -> model turn. As soon as total context exceeds that it compacts, so there's no summarisation overflow (you can enable an agent to access past summarisations from past compactions, but the snake starts to eat it's own tail).
The advantage of running local stack is that you can do the compaction at the time of inference, i.e. some tool call runs out of context, you can just pause inference, purge/replace old tool calls with their summaries or just logs by operating directly over tokens on a GPU, rebuilding KV cache (one time prefill hit) and resuming the inference, easily being able to e.g. read 1000 markdowns, each 50k long, in a single LLM call. That's not possible with current agentic harnesses using LLM calls.
Compaction has been a pretty painful part of local llm usage. Scrapping the current context and parsing almosy 128k of context then generating something like 5-10k tokens - that can take quite a while when you’re working with 10t/s-45t/s (depending on the model).
I pretty much just start a new session whenever i fill the context.
In my opinion this is one of the areas where GPUs provide a qualitatively different experience than unified memory boxes.
For an EPYC with a 5090 (no layers on CPU) vs an M3 max 128GB, qwen 3.6 27B at 128k context / 7k generation:
Cold: prefill + decode Hot (KV cached)
5090 40s + 2-3m = 3-4 min 2-3 min
M3 Max 128GB 14m + 8-10m = 22-25 min 8-10 min
This is for dense qwen (which I wouldn't run day to day on the mac) - in reality the mac is quite usable with MoEs but you definitely notice a difference.
I keep my max context really small for personal assistant agents; they don't need it. Especially since compaction keeps anything important around anyway. I use 60k with Pi.
I found hermes to be really lightweight, though I am on a relatively older version and built a custom plugin to lazily load mcps (that's probably in hermes proper by now). Compared to kilo it seems to consume far fewer tokens.
I expect Pi is mostly used with OpenAI plans, and OpenAI has a dedicated compaction endpoint you should probably be using with their models instead of a compaction prompt.
TLDR: It keeps ~20k tokens of recent conversations, then hands the rest of the conversation to another model with a special system & user prompt. This then fills out a template with relevant information.
> Extensions can intercept and customize both compaction and branch summarization
https://pi.dev/docs/latest/compaction
Just make an extension (or ask Pi to write an extension for itself) that intercepts compaction and leaves only what you want, or rewrites it in any other way. Should be just a few lines.
One trick I find useful is to have one model with two KV caches running and while first cache has produced tokens, second cache immediately summarizes them during input tokens are being generated (tools time), then harness switches to the second KV cache which takes newly produced input tokens while KV in first cache is getting replaced with compacted summary tokens. This is a kind of ping pong, so we trade more space for less time. Still experimenting but it looks it works, and nice bonus it improves GPU utilization. Btw I have my own harness and model serving code, but it can be easily implemented in any other harness and model server.
Say, what happens when chain of summaries grows so long, that it still overflows context window. Is summarization runned over the summaries in the context window?
I pretty much just start a new session whenever i fill the context.
For an EPYC with a 5090 (no layers on CPU) vs an M3 max 128GB, qwen 3.6 27B at 128k context / 7k generation:
This is for dense qwen (which I wouldn't run day to day on the mac) - in reality the mac is quite usable with MoEs but you definitely notice a difference.Evil cannot create only imitate.
See: https://github.com/earendil-works/pi/blob/main/packages/codi...
I get it into 1M+ routinely on local models with operations between 50k-85k