Last week I watched a client’s support agent get slower and more expensive the longer it ran. Nothing was wrong with the model. The problem was memory hygiene: every time the agent took a new step, it paid to re-read the entire conversation from the top.
If that sounds familiar, some new research out of Tsinghua is worth ten minutes of your time. A team from Tsinghua University and the Chinese University of Hong Kong open-sourced a system called AKV that lets AI agents cut memory costs by an average of 27.7% — while scoring slightly better on the tasks, not worse. Here is what it does, why it matters for the Mittelstand, and what you can actually use today.
The problem: agents pay rent on their own history
Most AI agents waste money by re-reading the entire chat history every time they take a new step. When that history gets too long, the usual fix is to summarize old data into a compressed recap. That helps with token count, but it has two real costs: summarizing itself burns tokens and latency, and the agent loses fine detail — the exact customs code, the exact date the client complained, the exact number from the carrier portal.
Agent researchers call this context or memory management, and it is one of the quiet line items that makes an “automated” workflow more expensive than the spreadsheet it replaced. I have written about the what an SMB should actually budget for AI automation before, and memory handling is exactly the kind of hidden cost that makes budgets slip.
What AKV does differently
The Tsinghua team’s insight: instead of summarizing, let the agent engineer its own cache. Two moves:
- Drop — the agent marks KV cache entries it no longer needs (say, the raw output of a search it already digested) and removes them from attention. The text history stays untouched on disk.
- Repos — the remaining cache entries are repositioned so the context stays compact and positionally valid. Retained memory is reused as-is with no re-prefilling.
The system is built as a serving engine (AKV, open-sourced under Apache-2.0 on top of SGLang) with an OpenAI-compatible API. In their paper — “Cache Engineering for Agents” — across three models and four agent benchmarks, the average cost fell 27.7% compared to threshold-triggered summarization, and average benchmark scores rose by 1.7 points. On SWE-bench Verified, the cost per task for one model dropped from 20.0 to 9.7 cents while the score went up.
Latency improved too: at concurrency 16, the 95th-percentile request latency dropped from about 230 seconds to about 31 seconds, and output throughput gained up to 1.2×. Agents that manage their own memory also finished long tool-calling runs faster — less time spent re-digesting stale history means more time doing the actual work.
Why this matters for a 12-person logistics firm
Imagine a 12-person logistics firm managing complex shipping disputes. An agent needs to remember specific customs codes, dates, and client complaints across a long thread. Under the old approach, that thread gets summarized every few steps: customs codes blur into “shipping documents,” the complaint from March 3rd becomes “the client was upset about delays.”
With cache engineering, the expensive facts stay in the cache verbatim. The agent handles the dispute with precision — quoting the right code, the right date, the right invoice line — while the owner pays meaningfully less for the API. Nothing about that scenario is exotic: multi-day threads, dozens of tool calls, one stubborn fact that must survive the whole run. That is exactly the workload where summarize-and-forget quietly breaks things and where the 27.7% number would compound.
This is also why I keep saying agents should work like members of your team, not chatbots: a good employee does not re-read the entire case file before every email. They keep the operative facts at hand and drop the rest.
The pairing that makes agents safe enough to trust
Cheap and precise is half the equation. The other half is acting on your behalf without risking your credentials — and this is where the second piece of news from last week fits: 1Password launched 1Password for Agents.
The flow works like this: when an agent reaches a sign-in form, it presents a unique digital key to 1Password. You see which credential it needs and why, and you approve. 1Password — not the agent — fills the credentials into the form, then removes them so the agent can never read or reuse them. The agent gets the outcome (a signed-in session) without ever holding the secret.
Combined with tools like this, and with the pattern we already use to stop agents from going rogue — the principle that an agent is only as safe as its API permissions — these bots can finally act on your behalf safely. The agent holds a permission, not a password. That is the difference between automation you audit and automation you hope for.
The honest limits (because there always are)
Before anyone wires the invoices to a new line item:
- AKV is infrastructure, not a plug-in. It is a serving engine. You cannot install it on top of a hosted API like the OpenAI or Anthropic endpoints in your n8n flow — it applies when you run the model server yourself (self-hosted Qwen, GPT-OSS, or MiniMax-M2 today, on SGLang).
- Model support is narrow. Drop and repos currently cover Qwen, GPT-OSS and MiniMax-M2 text models. That list will grow or it won’t.
- Not universally better. One benchmark combination scored below the previous state of the art in the paper’s own tables (58.9 vs 62.3 on BrowseComp-Plus for one model). Read the tables, not the headline.
- These are the authors’ numbers, in their setups. They are not directly comparable with other published cache systems. I treated them as directionally credible, not as a guarantee for your stack. If you run self-hosted agents on your own hardware, the Qwen-in-n8n setup I documented previously is the class of deployment where AKV-style engineering eventually lands.
What an SMB should do with this today
Three practical takeaways, in order of effort:
- Design agents cache-friendly now. Prefer tool designs that store state (a cache table, a CRM field, a database row) over long threads the model must re-read. Retrieve the two facts you need instead of replaying the conversation. This delivers 80% of the benefit with zero research code.
- Use credential brokers, not pasted passwords. If your agents sign into anything, route them through 1Password for Agents or an equivalent vault-broker pattern. If a vendor asks you to paste an API key into a chat window, treat that as a red flag.
- Watch AKV’s model support list. The moment your self-hosted agent stack is on the supported list and your monthly token bill justifies it, dropping and repositioning instead of summarizing becomes a real option. For most Mittelstand teams, that is a 2027 conversation, not this quarter’s.
So, my usual question, adapted: how many hours a week does your team spend repeating the same context to an AI — and how much of your API bill is history rent?
Related reading
- AI Agents at Work: The Agentic Workforce
- AI Automation Cost: What an SMB Should Actually Budget in 2026
- We Ran 2,400 Prompts Through n8n’s AI Gateway: When Credits Beat API Keys
P.S. We build these practical setups for the Mittelstand in Regensburg — and yes, we will tell you when the research is not yet worth your money. That is most of the time, and saying so is the job.
Sources: Tsinghua/CUHK paper · MarkTechPost analysis · TechCrunch on 1Password for Agents
