Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood.
To make an anology, in my mind, this is akin to saying "fuel air mixture system" when referring to direct fuel injection specifically, when of course, a carburetor also lives in that category.
I agree. We're seeing more variants of "RAG" that aren't semantic at all (e.g. coding agents or simple memory systems that feed summary indexes directly into context).
I think, over time, it's going to become a SQL / NoSQL sort of divide. There will be the right kind of RAG for the job and lots of forcing the wrong kind because the developer doesn't understand the nuances.
RAG, as far as I understand, is a term that came about before LLM tool-calling was as prevalent. Your options were to have an LLM hallucinate up a response, or instead do a [document -> chunk -> embedding -> vector db -> query -> context window] pipeline. I haven't heard anyone talk of LLMs + web search or other tool calls as RAG, even though if you pull apart the semantics the term is applicable. In fact I don't hear people talk about RAG much at all. I suppose much of what people were trying to solve with document chunking/embedding pipelines has been solved with bigger models and tool calls. And along with that change in tooling we have left behind the term "RAG", which leaves it attached to the concept of those pipelines.
I do wonder if the term will make a comeback, Retrieval Augmented Generation as a concept is a fairly fundamental idea, or maybe it's considered too generic which is why tool calling is used in favour if it?
The problem with tool calling is it's too generic in my mind, maybe RAG will make a return when we get around to having different flavours of it, digging into the rich vein that is Information Retrieval
Also is it just me that doesn't like this sort of wording?
> agent has to hold
I find it very generic, I'd much prefer process or recall or any term that indicates what the agent is doing with tokens in that context
It is to me. And I agree that the term is losing value because it's becoming ubiquitous but it's the differentiation from the first versions of ChatGPT etc., which were purely user input -> LLM -> output driven.
The risk in relevance-based pruning is the same one summarization has: it's tuned to drop whatever's rare, and in a lot of domains the rare chunk is the whole answer. A contraindication, an "except when," the single row that contradicts the other forty. Those score low against the query precisely because they're phrased in the exception's language, not the question's, so a similarity cut throws them out first.
The strongest version of the pro-pruning case is "prune by relevance to the query, not blindly," and I still don't trust it where being wrong is expensive, because relevance-to-the-query is exactly the signal that misses the buried caveat. If I ask whether drug A interacts with drug B and the warning is written under B's contraindications without naming A, semantic pruning helps me lose it.
What's worked better for me is pruning by structure instead of by score: keep whole records or whole sections as units and drop by type (this entire category of source isn't relevant) rather than by ranking individual sentences. You give up some of the token savings of aggressive sentence-level pruning, but you stop silently deleting the one clause the answer depended on. If you must prune fine-grained, the honest test is recall on a set where the answer hinges on a rare chunk, not average-case QA where there's lots of redundancy to spare.
"relevance-to-the-query is exactly the signal that misses the buried caveat."
You are exactly hitting the nail on the head. The underlaying problem of a pruner is to determine document relevance to a user query. And document relevance comes in different flavours:
1. Direct relevance: a document can be relevant to the user query because it directly answers it.
2. Subquery-Conditional Relevance: a document can be relevant because it answers a decomposed subquery of the user query.
3. Document-Interdependent Relevance: a document is relevant only because another document provides bridge context, domain knowledge, disambiguation, a definition or a constraint.
This means the right question is therefore not "is this document relevant to the original query in isolation?", but:
Given the original user query, the decomposed sub-queries, and all other retrieved documents, does this document contribute to form a complete and sufficient set of documents for answering the original query?
This is exactly why we tuned our pruner at kapa based on recall against this Document Relevance Model.
Cliche topic - from a few years ago (the "RAG is dead" vs "All You Need Is Advanced RAG" BS - it came in waves and cycles, spread by bots on social media networks).
"Pruning RAG Context" is trying to recycle the old stuff (again), presuming the reader is naive (implies kapa.ai is not going anywhere). The current cycles were "openclaw" (I think that died), now we are on "harnesses" - when that dies the paid social media bots will give you something else. Shell game.
Just declare / define dictionary as a variable in your prompt to carry forward (when you decide to continue using LLMs for certain things). Also either summarize or truncate history. 3-4 year old concept. Not a big thing.
"RAG is dead" was declared because of ever expanding context windows.
However, resources consumed expand so much with context that I think there is a huge practical barrier there. In the beginning AI services were basically free so nobody cared but this is rapidly shrinking.
Personally I see more in a combination of RAG and live querying during the thinking process (e.g. by tools).
Also I don't think dumping any context that might be relevant into the model really helps accuracy. In my experience models just get lost when they get an overload of irrelevant stuff in their context and start overlooking the relevant parts even if it does fit the window.
To make an anology, in my mind, this is akin to saying "fuel air mixture system" when referring to direct fuel injection specifically, when of course, a carburetor also lives in that category.
I think, over time, it's going to become a SQL / NoSQL sort of divide. There will be the right kind of RAG for the job and lots of forcing the wrong kind because the developer doesn't understand the nuances.
The problem with tool calling is it's too generic in my mind, maybe RAG will make a return when we get around to having different flavours of it, digging into the rich vein that is Information Retrieval
Also is it just me that doesn't like this sort of wording?
> agent has to hold
I find it very generic, I'd much prefer process or recall or any term that indicates what the agent is doing with tokens in that context
It doesn't have to be a vector database.
Kapa is one of the few companies doing RAG right.
it will be an intellectual bottleneck when it comes to processing the very information your just arduously fished out of the ocean!!
You are exactly hitting the nail on the head. The underlaying problem of a pruner is to determine document relevance to a user query. And document relevance comes in different flavours:
1. Direct relevance: a document can be relevant to the user query because it directly answers it. 2. Subquery-Conditional Relevance: a document can be relevant because it answers a decomposed subquery of the user query. 3. Document-Interdependent Relevance: a document is relevant only because another document provides bridge context, domain knowledge, disambiguation, a definition or a constraint.
This means the right question is therefore not "is this document relevant to the original query in isolation?", but:
Given the original user query, the decomposed sub-queries, and all other retrieved documents, does this document contribute to form a complete and sufficient set of documents for answering the original query?
This is exactly why we tuned our pruner at kapa based on recall against this Document Relevance Model.
"Pruning RAG Context" is trying to recycle the old stuff (again), presuming the reader is naive (implies kapa.ai is not going anywhere). The current cycles were "openclaw" (I think that died), now we are on "harnesses" - when that dies the paid social media bots will give you something else. Shell game.
Just declare / define dictionary as a variable in your prompt to carry forward (when you decide to continue using LLMs for certain things). Also either summarize or truncate history. 3-4 year old concept. Not a big thing.
However, resources consumed expand so much with context that I think there is a huge practical barrier there. In the beginning AI services were basically free so nobody cared but this is rapidly shrinking.
Personally I see more in a combination of RAG and live querying during the thinking process (e.g. by tools).
Also I don't think dumping any context that might be relevant into the model really helps accuracy. In my experience models just get lost when they get an overload of irrelevant stuff in their context and start overlooking the relevant parts even if it does fit the window.