Retrieval-augmented generation became the default enterprise AI pattern quickly and for good reason. Convert documents into vectors, find the passages most similar to a question, hand them to a model, generate a grounded answer. It is straightforward to build and works well for a large class of problems.
It also has a specific limitation that becomes apparent precisely where the commercially valuable questions live.
What RAG is actually doing
Vector search finds passages semantically similar to the query. Similar, not relevant, and certainly not authoritative or complete. The system has no model of what your documents mean or how the facts inside them relate to one another.
For a question like what is our refund window, that is fine. The answer sits in one passage, and similarity search finds it.
For a question like which of our current suppliers are affected by the new packaging regulation, and which contracts need renegotiating before March, similarity search is close to useless. That answer requires knowing which suppliers exist, which contracts govern them, which contracts contain packaging clauses, which clauses conflict with the regulation, and which renewal dates fall before March. The information exists across dozens of documents, but the relationships between those facts exist nowhere in the text.
The specific failure modes
Multi-hop questions. Anything requiring facts from document A to locate the relevant part of document B. Retrieval gets one hop and stops.
Aggregation. How many, which ones, what is the total. Retrieval returns a handful of passages; it cannot count across a corpus.
Negation and absence. Which clients have not signed the updated terms. Absence is not a thing similarity search can find.
Temporal reasoning. Which version of this policy applied when this claim was filed. Documents rarely carry the metadata to answer this, and the model has no concept of supersession.
Contradiction. When retrieval surfaces two conflicting passages, the model typically synthesises them into something plausible and wrong, rather than flagging the conflict.
What knowledge-augmented generation adds
KAG puts a structured representation of your domain between the documents and the model. Entities are extracted and given identity. Relationships between them are made explicit. The result is a graph the system can traverse rather than a pile of text it can only search.
Concretely: instead of hoping the phrase governing contract appears near the supplier name in retrievable text, there is an edge in a graph linking that supplier to that contract, with a start date, an end date and a status.
Once the relationships are explicit, the earlier failure modes become tractable. Multi-hop questions become graph traversal. Aggregation becomes a query. Absence becomes checkable, because you can enumerate entities and test which lack a relationship. Temporal questions become answerable when the graph carries validity periods.
The distinction in one line: RAG finds text that resembles your question. KAG reasons over facts that are actually related to it. Both have a place, and the second costs considerably more to build.
The cost of the graph
KAG's advantages are real, and it is important not to present them as free.
Extraction is imperfect and consequential. Building the graph means identifying entities and relationships from unstructured text. Errors here propagate everywhere downstream, and unlike a bad retrieval result they are invisible at query time. A wrong edge produces a confidently wrong answer with no obvious tell.
Schema design is real work. Someone must decide what entity types exist and what relationships matter. This is domain modelling, it needs subject expertise, and it is difficult to change once populated.
Maintenance never stops. Documents change, entities change, relationships change. A graph reflecting last quarter's reality is worse than no graph, because it looks authoritative.
Resolving identity is harder than it sounds. Determining that Acme Ltd, ACME Limited and Acme (Holdings) are one entity, or three, is a genuine problem that consumes real project time.
Hybrid is usually the right answer
In practice, most effective systems use both. The graph handles structure, relationships, and questions requiring traversal or aggregation. Vector retrieval handles nuance, explanation and the long tail of content that resists structuring.
A useful division: model the entities and relationships that appear in the questions people actually ask repeatedly, and leave the rest as retrievable text. This avoids the common failure of attempting to model an entire document estate before delivering anything.
When each is appropriate
RAG alone is sufficient when questions are answerable from single documents, the corpus is relatively static, and the cost of an occasional imprecise answer is low. Internal knowledge bases, policy lookup, support documentation.
KAG earns its cost when questions span multiple documents, when relationships between entities are the substance of the question, when aggregation or absence matters, or when a wrong answer carries regulatory or financial consequence.
Neither is the answer when the underlying data quality is poor. A graph built from unreliable source data is a very expensive way to be confidently wrong. Fix the data first, which is a less interesting project and frequently the correct one.
A realistic sequence
The approach that tends to work is deliberately incremental. Start with retrieval, since it is fast to build and reveals what people actually ask. Log the questions it handles badly, which is the genuine requirements document for the graph. Model only the entities those questions need. Expand the schema when specific questions justify it.
This inverts the instinct to build the complete knowledge model first. That approach produces long projects that deliver nothing for months and frequently model the wrong things, because the questions people actually ask are rarely the ones anticipated in a workshop.