Beyond Vector Search, When Enterprises Need a Knowledge Layer

The problem in one sentence

Your discovery system must answer questions that cross documents, legal entities, and time, without blending distinct things that share vocabulary.

Walk into a CIO office and “RAG” is often treated as a single purchase: embed the archive, attach a chat window, declare victory. That works until the first audit asks how the system knew that “Acme Malta” and “Acme Ltd” were the same counterparty, or why an answer cited a superseded policy. At that moment the board is not debating embedding models. It is debating whether the firm can trust machine-assisted discovery at all.

McKinsey’s State of AI 2025 survey reports that nearly nine in ten organisations regularly use AI in at least one function, while only about 6 per cent qualify as high performers attributing more than 5 per cent of EBIT to AI. High performers redesign workflows and govern data access. A knowledge layer is part of that governance when vector-only retrieval demonstrably fails on identity and multi-hop questions.

Vector RAG, plainly: retrieve text chunks by semantic similarity, then ask a model to answer from those chunks. In practice: “Find clauses like this” across 40,000 contracts works until the question becomes “which active contract for Customer X in Germany overrides clause 4.2?”

Knowledge layer, in short: a governed map of entities (customers, contracts, policies, assets) and how they relate, used alongside retrieval so answers follow real structure instead of averaged language. Worked case: the board pack shows not only an answer but the path: Customer X -> Contract 2023-14 -> Amendment B -> superseded clause.

The averaging problem

Vector retrieval finds passages that are “nearby” in embedding space. That is powerful. It is also how systems blur distinct entities that share vocabulary, two policies, two customers, two product lines, into an averaged answer that sounds coherent and is wrong.

In the boardroom, the failure is rarely abstract. A group may have similar contracts across Malta, Italy, Germany, and the UK; one customer may appear under three legal names; an amendment may supersede a clause in only one market. A system that retrieves nearby language but cannot preserve identity will mislead politely.

What you see globally right now is “chat with your documents” becoming a default product claim. That is useful for simple retrieval. It is not enough for discovery questions where the answer is a path through entities, versions, and relationships.

Enterprise AI discovery fails in familiar ways:

1. Averaging. Similar language from different entities gets blended into one fluent answer that sounds authoritative but misstates which customer, policy, or contract applies. The model has no mechanism to distinguish two contracts that share wording but bind different legal entities, so it picks the most semantically similar passage and presents it as the answer. A renewal team acting on that output can quote the wrong liability cap to the wrong subsidiary without realising the system averaged across a border.
2. Multi-hop blindness. The question needs fact A from document 1 and fact B from document 2 in sequence, but retrieval returns only one nearby chunk and the model fills the gap from memory. When the missing hop is a supersession date or a jurisdiction qualifier, the fabricated bridge is plausible and untraceable. Operators cannot tell which part of the answer came from a real document and which came from the model’s prior.
3. Lost-in-the-middle. Relevant evidence is retrieved but buried in a long context window, so the model cites plausible-sounding passages while missing the controlling clause in the middle of the pack. Attention degrades with position, and the clause that actually governs the decision sits between two verbose, irrelevant sections. The system returns a confident answer that a careful reader would overturn in ten minutes.
4. Identity drift. Distinct legal entities such as “Acme Malta” and “Acme Ltd” collapse into one counterparty or disappear entirely, breaking audit trails and renewal negotiations. The drift is silent because embeddings cannot encode legal entity boundaries, only lexical similarity. By the time finance reconciles exposures, the system has already told three teams the same wrong story about who owes what.

Multi-hop retrieval means: answering requires connecting facts from more than one source in sequence, not finding one similar paragraph. On Monday: revenue recognition for a group deal needs the master agreement, the local addendum, and the latest amendment; three hops, and one wrong merge distorts margin.

Entity resolution, put simply: deciding whether two names in text refer to the same real-world thing, and keeping that decision stable over time. Operating case: your CRM says “Acme MT”, legal says “Acme Malta Limited”, finance uses an old trading name; three strings, one golden customer record or three expensive mistakes.

A knowledge layer attacks these failures with explicit entities and relations. It is not a rebrand of “we added a graph database.” It is a commitment to govern identity.

Figure 1. Share of EU enterprises using AI by firm size, 2025. Plain read: larger firms adopt faster; mid-market teams must scope knowledge layers to proven failure modes, not enterprise-wide ontology theatre. **Board reading:** your large competitor may already fund structured discovery, and your answer is narrow depth on the workflows that lose money today. Source: Eurostat enterprise survey, December 2025. AIMonger redraw.

Decision rule: mandatory versus overkill

Mandatory signals

• Explainable path required. Regulated or audit-heavy answers must show how the system reached the conclusion, not only which paragraph looked similar in embedding space. An auditor who asks “why did the system conclude Customer X is capped at EUR 2m?” needs the chain of documents and relations, not a similarity score. Without that path, the answer is indefensible in a review and the firm is relying on the model’s rhetoric rather than its reasoning.
• Repeated multi-hop questions. Operators keep asking questions that join facts across documents, versions, or legal entities, a pattern vectors alone cannot reliably answer. When the same class of question recurs weekly and each instance needs a traversal, the cost of wrong answers accumulates faster than the cost of building the layer. A vector system will approximate these joins and be wrong in different ways each time, which is harder to catch than a consistent, fixable failure.
• Known entity collisions. Incidents already trace to blended customers, policies, or assets that share vocabulary but differ in legal or financial consequence. If the operations team has a spreadsheet of “times the system confused two customers,” that is the failure dossier a knowledge layer is built to fix. Funding the layer without that dossier is architecture speculation; funding it with the dossier is fixing a documented leak.
• Supersession and relationships matter. Amendments, overrides, and parent-child links change the correct answer; similarity search cannot encode which clause is currently in force. A vector store returns the textually closest clause regardless of whether a later amendment voided it, so the system will confidently cite dead policy. The knowledge layer’s job is to carry the temporal and structural edge that similarity cannot represent.

Overkill signals

• Small single-purpose corpus. The archive fits in one index and questions are mostly single-hop FAQ retrieval, so a graph adds cost without fixing a documented failure mode. The team cannot name a question that vectors got wrong in a way a graph would fix. In that setting, graph maintenance is overhead with no offsetting incident reduction, and the budget is better spent on evaluation.
• FAQ-like single hop. Operators need passage lookup, not traversal across entities, versions, or jurisdictions; vectors plus provenance may suffice. If the hardest question is “what does our refund policy say?”, a knowledge layer is a sledgehammer for a paragraph lookup. Provenance back to the source page is the only structured requirement, and retrieval with citation handles it at a fraction of the cost.
• No ontology owner. Nobody is named to maintain entity definitions as products, markets, and legal structures change, which guarantees graph decay within quarters. The graph is only as current as the last entity review, and without an owner it drifts silently. A stale graph is worse than honest vectors, because it lends false confidence to answers that follow relations that no longer hold.
• Latency budget too tight. The workflow requires chat-instant responses and cannot absorb multi-hop query time, so a heavier layer will be bypassed or disabled in production. Operators will route around latency they cannot tolerate, and the knowledge layer becomes shelfware that the team built but never actually serves. If the SLA is sub-second and the traversal is two seconds, the layer is architecturally correct and operationally dead.
• Evaluation and provenance missing. The team still lacks basic proposition tests and source tracing; fix those first before funding structured extraction and entity governance. A knowledge layer on top of an unevaluated retrieval system compounds opacity rather than reducing it. The board cannot tell whether the graph improved answers or merely made wrong answers more elaborate.

McKinsey’s State of AI 2025 pattern, widespread use with scarce enterprise EBIT impact, applies here. A graph without a workflow metric is another sophisticated pilot museum exhibit.

Cost realities boards should hear early

Fund a knowledge layer only when these costs are cheaper than the cost of wrong multi-hop answers in the target workflow.

this is where restraint matters. A narrow entity layer for contracts, policies, or assets can beat an enterprise-wide ontology programme that nobody has the staff to maintain.

Worked example: group contract discovery (synthetic)

A Malta-headquartered group sells across Italy, Germany, and the UK. Legal stores contracts in SharePoint by country. Finance uses one ERP customer code per legal entity. Operations asks: “What is our current liability cap for Customer X in Germany?”

Vector-only path: retrieval returns three chunks mentioning “liability cap” from UK, Italian, and German templates. The model averages language and cites the UK template because it is worded most similarly to the question. The answer is fluent and wrong.

Failure mode: identity drift (Customer X vs subsidiary legal name) plus temporal error (2022 template vs 2024 amendment).

Knowledge-layer path (scoped): entities = Customer, LegalEntity, Contract, Amendment; relations = governs, supersedes, applies_in_country. Query traverses Customer X -> active Contract DE-2024-07 -> Amendment 2 -> liability clause. UI shows the path and document versions.

Business consequence of vector-only failure: wrong cap in a renewal negotiation or audit response.

Pilot scope: twenty propositions like this one, not an enterprise ontology.

Provenance (plain English): a reconstructable record of which sources and steps produced an answer. On the P&L: when external audit asks “why did the system say X?”, you export entities, relations, and document IDs, not a chat transcript.

Sequencing with trust controls

Do not start with ontology theatre. Sequence:

1. Workflow and metric first. Name one process with a baseline measure so absorption, not architecture novelty, decides whether the investment continues. If the board cannot see a before-and-after number on a named workflow, the project is a capability claim rather than an operating change. The metric is what lets finance defend the spend in the next budget cycle.
2. Provenance and holdouts next. Install reconstructable traces and a frozen promotion set before tuning retrieval, so improvements are measurable rather than anecdotal. Without provenance the team cannot replay a wrong answer to diagnose it, and without holdouts they cannot tell whether a tuning run improved generalisation or merely memorised the demo. These two controls are cheap relative to the graph and they protect every later investment.
3. Honest vector baseline. Run vectors alone and document where they fail on identity, multi-hop, or temporal questions; failure evidence precedes graph budget. The baseline is the control the graph must beat. Skipping it means the committee is funding a graph on faith, and faith is not a budget justification a CFO can take to the board.
4. Scoped knowledge layer. Build entities and relations only for proposition families where the vector baseline demonstrably fails, not for an enterprise-wide ontology programme. Scope to the failure dossier, not to the org chart. A graph that covers twenty recurring multi-hop questions and beats vectors on them is a win; a graph that covers the enterprise and beats vectors on nothing measurable is a write-off.
5. Operator training on paths. Teach the people who approve answers how to read explainable traversals, abstentions, and version IDs, not only how to type questions into chat. An operator who cannot read the path will rubber-stamp whatever the system returns, which removes the human check the layer was built to enable. Training is the bridge between the backend capability and the operating safeguard.

Gartner projected mainstream GenAI application use by 2026. That wave includes many “chat with docs” tools. Differentiation for serious enterprises is not chat; it is trustworthy discovery under identity constraints.

Board scorecard

Entity resolution: the silent budget killer

Graphs fail quietly when “Acme Malta Limited”, “Acme MT”, and an old trading name resolve to three nodes or collapse into one wrongly. Entity resolution is labour. Budget humans and time, or do not start.

Operational practices:

• Golden record rules. Define one authoritative identity rule per entity type (customer, policy, contract, asset) so merges and splits follow policy instead of whichever extraction model guessed first. The rule is a written decision about which source system wins when two records disagree, and it removes the ad-hoc judgement that lets collisions fester. Without it, every pipeline run can produce a different entity set, and downstream answers shift from week to week without any logged change.
• Human adjudication queues. Route entity collisions above a confidence threshold to named reviewers with SLA, because silent auto-merge is how graphs fail in production without anyone noticing until audit. The reviewer sees both records, the evidence for each, and makes the binding call with a reason code. This queue is the place where entity resolution labour becomes visible and budgetable, rather than a hidden cost that explodes during a compliance review.
• Change logs for merges and splits. Record every entity merge or split with actor, reason, and effective date so downstream answers can be replayed after identity corrections. When an auditor asks why exposure changed between two report dates, the change log is the answer. It also lets the team undo a bad merge without rebuilding the graph, which is the difference between a ten-minute fix and a weekend of reprocessing.
• Regression fixtures for collisions. Freeze known collision cases as automated tests so a model or pipeline upgrade cannot reintroduce yesterday’s “Acme Malta vs Acme Ltd” mistake. The fixture runs on every build and fails loud if the pipeline re-splits or re-merges a case the team already settled. This is how a graph estate defends its identity invariants against model and schema drift over time.

Explainability operators can use

Explainability that only engineers can parse will not survive contact with audit or customer ops.

Minimum operator view:

1. Entities touched. Show which customers, contracts, policies, or assets participated in the answer so operators can verify the system did not blend the wrong party. A list of entity names with links to their records is the first thing an operator scans before trusting the answer. If the list includes a subsidiary the operator did not expect, that is the signal to stop and check before the answer reaches a customer or a regulator.
2. Relations traversed. List each hop (governs, supersedes, applies_in_country) so audit can follow the reasoning path instead of trusting a summary paragraph. The relations are the reasoning, rendered as a short path the auditor can read in seconds. A summary without the hops is a press release; the hops are the evidence.
3. Source documents per hop. Attach the exact document and section that supported each step, because “the embedding was close” is not an explainable citation. Each hop links to the passage in the source document, so the auditor can open the contract and read the clause themselves. This is what makes the answer defensible in a dispute rather than merely persuasive.
4. Version IDs. Display which document version was active at query time so superseded clauses cannot masquerade as current policy in a board or regulator response. The version ID is the temporal anchor that prevents a 2022 clause from answering a 2025 question. Without it, the system cannot prove it used the in-force document, and the answer is time-insensitive in a domain where time is the governing constraint.
5. Abstention points. State clearly where the system refused to answer because confidence, identity, or linkage fell below threshold; abstention beats a fluent wrong merge. Abstention is a feature, not a gap. It tells the operator where the system’s knowledge ends, which is exactly the boundary the operator needs to know before deciding whether to proceed manually.

If the UI cannot show this, the knowledge layer is a backend hobby.

Hybrid retrieval without magical thinking

Most serious systems combine:

• Keyword and sparse retrieval. Use exact-match retrieval for identifiers, SKUs, registration numbers, and clause numbers where embeddings routinely confuse near-duplicates. A registration plate “ABC 123” and “ABC 123D” are close in embedding space but different vehicles, and only exact match will keep them apart. Keyword retrieval is the cheap, reliable backbone for any field where a single character changes the meaning.
• Vectors for semantic similarity. Deploy embeddings where wording varies but the underlying concept is stable, such as finding policy language similar to a draft question. Two clauses can say the same thing in different words, and embeddings bridge that gap where keyword match sees no overlap. This is the layer’s strength, and it complements rather than replaces the keyword and structured paths.
• Structured paths for multi-hop constraints. Traverse governed entities and relations when the answer depends on joins, supersession, or jurisdiction, not on averaging nearby language. The structured path encodes the rules the business actually operates under, which no similarity score can represent. It is the only layer that can answer “which in-force clause governs this customer in this market today” correctly.

The failure mode is three indexes that disagree. Assign a consistency owner and a reconciliation job. Measure disagreement rate monthly.

When to refuse a knowledge-layer project

Refuse when:

• No honest vector baseline. The team has never measured vector-only accuracy on holdout questions, so the graph proposal is speculation rather than a response to documented failure. The committee is being asked to fund a fix for a problem nobody has shown exists, in a quantified way. The first deliverable should be the failure dossier, not the graph schema.
• No proposition map. Nobody can list which user questions the system must answer correctly, which makes scope impossible to bound and success impossible to score. Without a proposition map, every answer is judgement and every release is opinion. The graph will grow without limit and nobody can say whether it is done, because “done” was never defined.
• No ontology owner. Entity definitions have no named maintainer with weekly capacity, which guarantees the graph becomes stale as soon as the build team moves on. Ontology is a living artefact, not a one-time build, and without an owner it decays on a quarterly cadence. A stale graph is a liability, because it lends false structure to answers that follow relations that no longer hold.
• Chat-instant latency SLA. The workflow cannot tolerate multi-hop query time, so operators will bypass the knowledge layer or disable it under load. The layer is correct in principle and dead in practice, because the SLA will not wait for the traversal. If the latency budget is fixed and sub-second, the graph is the wrong tool for this workflow.
• Competitor-graph business case. The only justification is that rivals mention graphs in conference slides, not that a workflow metric or incident rate improves. Conference slides are marketing, not evidence, and a competitor’s architecture is not a board problem. Fund the layer when your incident dossier demands it, not when a vendor deck implies it.

Appendix: ninety-day knowledge-layer pilot

Failure-case dossier (required before build)

Each candidate failure for a knowledge layer must include:

• User question. Record the exact operator or auditor phrasing that triggered the failure, including paraphrases that should resolve to the same correct answer. The phrasing matters because two wording variants of the same question can fail in different ways, and the fix must cover both. A dossier entry without the real question is a guess at what operators actually ask.
• Vector-only answer. Capture what the baseline system returned so the dossier compares wrong output to ground truth, not to aspirational architecture slides. The wrong answer is the control the graph must improve on, and it is the artefact the committee will ask to see. Without it, the proposal is a graph in search of a problem.
• Failure diagnosis. Classify why the answer failed: averaging, multi-hop miss, identity collision, or temporal error, so the fix targets the failure mode rather than adding generic graph capacity. Different diagnoses need different structures, and a graph that does not address the diagnosed class will not fix the failure it was funded to fix. Classification also lets the team prioritise the failure modes that recur most often.
• Business consequence. State what breaks in money, compliance, or customer trust when this answer is wrong, because that number funds or kills the pilot. A failure with no consequence is an academic curiosity. The consequence is what converts a retrieval problem into a budget line, and it is the number the CFO will challenge first.
• Proposed entities and relations. Name the minimal entity set and edges that would have prevented the error, scoped to this proposition family rather than an enterprise ontology. The proposal proves the team can see the fix, not just the failure. Minimal scope is the discipline that keeps the pilot from becoming an enterprise programme before it has earned one.
• Holdout paraphrase set. Freeze paraphrases used only for promotion decisions, so tuning cannot overfit the demo questions that sold the project internally. The holdout is the honesty layer of the dossier; it is the set the graph must generalise to, not the set it was tuned on. Without it, the pilot reports accuracy on questions it was built to answer, which proves nothing.

If the dossier is empty, the project is speculative infrastructure.

Control of entity changes

Merges and splits of entities are production changes. Require review, regression fixtures for known collisions, and an audit log. Silent entity edits recreate the evaluation theatre problem in structured form.

Worked example: insurance claims entity collision (synthetic)

A mid-market insurer handles motor claims across brands acquired over a decade. Adjusters ask: “Has this registration number been flagged for fraud in another brand?”

Vector-only path: semantic search finds narratives mentioning the plate number and the word “fraud” in unrelated incidents. The model conflates a disputed claim with a confirmed fraud case because both mention “suspicious damage.”

Knowledge-layer path: entity VehicleRegistration linked to Claim, Outcome, and FraudStatus with strict identity rules. Multi-hop query: registration -> all claims -> fraud outcomes. Abstain when linkage confidence is below threshold.

Architecture case: build golden-record rules for registration numbers and VINs before any graph visualization demo.

Evidence base: when unstructured retrieval is not enough

Figure 2. Selected concerns from Deloitte's 2026 survey of 3,235 AI leaders. In short: privacy and explainability sit at the top of the worry list, and unstructured retrieval alone does not satisfy either. **Operating case:** when legal asks for an explainable path, "the embedding was close" is not an answer. Source: Deloitte State of AI in the Enterprise 2026. AIMonger redraw.

Plain-English read: boards should treat explainability and privacy as joint requirements. A knowledge layer earns budget when it improves both on documented failure cases, not when a vendor says “graph.”

Methodology note

This paper does not invent retrieval accuracy percentages. Decisions should be based on organisation-specific evaluation: document the vector-only failure cases first, then measure whether a structured layer improves proposition accuracy, latency, and ops cost on a holdout.

Counter-position: graphs create maintenance debt

They do. Entity resolution, ontology ownership, and multi-index consistency are real costs. The correct response is narrow scope: build entities and relations only for proposition families where vectors demonstrably fail, then expand. Enterprise-wide ontology programmes without a scoreboard are architecture museums.

Investment committee test

Fund a knowledge-layer pilot only if:

1. Documented vector failures. At least ten multi-hop or entity-collision failures from the honest vector baseline are on record with business consequence, not anecdote from a vendor demo. The committee is funding a fix to a measured leak, not exploring an architecture. Each failure entry names the question, the wrong answer, the diagnosis, and the money or compliance consequence.
2. Named ontology owner. A person with weekly capacity owns entity definitions and collision adjudication, not a slide title assigned to “the data team.” Capacity means calendar hours, not a RACI box. Without an owner the graph decays inside two quarters and the investment becomes a liability rather than an asset.
3. Pre-registered holdouts and latency. Accuracy metrics, latency budgets, and promotion holdouts are frozen before tuning begins, so success cannot be redefined mid-pilot. Freezing the holdout is what makes the day-90 decision honest. If the team can change the scoring set after seeing results, the pilot is theatre.
4. Ninety-day graduate or kill criterion. A written decision rule states what accuracy, cost, and ops burden must look like at day ninety to expand scope or stop spend. The criterion is a calendar commitment, not an aspiration. It prevents the pilot from drifting into a perpetual learning phase that consumes budget without ever producing a decision.

Research addendum: GraphRAG is conditional, not universal

GraphRAG, briefly: augment retrieval with a graph built from entities and relationships in the corpus, then answer questions that need community-level or multi-hop structure. Finance reading: useful for “what themes appear across all supplier contracts?”, not automatically worth it for “extract line 14 from this invoice.”

Comparative evidence supports graph-augmented retrieval for corpus-wide, relational, and multi-hop questions (Edge et al., 2024). It does not establish graphs as a universal replacement for vector retrieval. Vector RAG often remains stronger and cheaper for many single-hop, detail-oriented questions, with lower construction and serving cost.

Board rule stands: fund a knowledge layer only after a failure-case dossier shows multi-hop or identity failures that vectors cannot fix on a holdout.

CIO decision standard (investment committee)

Approve a knowledge-layer pilot only when the brief includes:

1. Failure dossier with consequence. Ten or more documented vector-only failures include business impact, not only retrieval scores, so the committee funds a fix rather than infrastructure theatre. The consequence is the line that makes the failure a budget problem rather than an engineering curiosity. Without it, the committee is guessing at the return on the graph.
2. Entity owner with capacity. A named ontology or entity owner has weekly hours allocated for definitions and collision review, not a RACI box with no calendar time. Allocation means the owner’s manager has signed off on the hours, and the work appears in their plan. A named owner without hours is a veto waiting to happen.
3. Frozen holdout paraphrases. The promotion paraphrase set is locked before any extraction or graph tuning, preventing overfit to the questions that justified the budget request. The holdout is the set the graph must generalise to, and it is frozen so the team cannot quietly swap in easier questions after a poor result. This is the control that keeps the accuracy claim honest.
4. Hybrid consistency owner. One role owns reconciliation when vector, keyword, and structured indexes disagree, because three silent indexes produce three silent wrong answers. The consistency owner runs the monthly disagreement job and adjudicates which index wins for each conflict class. Without that role, the indexes drift apart and the team discovers the disagreement during an incident.
5. Operator explainability UI. A mock-up shows auditors and operators the traversable path in plain language, not only engineering logs readable by the build team. The mock-up is a design commitment, not a phase-two promise. If the operator cannot read the path, the human check the layer depends on does not exist in practice.
6. Graduate or kill at ninety days. Pre-register accuracy, latency, and ops cost per successful multi-hop answer so the pilot ends in a decision, not an indefinite “learning phase.” The pre-registration is a calendar entry with a named decision-maker. It prevents the pilot from becoming a permanent line item that nobody ever closes.
7. Regulatory mapping where consequential. EU AI Act and sector audit expectations are mapped when answers influence hiring, credit, claims, or contract terms that regulators may review. The mapping shows which obligations the knowledge layer helps discharge and which it does not. It converts a generic “governance” claim into a specific, checkable list of regulatory touchpoints.

Operating cadence after launch

Monthly steering should review:

• Collision queue depth and adjudication time. Track how many entity merges await human review and how long they sit, because a growing queue means identity drift is outpacing governance capacity. A queue that grows month on month is the leading indicator that the entity owner is under-resourced or the extraction pipeline is producing more collisions than the team can adjudicate. The number turns a silent quality risk into a visible staffing question.
• Index disagreement rate. Measure how often vector, keyword, and structured paths return conflicting candidates for the same question, and reconcile before operators see blended answers. Disagreement is the symptom that one of the three indexes has drifted, and unreconciled disagreement is the source of the “three different answers” complaint that erodes operator trust. The rate is the metric that tells the consistency owner whether the reconciliation job is keeping up.
• Extraction hallucination incidents. Count invented relations or entities the pipeline asserted with high confidence, since plausible graph edges that never existed are a common silent failure mode. A hallucinated relation is worse than a missing one, because it produces a confident wrong answer that follows a path that looks legitimate. The incident count is the metric that tells the team whether the extraction prompt or model needs retraining.
• Cost per successful multi-hop answer. Compare hybrid path cost to the vector baseline per correct answer, not per query, so finance sees unit economics rather than retrieval vanity metrics. Per-query cost hides the cost of wrong answers that need rework; per-correct-answer cost includes it. This is the number the CFO needs to decide whether to expand the layer or cap it.
• Operator override rate. Log how often humans correct or reject system paths, because rising overrides signal explainability or accuracy decay even when automated scores look stable. Overrides are the ground truth the automated scores cannot see, because they capture the cases where the operator’s judgement diverged from the system’s output. A rising rate is the early warning that the layer is degrading in a way the eval set does not cover.

Quarterly board view: one slide on metric movement, one on incidents, one on scope expansion requests; reject expansion without new failure evidence.

Closing position

Beyond vector search is not a fashion upgrade. It is an operating decision about identity, relations, and explainability.

Leadership teams should demand failure evidence before graph budgets, and refuse knowledge-layer programmes that skip evaluation and ownership.

References

1. NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile,” NIST AI 600-1, July 2024. https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.600-1.pdf
2. Deloitte AI Institute, “The State of AI in the Enterprise: The Untapped Edge” (2026 edition); survey of 3,235 leaders, Aug-Sep 2025. https://www.deloitte.com/us/en/what-we-do/capabilities/applied-artificial-intelligence/content/state-of-ai-in-the-enterprise.html
3. McKinsey & Company / QuantumBlack, “The State of AI: Global Survey 2025.” https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
4. Eurostat, “20% of EU enterprises use AI technologies,” 11 December 2025 (20.0% in 2025; 13.5% in 2024). https://ec.europa.eu/eurostat/en/web/products-eurostat-news/w/ddn-20251211-2
5. Gartner, “Gartner Says More Than 80% of Enterprises Will Have Used Generative AI APIs or Deployed Generative AI-Enabled Applications by 2026,” press release, 11 October 2023. https://www.gartner.com/en/newsroom/press-releases/2023-10-11-gartner-says-more-than-80-percent-of-enterprises-will-have-used-generative-ai-apis-or-deployed-generative-ai-enabled-applications-by-2026
6. Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027,” press release, 25 June 2025. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
7. Regulation (EU) 2024/1689 of the European Parliament and of the Council (EU AI Act). https://eur-lex.europa.eu/eli/reg/2024/1689/oj