LangGraph deep research: verify citations instead of just linking

A link after a sentence does not prove that the source supports the sentence. It only proves that a URL is present. For Abundance, I translated the stronger requirement into code: every claim-evidence reference may point to admitted evidence only, every quoted passage used for verification must occur verbatim in that exact evidence record, and the semantic verifier may invent neither sources nor claim-evidence pairs.

The result is not an automatic truth machine. It is a controlled inspection stage between a research report and its quality metrics. This article covers the runtime architecture. A separate article on LLM evals with twelve reference cases explains how I measured the semantic component before promotion.

The three questions behind an auditable citation

An automatically generated research report needs to answer three separate questions:

The distinction matters. A real URL can be irrelevant. A correctly copied sentence can contradict a claim. And a record can be on topic while remaining insufficient for a causal or current claim.

LangGraph fixes the order of operations

The public Abundance workflow contains eight named stages:

1. scope_inquiry
2. create_plan
3. collect_evidence
4. review_evidence
5. assess_evidence
6. synthesize_report
7. verify_claims
8. publish_report

The graph decides which stage runs. The model writes a plan and a report, but it does not select external capabilities. Evidence collection uses preconfigured read-only adapters. The rationale is recorded in the public architecture decision for the deterministic LangGraph workflow.

The order is especially important for citation verification: a structured report is produced first, verified second and then published unchanged. The verification output becomes an evaluation metric. It is not fed back into synthesis.

What the code binds

A report contains structured claims and admitted evidence. Each claim can point to one or more records through evidence_ids. The application derives the only permitted pairs from that existing structure:

expected_pairs = {
    (claim.id, evidence_id)
    for claim in report.claims
    for evidence_id in claim.evidence_ids
    if evidence_id in evidence
}

The verifier may return results for those pairs only. The application discards a result when the pair was not expected, when it is duplicated or when its fingerprints do not match the current claim and evidence. Normalised claim text and evidence content are bound with SHA-256.

A quote proposed by the model is not trusted either. bind_exact_quote accepts it only when the string occurs verbatim in the admitted excerpt. The complete deterministic aggregation lives in application/claim_verification.py; provider output is also constrained against known IDs and exact quotes in adapters/models.py.

This code boundary catches four concrete failures:

• an invented evidence ID;
• a claim-evidence pair that never existed in the report;
• a fabricated or paraphrased “verbatim” quote;
• a result produced for an older version of the claim or evidence text.

Four verdicts instead of “citation present”

The semantic component classifies each permitted pair as supports, contradicts, insufficient or unverifiable.

• supports: The quoted passage supports the specific claim.
• contradicts: The quoted passage and the claim cannot both be true as stated.
• insufficient: The record is relevant but does not establish the claim – for example because independence, causality or current data are missing.
• unverifiable: The claim cannot be checked with the available material at all, such as a forecast beyond the source's time horizon.

When a claim cites multiple records, the code aggregates conservatively: contradiction takes precedence over support, and support takes precedence over insufficient evidence. A claim without any valid verification remains unverifiable. The evaluation also counts high-confidence claims that received no supporting verdict.

These rules are not a prompting convention. They live in deterministic aggregation and are covered by component tests, including invented pairs, invalid hashes, fabricated quotes and duplicate outputs.

Why verification runs in shadow mode

Verification sits between synthesis and publication, but it cannot rewrite the report. That is a deliberate safety boundary.

If a new verifier could immediately delete text, change confidence or reformulate claims, an error in that component would become a product error. Shadow mode does something narrower:

1. The report is finalised and checked against the deterministic report contract.
2. The verifier assesses the existing claim-evidence pairs only.
3. The application binds IDs, hashes and quotes again in code.
4. The quality summary is published next to the unchanged report.
5. If the provider is unavailable, the status becomes unavailable; the report is not silently treated as verified.

The graph implementation describes the stage explicitly as measuring semantic claim support “without rewriting the report”. Only after a component repeatedly clears an independent evaluation gate does it make sense to decide whether and how it may affect reports later.

A concrete example

Suppose a claim states: “The study demonstrated a causal treatment effect.” The admitted evidence says that after adjustment for baseline health and income, the association was no longer statistically distinguishable from zero, and that the observational design cannot establish causality.

A link-only check sees a claim with an evidence ID and passes it. The Abundance inspection stage requires more:

• the claim ID and evidence ID must form an existing pair in the report;
• the quote must be copied exactly from the evidence;
• the verdict must be contradicts;
• both fingerprints must match the inspected text versions.

This case is part of the publicly versioned claim-verification dataset. It does not test whether a system is broadly “scientifically intelligent”. It tests a narrow, product-relevant behavioural boundary.

What this architecture does not establish

The limits matter as much as the mechanism:

• A verbatim quote does not establish that its source is reputable or its statement true.
• Verification uses the admitted evidence text or stored excerpt, not automatically every passage in the original document.
• A supports verdict proves neither research completeness nor the absence of stronger counterevidence.
• Hashes bind text versions; they do not authenticate external origin.
• Shadow metrics do not improve a particular report. They make failures measurable first.

The architecture therefore solves a specific problem: it creates a machine-enforceable chain between “the report contains links” and “each inspected claim is bound to known evidence and a real passage from that evidence”.

Primary sources and version scope

This analysis targets public Abundance commit b4fc4c3, which added the evaluated claim verifier. The v1.0.0 tag from the same day records the preceding production-grade core and does not yet include this later verification component. That distinction keeps the tagged core and the subsequent quality stage auditable.

The linked Academic DeepSearch case study covers the complete product architecture, interface and remaining quality boundaries.

Read the case studyAcademic DeepSearch→