LLM evals with a small reference dataset: what 12 cases can do

Twelve cases are too few to quantify the general quality of an LLM. They can still protect a production-critical boundary reliably – if every case represents a concrete failure class, scoring checks more than the model's prose, and repeated trials face a gate defined before the results are known.

That is what Abundance uses a small, versioned dataset for in claim verification. The accepted artifact contains three trials: 11/12, 10/12 and 11/12 passing cases. Together that is 32 of 36 executions and an 88.9% mean pass rate. This number is not a universal quality score. It belongs to a pinned model profile, prompt version claim-verification-v3, dataset version 2026-08-25.1 and one narrowly defined component.

The companion article LangGraph deep research: verify citations instead of just linking covers the runtime architecture. This article is only about eval design and what its result can support.

What the twelve cases actually test

The public claim-verification dataset contains twelve golden cases across ten categories, including public policy, public health, technology, safety, source provenance, economics, research methods, forecasting, finance and legal policy.

Every case contains:

• a research inquiry;
• exactly one structured claim;
• exactly one admitted evidence record;
• the expected verdict;
• a passage that must occur in the verbatim quote;
• where needed, terms that must occur in the documented limitations.

The set covers all four possible verdicts: supports, contradicts, insufficient and unverifiable. It deliberately tests boundary decisions instead of twelve variants of straightforward support.

The fixture URLs deliberately use example domains. These are controlled synthetic component cases, not a sample of live web research.

A case passes only when every dimension is correct

Matching the verdict is not enough. The public claim_verification_eval_harness.py scores several boundaries for every case:

1. correct claim ID;
2. correct evidence ID;
3. correct verdict;
4. correct claim fingerprint;
5. correct evidence fingerprint;
6. a quote bound verbatim to the evidence;
7. the expected evidence passage inside that quote;
8. required limitations when the case defines them.

If one dimension fails, the entire case fails. That makes the metric strict but interpretable: a model can make the right semantic distinction and still fail because it did not provide the matching quote or omitted a required currency limitation.

The scoring structure also prevents a plausible result from passing through invented IDs or quotes. The eval harness uses the same deterministic hash and quote boundaries as the runtime application.

Why the old check scores 0 of 12

The public legacy baseline deliberately represents a weak heuristic: if a citation link exists, treat the claim as supported. It therefore assigns supports to every pair and returns no verbatim quote.

It passes 0/12 cases under the composite gate. That does not mean the old system was semantically wrong in all twelve cases. Even the direct-support cases fail because the required bound quote is missing. Contradiction, insufficient and unverifiable cases also receive the wrong verdict.

The accurate statement is therefore:

The legacy heuristic passes none of the twelve complete contract cases.

“0% claim-verification accuracy” would be misleading because the gate scores provenance, quote binding, semantics and limitations together.

Define the promotion gate before the run

The candidate is not promoted on its best trial. The harness runs repeated trials and applies predefined rules:

python -m abundance_research.claim_verification_eval_harness \
  --model deepseek-v4-flash \
  --runs 3 \
  --max-concurrency 3 \
  --output evals/results/claim-verification-candidate.json

Promotion requires:

• at least three trials;
• at least 80% mean pass rate;
• at least 70% in every individual trial;
• a measured improvement over the baseline;
• the same dataset version for candidate and baseline.

The alias deepseek-v4-flash is pinned to revision deepseek/deepseek-v4-flash-0731. The verifier version is stored in the artifact as well. This records the exact combination behind the measurement; a later prompt or alias change cannot silently inherit the same score.

What failed in the three trials

The accepted, source-controlled v3 baseline artifact reports every failure rather than just the mean:

The mean is 88.9%, and the weakest trial is 83.3%. Both clear the gate. The failure signatures are more informative than the promotion itself: the system did not fluctuate on IDs or fabricated quotes, but on several semantic boundaries and one missing limitation. Those observations identify the next regression cases to add.

What twelve cases can do

A small dataset is useful when the task is narrow and the cases are chosen deliberately. Here it can:

• reject a known weak heuristic consistently;
• cover all four output classes at least once;
• preserve safety-relevant boundaries around prompt injection, causality and stale sources;
• reveal schema, provenance and quote-binding regressions immediately;
• expose stochastic variation through repeated trials;
• measure a pull request or model change against a stable minimum contract.

This is closer to an executable component contract than a benchmark leaderboard. For that purpose, twelve curated cases can be more useful than hundreds of unreviewed examples because every failure has a known meaning.

What twelve cases cannot do

The result explicitly does not establish that:

• “The model verifies 89% of all scientific claims correctly.”
• “Quote binding proves a claim true.”
• “Performance transfers to other models, prompts or languages.”
• “The dataset represents the distribution of real user inquiries.”
• “Three trials already provide a narrow statistical confidence interval.”

The twelve fixtures are small, synthetic and deliberately curated. They test a component with one claim and one evidence record at a time. Real reports contain multiple claims, multiple records, ambiguous excerpts and errors inherited from earlier research stages.

Abundance therefore keeps evaluation layers separate. The twelve claim-verification cases are not twelve cases taken from the larger report set. There are also ten dedicated evidence-assessment fixtures and a separate dataset of 30 report-level reference cases. Each suite answers a different question.

How I would extend the dataset

The next useful step is not simply “more cases”, but broader evidence-backed failure coverage:

1. Preserve every real production failure as a minimised regression case.
2. Add multi-evidence claims containing both support and counterevidence.
3. Measure German and English pairs separately.
4. Include long excerpts, tables and numerical units.
5. Separate a frozen holdout set from development cases.
6. Audit a small, manually reviewed sample of complete live research reports.

Only the last step connects the component contract to the real usage distribution. Until then, the supported claim stays narrow: with the pinned configuration, the system passed 11, 10 and 11 of twelve versioned contract cases in three documented trials.

Primary sources and version scope

The dataset, harness and result artifact come from public Abundance commit b4fc4c3. The earlier v1.0.0 tag contains the production-grade core but not this subsequent evaluation gate. The linked Academic DeepSearch case study places the measurement in the context of the complete product.

Read the case studyAcademic DeepSearch→