How AI assessment works

Last updated 2 November 2026

The pipeline

1. Claims. Every sentence that contains a numbered citation becomes a claim. A sentence citing several papers is checked against each paper separately.
2. References. Each DOI must resolve at Crossref or DataCite, and the cited title must match the registered title.
3. Sources. We fetch only legally open full texts (from arXiv, Unpaywall and OpenAlex records) and confirm the PDF carries the registered title. Authors cannot upload sources.
4. Retrieval. The passages of the source most relevant to the claim are selected by keyword search (BM25).
5. Verdict. A language model reads the claim and the passages and returns a verdict, a short rationale, and a supporting quote.
6. Guards. A verdict of "supported" is downgraded to "partially supported" unless its quote is found verbatim in the source, and unless every number in the claim appears in the retrieved text.

What the verdicts mean

Supported

The cited paper states what the claim says, and the quote shows it.

Partially supported

Part of the claim is supported, or the claim overstates, generalises or changes a detail, or a guard failed.

Contradicted

The cited paper says something incompatible with the claim.

Not supported

The retrieved text does not support the claim, though it does not contradict it either.

Could not be verified

The check could not be completed, for example because the text could not be read.

Known limits

• The assessment checks fidelity to the cited sources, not whether the review is complete, balanced or important.
• A claim citing two papers is checked against each paper alone, so a claim that needs both together may be marked partially supported.
• Retrieval is keyword-based, so evidence phrased very differently from the claim can be missed. Tables, figures and formulas are read as extracted text only.
• The model can be wrong in both directions. Its accuracy is being measured against human judgements; until those results are published, read each verdict with its quote.
• Only open-access sources are used. Reviews in fields with little open-access literature will often be ineligible.

Models and infrastructure

Verdicts are produced by an open-weight language model, identified with each article, running on a temporary GPU machine rented from vast.ai hosts outside China during a weekly batch. Each request contains one citation-bearing claim and selected passages from the cited source, which may contain personal information that appears in the manuscript or the source. These inputs are not used to train any model. The rented machine is destroyed after the run; the provider or host may keep their own security logs under their terms.