claimaudit

A first-pass checker for the claims, numbers and citations in a folder of documents.

What it is

claimaudit reads the Markdown, LaTeX, text, BibTeX, notebook and Word files in a folder and lists sentences a person should look at: wording such as "we are the first" or "proven", numbers that do not match the data files beside them or disagree between documents, citations that do not resolve, and arithmetic in a sentence that does not add up. It does not decide whether anything is true, and it will miss most of what a careful reader would catch. It runs locally, uses only the Python standard library, and uploads nothing.

What it has been measured to do

These are small samples. They are reported as they came out, not rounded up.

Frozen corpus of eight known errors: it catches 3 of 8. It reaches 4 of 8 only if the author tags claims and supplies a registry of the values their own pipeline computed. The original replay cannot be re-run; a reconstruction of the eight errors is checked in with tests to catch drift, not as independent evidence.
Fifteen third-party ML papers, the free overclaim check: after tightening the priority and superiority patterns so they fire only on the construction, flags fell from 670 to 110. Of 60 sampled flags, 3 were correct, which is 5% precision. A sample of 60 cannot distinguish that from the 1.7% measured before that tightening. Most of what it flags is fine. Treat its output as a reading list, not a verdict.

Free and paid

Check	What it does	Tier
`overclaim`	Flags priority, universal-negative, proof and absolute-superlative wording for you to review.	Free
`source`	Looks for a matching value for numeric claims in CSV/JSON files in the same folder.	Paid
`citation`	Checks that arXiv IDs, DOIs and URLs resolve; compares `.bib` authors and titles with arXiv / Crossref.	Paid
`consistency`	Flags similar sentences in different files that state different numbers.	Paid
`registry`	Checks numbers you tagged in the text against a registry your own pipeline wrote.	Paid
`benchmark`	Checks that a stated gain matches the two numbers it is stated between.	Paid
`seeds`	Checks that a stated mean, spread and seed count can describe the same runs.	Paid

Buy a license key

Pay with Stripe Replace this with the Stripe Payment Link. One line.

After payment, Stripe sends you straight to a page with the install steps and the license key; there is no email step. For now that key is one shared early-access key, the same for every buyer, not a key issued to you.

Install

claimaudit is not on PyPI; pip install claimaudit will not find it. Install from a copy of the source folder (the page you land on after payment says where to get it). Python 3.9 or newer, no dependencies:

cd claimaudit
python3 -m pip install -e .
claimaudit check path/to/folder

If the claimaudit script is not on your PATH, this does the same thing:

python3 -m claimaudit check path/to/folder

To unlock the paid checks with the key from that page:

claimaudit activate YOUR-KEY
claimaudit status

In CI, set CLAIMAUDIT_LICENSE_KEY in the environment instead.

How the license works

The code is MIT-licensed. The key is an honor-system convenience, not a legal condition of use.

The key is a shared token shown to you after purchase, not a cryptographic license. claimaudit activate stores it on your machine and sends nothing anywhere. Any non-empty key unlocks the paid checks, and no server ever checks it. Anyone who reads the code can bypass it. If the tool is useful to you, please pay for it.

Limitations

Precision over recall. It stays quiet unless it has something to compare against, so it misses far more than it flags.
Honor-system license. No server validates the key.
No license server and no telemetry. By default the only network requests are the citation lookups (arXiv, Crossref, and URLs in your documents), and --offline turns those off. The experimental support check, which you have to ask for by name, also downloads the cited papers.
No language model by default. None of the checks uses one unless you add the experimental --llm option to the experimental support check, which sends a sentence and excerpts of the cited paper to the Anthropic API. One live-model run has been recorded (Claude Sonnet 5, 2026-09-26, 16 cases): the hard rule held and yield on true claims was 1 of 8 SUPPORTED; see the repository for the details. One run is not extensive validation. Nothing is sent unless you pass it.
It treats all prose as the author's voice. Quoted text, prompt exemplars, dataset rows and model-generated samples are audited as if the author asserted them, so they can be flagged. Use --exclude on those files.
PDFs are not read. Text inside a PDF pulled in with \includepdf is not read; the tool says so rather than reporting a clean result.
The citation check resolves identifiers. It does not check that the cited paper supports the sentence citing it. The experimental support check tries to, and can only catch a few kinds of mistake.