How we came up with this number

The two tracks, and how often each applies

Classification is a different job depending on whether U.S. Customs has already ruled on a product like yours. We measure the two separately and publish both. We never average them into one headline number, because which one applies to your shipment is not something an average can tell you.

How the benchmark was built

The benchmark holds 150 cases, each drawn from a public CBP ruling in the CROSS database. A case is a short product description plus the 10-digit code the ruling gave that product. The descriptions are written from the ruling's statement of facts; they never contain an HTS code or a ruling number, and a test in the repository fails if one appears.

Of the 150 cases, 143 are gradable — that is the n behind every figure on this page. The other 7 are excluded because no single code can fairly grade them; they are listed below by name, not quietly dropped.

129 of the gradable cases are graded at the full ten digits. The remaining 14 are graded at 8, 6 or 4 digits, because the ruling itself does not decide the digits below that — it is conditional ("if it meets the standard … otherwise …"), or the statistical suffix was created after the ruling and nothing in the ruling chooses between the new lines. Every figure on this page is the strict 10-digit figure over all 143 gradable cases, so those cases can and do score zero; grading each case at its own fair level would give a higher number, and that is not what we publish.

The 7 excluded cases, and why

• HTS-082 — The gold code is a glass code; the ruling classifies the bars under four different steel codes. No single code can grade the description.
• HTS-099 — The ruling assigns both an ordinary classification and a Chapter 98 treatment. One code cannot represent the right answer.
• HTS-101 — The ruling gives three outcomes depending on a benzene content the description does not fix; the gold is merely the first branch.
• HTS-102 — The gold code appears nowhere in the ruling, which classifies two of the three products separately and declines to classify the third.
• HTS-129 — The ruling covers seven items under six codes; the gold belongs to none of the described goods.
• HTS-136 — The gold comes from a ruling CBP revoked, and the two articles it covers take two different codes.
• HTS-141 — A bare trade name with no composition or function stated. Nothing can be added that would not simply restate the answer.

Excluding them raises the published figures slightly: on both tracks these cases were being scored against a code the ruling does not support, so almost all of them were counted wrong. We think grading a model against an answer we know is wrong tells you nothing, and we would rather say which seven we dropped.

What the tools actually ship

A score is only as good as the data behind it. This is that data: what is in it, whether each code is still a line of the schedule, and how far each row was checked.

Rulings CBP has revoked are dropped when the corpus is built; the ruling that did the revoking is kept instead. The corpus is fingerprinted and every index records the fingerprint it was built from, so a run you disagree with can be reproduced against the exact data it saw.

Counts here are read from the shipped data files (cross_manifest.json, cross_build_stats.json, _metadata.json) rather than typed into the page.

What the September 2026 audit changed

The first version of this benchmark was built from a third-party dataset whose question-and-answer pairs had been written from CBP rulings by a language model. Its label was usually the first tariff number in the ruling — which is often the code of a different item, a raw material, or a proposal CBP rejected. In September 2026 we read the underlying rulings on rulings.cbp.gov and checked every case against the text. The audit made no model calls; it was source reading.

• 9 answers were wrong and were corrected. Each of these 9 cases carried a code the ruling's own text contradicts: three came from rulings CBP later revoked or modified, two were the code of a different item in the same ruling, one was the importer's rejected proposal, and three appear nowhere in the ruling at all. A correction required a sentence of ruling text, never a classification opinion of our own. The original code is kept in the case file next to the corrected one.
• 7 cases were excluded as ungradable — the seven listed above.
• 58 descriptions were made answerable. These cases were missing a fact the ruling relied on — a composition percentage, a dimension, whether a fabric is woven, whether a fitting is threaded, a unit value — or described a list of goods while carrying one item's code. The missing fact was copied from the ruling, or the description was narrowed to the item the code belongs to. This makes a case fair, not easy: many of them are still hard, and where the ruling was silent we lowered the digit level a case is graded at rather than invent a fact.

The audit is written up in full in docs/BENCHMARK-AUDIT-2026-09.md, including its own limits: for 87 cases the ruling's summary record was checked rather than the full text, which could hide an error of the same kind; 8 rulings could not be retrieved at all; and 9 of the 150 cases are duplicates of another case drawn from the same ruling, which we have not yet removed.

How a case is scored

Conditions

Measured

21 September 2026 (2026-09-21)

Model

Claude Opus 5 (claude-opus-5)

Tools

the real trade-core MCP payload tools, over MCP, the same path a customer's workspace uses

Web search

off — no outside lookups of any kind

Thinking

the API default for the model (adaptive, high effort)

Determinism

PYTHONHASHSEED=0 on the tool server

Runs

one run per configuration

n

143 gradable cases of 150, per track

Reference data

2026 HTS Revision 19, 97,714 CBP ruling rows

The figures are measured against the same MCP tools a customer connects to, not against an internal pipeline. The model is given the product description and the tools, and nothing else.

What we got wrong before

1. 1. The test products' own rulings were retrievable.

   111 of the 150 benchmark products appeared word for word, with their correct code, in the ruling corpus the tools search — 126 rows in all. An exclusion list existed and named them, but the script that built the search index applied only its list of ruling ids, which held a single id. For every one of those 111 cases the top-ranked precedent the tools returned was the case itself.

2. 2. The 100 cases were not 100 cases.

   The sweep drew ten samples of ten with overlapping seeds: 76 distinct products, 25 of which had also been used while tuning the system.

3. 3. The number was not an accuracy rate.

   0.905 was a partial-credit mean that scored a correct 6-digit subheading as half a point. The plain exact-match rate was never printed.

4. 4. The customer's path was never measured.

   The figure came from an internal pipeline, not from a model driving the MCP tools the way a customer's workspace drives them.

What changed, so it cannot happen again

1. 1. Hold-out by ruling id and family.

   The novel-products track withholds each case's source ruling, every other row of that ruling, and any ruling that revoked, modified or is a same-day sibling of it — 173 ruling ids, 318 rows. Matching is on the ruling id, not on the wording of a description.

2. 2. Every retrieval surface, not just one.

   The hold-out covers the ruling search, the evidence cards, the leaf payloads, the alias table, the contrastive examples and the ruling detail text that payloads quote.

3. 3. A per-case proof test.

   A test runs once for each of the 150 cases and fails if any benchmark row can surface in any payload with the hold-out on. A matching positive-control test fails if the same checks stop detecting the leak when the hold-out is off — so the proof cannot quietly stop working.

4. 4. Two tracks, published separately.

   Whether a ruling was available is now the thing being measured, not an accident of how the corpus was built. Neither track is presented as the other, and the blend of the two is never the headline.

One consequence is worth stating plainly: the ruled-products figure on this page is measured with the hold-out off, because that is the configuration we ship — a customer's products are supposed to match our rulings. It is not a measure of reasoning from first principles, and it is not comparable to the old claim. The novel-products figure is what the system does when the answer is not in the data.

Known limitations and failure modes

How to reproduce this

The benchmark, the audit and the harness are in the repository. Each run writes a per-case record — the description, the tools called, the model's full answer, the code it chose and whether it matched — and we keep those records for every published figure, so any single case can be inspected rather than taken on trust.

Ruled products (the headline figure)

Novel products (hold-out on)

Runs pin PYTHONHASHSEED=0 on the tool server. The only difference between the two commands is --no-holdout versus --holdout: whether the benchmark's own rulings are withheld from every retrieval surface.

Where everything lives

• proofwork/eval/benchmarks/hts/cases_v2.json

  all 150 cases: input text, gold code, scoring level, audit verdict

• proofwork/eval/benchmarks/hts/audit_2026_09.json

  the per-case audit: what was checked, what changed, the ruling text behind it

• docs/BENCHMARK-AUDIT-2026-09.md

  the audit written up, with the method, its sources and its limits

• proofwork/eval/harness/

  the harness: the agent runner, the hold-out and the scorer