Codebook

This codebook documents the capture dataset behind the AI Shopping Divergence Study. Every column in the published CSV is defined here, alongside the scoring rubric used to turn raw answers into the conflict, fabrication-risk, severity, and price-accuracy findings quoted in the report.

A reader who has both this codebook and the dataset can look up the exact answer text behind any claim quoted in the published report.

Study Summary

For the full scoring methodology, how a raw answer becomes a confirmed conflict, a fabrication-risk flag, a severity tier, or a price verdict, see §Methodology of the published report.

File Structure

The published dataset (zipped CSV, 9.9 MB) contains:

• 8,794 rows, one row per answer (220 questions × 8 configurations × 5 samples, minus the 6 failed calls).
• 15 columns: the first wave's 14, plus tier. Internal capture fields (raw-payload paths, content hashes, leg identifiers, a battery-design tag) were stripped before release. No respondent-level or personal data to strip; there were no human subjects in this study.

Column Dictionary

Stripped columns: roadmap_tag (an internal battery-design tag) and the capture plumbing (raw_path, answer_sha256, arm_leg, provider_surface, model_reported, ts, battery_version) were removed before publication. None carried analytical weight in the report; no claim depends on them.

Honesty note: 1 Gemini (free) row has an empty answer_text — the API returned a grounded but empty completion. The row is kept, not dropped; an empty answer is itself a data point about answer-shape reliability.

Scoring Rubric (how a raw row becomes a finding)

The report's claims — confirmed conflicts, fabrication-risk flags, severity tiers — are not columns in this CSV. They come from a second pass: an LLM judge reads all 40 answers for a given query_id (8 configurations × 5 samples) together, with configuration identities anonymized and re-shuffled, and produces a structured verdict per question. In outline:

1. Extraction: for every sample, the judge extracts the primary claim, named products, prices, spec claims, and cited sources.
2. Cross-engine findings: where two or more configurations make checkable, contradicting claims about the same referent (e.g. "current top Sony pick: XM5" vs. "XM6"), the judge classifies it as a CHECKABLE_FACTUAL_CONFLICT and records whether it replicated across samples. A question counts as a confirmed conflict only when such a finding replicated — the same bar as the first wave.
3. Within-engine findings: where a single configuration's own 5 samples disagree with each other (a spec figure, a product list) with no new information between calls, the judge flags COMPLETENESS_DRIFT or self-contradiction.
4. Fabrication-risk flags: claims presented with unwarranted certainty that other configurations' answers (or an external source) directly contradict, tagged by risk level.
5. Practical-consequence flags: a plain-language note on what a shopper would actually experience if they acted on the flagged claim (e.g. buying a superseded product).
6. Severity tier: a single rollup (TIER_1 highest) for the question, driven by the most serious flag it produced.

A cross-vendor spot-check re-judged a sample of question groups with a non-Anthropic judge; on the conflict pairs both judges flagged, classifications agreed 81% of the time, and the outside judge flagged more conflicts overall than the primary one.

Price-Accuracy Layer (new this wave)

The price findings in the report are scored against ground truth, not against other engines, and are also not columns in this CSV. For the 24 price-battery products where one SKU and one seller resolve cleanly, the seller's own page was read by hand within 24 hours of collection, recording both the price a shopper pays and the list price it is discounted from, each with a URL and timestamp. A per-answer extractor pulls the price each answer claims; an answer scores correct when it matches either recorded number within a small rounding tolerance. Conditional prices (trade-in, bundle, member pricing) never count as the price. Seven of the 31 price questions had no single reference price — five name a product family with no single model (flagged for review before capture) and two named a product for which no single price could be resolved that day — and sit outside the price check by construction, recorded as Amendment 1i to the pre-registered protocol; the 960 checked answers are the remaining 24 questions × 8 configurations × 5 samples. Counted as unverifiable instead, they would raise the unverifiable share to 26% of 1,240 answers; the accuracy rate is unaffected either way because it is computed on verified answers only. At the answer level, 47 of the 960 price answers (4.9%) are unverifiable and sit outside the accuracy denominator — 41 named a different product than the one priced (36 of them on TV questions, spread evenly across all eight configurations, which points at our product selection rather than at any engine) and 6 could not be resolved at all. Every rate in the report is computed on the 913 verifiable answers, and that share is reported without any exclusions. The ground-truth table and the per-answer verdicts are available from Product.ai Research on request.

Every Tier-1 finding quoted in the published report was additionally checked against an external, citable source before print; the rubric flags a candidate, it isn't the final word on its own.

Questions about this dataset or the scoring code route to Product.ai Research (see report bylines). Reproductions and citations are welcome under CC BY 4.0; please link back to the published report.