Codebook
This codebook documents the capture dataset behind the AI Shopping Divergence Study. Every column in the published CSV is defined here, alongside the scoring rubric used to turn raw answers into the conflict, fabrication-risk, and severity findings quoted in the report.
A reader who has both this codebook and the dataset can look up the exact answer text behind any claim quoted in the published report.
Study Summary
| Field | Value |
|---|---|
| Capture date | 2026-07-30 |
| Question battery | 200 pre-registered shopping questions (v1-20260729-prereg) |
| Engines | ChatGPT (gpt-4o-2024-08-06), Claude (claude-sonnet-4-6), Gemini (gemini-2.5-flash), Perplexity (sonar) |
| Samples per question per engine | 5 |
| Total answers (analysis basis) | 4,000 |
| Error rows | 0 |
| License | CC BY 4.0 |
For the full scoring methodology, how a raw answer becomes a confirmed conflict, a fabrication-risk flag, or a severity tier, see §Methodology of the published report.
File Structure
The published dataset.csv contains:
- 4,000 rows, one row per answer (200 questions × 4 engines × 5 samples).
- 14 columns, the full capture record, published as-is except one internal-only tag stripped before release (see below). No respondent-level or personal data to strip; there were no human subjects in this study.
Column Dictionary
| Column | Type | Notes |
|---|---|---|
query_id | String | Stable identifier for the question (d001–d200), matches the locked question battery. |
question | String | The exact question text sent to every engine. |
category | String | Product category (Headphones, Smartphones, Supplements, Robot Vacuums, TVs & Monitors, etc.). |
question_type | String | One of Recommendation, Comparison, Price, or Spec/Fact, the axis behind the confirmed-conflict-rate-by-question-type finding. |
rivalry | String | For comparison-type questions, the named product pairing (e.g. "Sony WH-1000XM6 vs. Bose QuietComfort Ultra"). |
paired | String | The P1–P9 tag linking a question to its framing-reversal counterpart (used in the framing-reversal-check analysis). Blank where a question has no pair. |
engine | String | chatgpt, claude, gemini, or perplexity. |
model | String | The exact model the engine's API echoed back for that call. |
sample | Integer (0–4) | The resample index, normalized to 0-based across all four engines (Gemini's own capture format is 1-based internally; it's re-indexed to 0-4 here for consistency with the other three). |
run_id | String | Run/session identifier for the capture batch. |
search_invoked | Boolean | Whether the engine actually invoked live web search for that answer. Perplexity is search-native and always carries citations instead of this flag being meaningful. |
n_citations | Integer | Count of sources the engine itself cited. |
citations | String | The engine's own cited source URLs, pipe-delimited (|) where more than one. 10,917 of these URLs are vertexaisearch.cloud.google.com/grounding-api-redirect/... links, which is Google's grounding API wrapping the real source rather than linking it directly. That's more than four times the next-most-common domain, so computing cited-domain share directly off this column will surface Google as the top source; that's an artifact of the redirect, not a finding about sourcing. The wrapped links resolve to third-party sources when followed. |
answer_text | String | The full verbatim answer. This is the field to read when confirming a specific claim in the report. |
Stripped column: roadmap_tag, an internal battery-design tag, was removed before publication. It carried no analytical weight in the report and no claim here depends on it.
Honesty note: 7 Gemini rows have an empty answer_text, the API returned a grounded but empty completion. These rows are kept, not dropped; an empty answer is itself a data point about answer-shape reliability.
Scoring Rubric (how a raw row becomes a finding)
The report's claims, confirmed conflicts, fabrication-risk flags, severity tiers, are not columns in this CSV. They come from a second pass: an LLM judge reads all 20 answers for a given query_id (4 engines × 5 samples) together and produces a structured verdict per question. In outline:
- Extraction: for every sample, the judge extracts the primary claim, named products, prices, spec claims, and cited sources.
- Cross-engine findings: where two or more engines make checkable, contradicting claims about the same referent (e.g. "current top Sony pick: XM5" vs. "XM6"), the judge classifies it as a
CHECKABLE_FACTUAL_CONFLICTand records whether it replicated across samples. - Within-engine findings: where a single engine's own 5 samples disagree with each other (a spec figure, a product list) with no new information between calls, the judge flags
COMPLETENESS_DRIFTor self-contradiction. - Fabrication-risk flags: claims presented with unwarranted certainty that other engines' answers (or an external source) directly contradict, tagged by risk level.
- Practical-consequence flags: a plain-language note on what a shopper would actually experience if they acted on the flagged claim (e.g. buying a superseded product).
- Severity tier: a single rollup (
TIER_1highest) for the question, driven by the most serious flag it produced.
Every Tier-1 finding quoted in the published report was additionally checked against an external, citable source before print; the rubric flags a candidate, it isn't the final word on its own.
Questions about this dataset or the scoring code route to Product.ai Research (see report bylines). Reproductions and citations are welcome under CC BY 4.0; please link back to the published report.