Agent-API Recommendation Attribution (Layer 2) — instrument the recommendation-decision moment

Why we want this done

The Phase 3 Content briefing identifies agent-API recommendation observability as the highest-leverage uncovered measurement surface in 2026. Existing AEO vendors (Profound, Peec, Siftly, Omnia, AirOps, Scrunch, Brandlight) and agent-observability vendors (Datadog LLM Observability, Honeycomb BubbleUp, Speakeasy, Braintrust) measure execution, not the recommendation-decision moment. Mintlify reports 45.3% of all doc requests come from AI agents (Claude Code 199.4M + Cursor 142.3M = 95.6% of agent traffic). The first-mover window terminates in first-party absorption by Anthropic and OpenAI within 18-24 months (Stripe-precedent: model providers will not leave the recommendation analytics surface to third parties indefinitely). Layer 2 is the structurally distinctive layer because it requires deployment surface that pure observability vendors don't have. Open-source Layers 1 and 3, proprietary Layer 2 — Stripe-precedent: run the test, publish the data, make Product.ai the standard. The candidate ships Layer 2 instrumentation on Product.ai's MCP / API surface and writes the methodology paper that becomes the category-convener artifact.

Scope

1. Layer 1 — developer-intent prompt battery (200-500 prompts, manually authored — not LLM-generated per Phase 3 axiom A2/E2). Each prompt is a realistic developer-asking-an-agent-for-help scenario relevant to Product.ai's commerce-knowledge surface
2. Automated execution harness against the major frontier agent surfaces (Claude Code, Cursor, ChatGPT, Gemini) with statistical floor (10 samples per prompt, 7-day rolling, ±5% threshold)
3. Layer 2 — API gateway agent attribution. Capture MCP session headers, user-agent strings, request-pattern fingerprints. Build the attribution pipeline that maps inbound API traffic to originating agent class
4. Layer 3 — code-generation trace. Parse the code agents generate when they invoke Product.ai's APIs; classify correctness, token efficiency, error patterns
5. Storage + dashboard — Datadog LLM Observability or Honeycomb (not custom time-series infrastructure per Phase 3 brief)
6. Drift instrumentation — track citation/recommendation drift week-over-week with statistical confidence intervals
7. Methodology paper — public-facing, Stripe-precedent, opens with the category gap (recommendation decision moment uncovered) and the empirical methodology Product.ai has pioneered

What success looks like

• All three layers produce real data on at least three frontier agent surfaces
• Statistical methodology applied — sample size, rolling window, signal-vs-noise threshold all defended
• Drift signature captured for at least one prompt cluster
• Methodology paper drafted (not necessarily published during trial) — concrete enough that an external reader can apply it
• Layer 1 and Layer 3 are open-source-ready (the methodology and harness can be released)
• Layer 2 is proprietary (Product.ai's deployment surface advantage)
• One real surprise emerges in the trial — a recommendation pattern the team did not previously see

References

Constraints

• Claude Code or Co-work as primary substrate
• Layer 1 prompts must be manually authored — LLM-generated context files perform net-negative (Phase 3 axiom A2/E2)
• Statistical methodology is non-negotiable — 10 samples per prompt floor, 7-day rolling, ±5% threshold
• Layer 2 deployment is proprietary; Layers 1 and 3 are open-source-ready
• Self-hostable observability tooling preferred; Datadog/Honeycomb acceptable but not Braintrust or LangSmith
• Privacy-respecting agent-attribution — user-agent strings and request fingerprints are inferential, not personally-identifying
• 3-week duration cap
• IP separation: API gateway and MCP servers are application-layer; methodology paths are out of scope