Product.ai / Join / Projects / Production-Trace Failure Taxonomy — review 100 traces, build the eval substrate from real failures
Project Open to Alpha Team

Production-Trace Failure Taxonomy — review 100 traces, build the eval substrate from real failures

Pick one Product.ai surface (chat, Alloy, or SimplyCodes' code-verification fleet). Review 100 anonymized production traces. Open-code the failure modes. Axial-code into a mechanism-distinct taxonomy with five-to-eight categories. Convert each load-bearing category into one or more task-level evals against a human-validated golden set. Wire the eval suite into the existing CI gate. Output is a working eval substrate producing real scores, derived from real failures, owned by the PM going forward.
Project Overview
Discipline
founding-product-manager · AI Systems — AI Engineer · product-manager-simplycodes
Duration
2 weeks
Compensation
Your stated freelance rate
Surface
Product.ai · SimplyCodes · Truth Graph
Kernels
productai · simplycodes · truth-graph
Outcomes
chat-expert · mlp-convert · dev-integrate
Tier
Consequential
Alpha Team
Open to alpha members who want to take this on
Tooling
Claude Code or Co-work

Why we want this done

The vocabulary of AI evaluation has industrialized faster than the practice — Hamel Husain and Shreya Shankar's Maven course alone has trained 4,500+ alumni. PMs who cite the vocabulary without the practice are a critical hiring failure mode at any AI-product company. The Anthropic April 23, 2026 Claude Code postmortem is the canonical demonstration: the firm with the most sophisticated eval infrastructure on the planet shipped a 50-day regression that internal evals did not detect. Three-overlapping-bugs slid past automated review, unit tests, end-to-end tests, dogfooding, and human review. User /feedback was the only signal that worked. The lesson is that the practice — actually reviewing 100+ production traces, doing open-coding then axial-coding, building task-level evals from real failure — is irreplaceable. PM candidates who have done this 50 times have intuition that vocabulary cannot reproduce. PM candidates who have not are theater regardless of resume. This project produces both real eval substrate Product.ai needs AND the highest-signal PM diagnostic in the library.

Scope

  1. Pick the surface (chat / Alloy / SimplyCodes code-verification) and argue for it
  2. Negotiate trace access on Day 1 with engineering ownership of that surface
  3. Review at least 100 anonymized production traces — open-code the failure modes (one annotation per trace minimum)
  4. Axial-code into a taxonomy: 5-8 mechanism-distinct failure categories with named mechanism per category
  5. Per-category, identify whether the failure is eval-able and design the eval (golden set with human-validated ground truth — at least 20 task cases total across the suite)
  6. Build the suite in a self-hostable eval framework (Phoenix, Langfuse self-hosted, or DeepEval)
  7. Wire it into the surface's CI gate as a deterministic check
  8. Write the taxonomy doc — one page per category with mechanism, illustrative examples, eval coverage, what's NOT eval-able

What success looks like

  • 100+ traces reviewed (the candidate shows the open-coding artifacts)
  • Taxonomy categories are mechanism-distinct (a stranger reading the doc cannot collapse two into one)
  • 20+ task cases live in the eval suite, all human-validated against the original failure traces
  • CI gate is operational — at least one PR during the trial window is blocked or flagged by the new evals
  • The candidate articulates which failures are NOT covered by the suite and why (the gap is named, not hidden)
  • LLM-as-Judge use is ≤30% of the suite, with named human-ground-truth validation per judge

References

references.md
PM Phase 3 briefing axiom D1 (Vocabulary Fluency Theater), D2 (Eval-Suite Bypass), D3 (Error Analysis on Production Traces), D4 (Trajectory-Metric Scoping), VERDICT 7 (Buy Eval Infrastructure, Build Domain-Specific Rubrics)
Hamel Husain enterprise eval methodology
Eugene Yan, Anthropic engineering blog on error-analysis-first methodology
Anthropic Claude Code April 23 2026 postmortem
AI Engineering Phase 3 briefing axioms A2, A3, D1, D5
Existing Product.ai chat / Alloy / SimplyCodes code-verification surfaces

Constraints

  • Claude Code or Co-work as primary substrate
  • Self-hostable eval substrate only (no Braintrust or LangSmith)
  • LLM-as-Judge limited to sub-frontier triage with explicit human-ground-truth validation
  • Privacy-respecting trace handling — PII scrubbing documented
  • IP separation: chat / Alloy / SimplyCodes are application-layer; methodology paths (aios-methods/_tools/arc-autopilot/) are out of scope
  • Code-merge rights to the surface's CI required — if blocked, surface immediately on Day 1
  • The candidate must personally do the trace review — outsourcing to junior staff or AI tools is "usually a big mistake" (PM briefing axiom D3) and is a fail in this project
Apply
01

Read the Codex (10 min)

The operating principles we work by. If they resonate, the rest of this will land. Open the Codex →

02

12-minute video screen

Hireflix, async. Questions are calibrated to this project specifically.

03

Chemistry call (30-60 min)

Direct call with the CEO. Strategic alignment and mutual fit. No problem-solving exercise.

04

Project begins within 2-3 weeks

1099 contractor agreement, NDA, paid at your stated rate. Day 1 in Santa Monica.

Alpha Team members can take this project without the screen-and-call sequence. Reach out via the Alpha Team channel.

Apply for this trial Browse other projects