Product.ai / Join / Projects / Production Eval Harness — error-analysis-first failure taxonomy on Product.ai chat or Alloy
Project Open to Alpha Team

Production Eval Harness — error-analysis-first failure taxonomy on Product.ai chat or Alloy

Build a production eval harness for one Product.ai surface (chat or Alloy). The candidate picks the surface, reviews 50-100 real production traces, builds a failure taxonomy via open-coding then axial-coding, ships 20-50 task evals derived from real failures, wires them into a CI gate, and instruments a continuous-production-eval sampling pipeline with privacy-respecting trace capture. Output is a working eval harness producing real scores on real traffic, not a deck.
Project Overview
Discipline
AI Systems — AI Engineer · ai-systems-engineer
Duration
2 weeks
Compensation
Your stated freelance rate
Surface
Product.ai · Truth Graph · Engineering
Kernels
productai · truth-graph · engineering
Outcomes
chat-expert · truth-graph-depth · dev-integrate
Tier
Consequential
Alpha Team
Open to alpha members who want to take this on
Tooling
Claude Code or Co-work

Why we want this done

Eval-driven development is theater until production observation wires the loop. Anthropic — the most eval-sophisticated organization on the planet, running AIRE at $325-485K and operating ablation studies, dogfooding, automated review — got caught flat-footed twice in 8 months by relying on closed-loop verification. Both incidents were detected by external community before internal eval apparatus. Pre-implementation evals are imagination-bounded; the closed-loop pattern (production observation → axial coding → eval → CI gate → recurrence prevention) compounds while open-loop saturates. Product.ai's chat and Alloy surfaces are running today without this loop. Whoever closes it on one surface materially raises chat-expert outcome traction and creates the substrate every other AI workstream depends on.

Scope

  1. Pick the surface (chat or Alloy) and argue why
  2. Pull 50-100 real production traces — the candidate negotiates access on Day 1 with Phil/AI Engineering
  3. Open-coding pass — annotate failure modes; resist the temptation to skip to a typology
  4. Axial coding — collapse into a failure taxonomy with mechanism-distinct categories
  5. Convert each load-bearing failure mode into a task-level eval (golden set with human-validated ground truth)
  6. Wire the eval suite into a CI gate against a real PR pipeline
  7. Set up privacy-respecting continuous-production-eval sampling with a stable baseline
  8. One-page handoff doc explaining the taxonomy, the eval coverage, the CI integration, and the on-call protocol

What success looks like

  • The eval suite catches at least one regression in a real PR during the trial window
  • The failure taxonomy is mechanism-distinct (a stranger reading it cannot collapse two categories into one)
  • CI gate is operational and not bypassable (deterministic, not advisory)
  • The continuous sampling pipeline is privacy-respecting (PII handling documented), with a baseline distribution captured and a drift detector running
  • The handoff doc reads like Hamel Husain's writing, not like AI marketing
  • Total LLM-as-Judge use is ≤30% of the eval suite, with explicit human-ground-truth validation on each judge

References

references.md
AI Engineering Phase 3 briefing axioms A2 (Generation-Verification Inversion), A3 (Eval-Driven Development Theater), D1-D5 (Verification & Quality)
Hamel Husain enterprise eval methodology (700+ engineer corpus)
Eugene Yan, "Don't Mock the Database" and eval methodology essays
Anthropic April 23, 2026 Claude Code postmortem; August 2025 postmortem
Product.ai chat infrastructure + Alloy agent loop code surfaces
axioms/Engineering/AI Engineering Verification Capability Pedagogy and Sourcing (2026-04-27).md
AIOS Phase mapping notes (Cortex Phase 2-3, AIOS Phase 1-2 per VERDICT 7)

Constraints

  • Claude Code or Co-work as primary substrate
  • Self-hostable eval tooling only (Phoenix, Langfuse, or DeepEval) — no Braintrust or other neutral-eval-vendor lock-in
  • IP separation: no aios-methods access required; this lives in the application layer
  • LLM-as-Judge limited to sub-frontier triage; Opus 4.7 cannot reliably judge Opus 4.7
  • Privacy-respecting trace capture — PII handling argued explicitly in the design doc
  • Code-merge rights to chat or Alloy required; if blocked, surface immediately on Day 1
Apply
01

Read the Codex (10 min)

The operating principles we work by. If they resonate, the rest of this will land. Open the Codex →

02

12-minute video screen

Hireflix, async. Questions are calibrated to this project specifically.

03

Chemistry call (30-60 min)

Direct call with the CEO. Strategic alignment and mutual fit. No problem-solving exercise.

04

Project begins within 2-3 weeks

1099 contractor agreement, NDA, paid at your stated rate. Day 1 in Santa Monica.

Alpha Team members can take this project without the screen-and-call sequence. Reach out via the Alpha Team channel.

Apply for this trial Browse other projects