Product.ai / Join / Projects / Cross-Surface Eval Taxonomy — design the measurement system from scratch on one Product.ai surface, prove substrate-builder phenotype
Project Open to Alpha Team

Cross-Surface Eval Taxonomy — design the measurement system from scratch on one Product.ai surface, prove substrate-builder phenotype

Pick one Product.ai surface (Alloy, Cortex memory, SimplyCodes code-verification, or Product.ai chat). DESIGN the eval taxonomy from scratch — what to measure, why, how to close the loop from production failure to eval case. Not "ran evals" — designed the taxonomy. Ship the v1 substrate against a paired metric: time-to-runnable-eval AND grader-pass-rate on N=10 unseen variants of a planted failure trace. Document the iteration on metric choice — including the moment the candidate's first metric was wrong and what they replaced it with.
Project Overview
Discipline
AI Systems — Data Scientist / ML Engineer · AI Systems — AI Engineer
Duration
2 weeks
Compensation
Your stated freelance rate
Surface
Product.ai · Truth Graph · SimplyCodes
Kernels
productai · truth-graph · simplycodes
Outcomes
chat-expert · truth-graph-depth · dev-integrate
Tier
Consequential
Alpha Team
Open to alpha members who want to take this on
Tooling
Claude Code or Co-work

Why we want this done

The Data Science Phase 3 briefing names substrate-builder vs substrate-rider as the load-bearing phenotype split. Substrate-builders have lived through specific moments of "my first metric was wrong because X" that cannot be improvised. Substrate-riders cite frameworks, run benchmarks, or use LangSmith. The hiring probe (axiom A4) is targeted; this project is the working version. The candidate designs the eval system from scratch, plants a known failure, prepares 10 variants, demonstrates the paired metric — sub-1-hour AND >7/10 — and produces a working eval substrate Product.ai needs. The deliverable is BOTH the substrate AND the substrate-builder phenotype demonstration.

Scope

  1. Pick the surface (Alloy / Cortex memory / SimplyCodes code-verification / Product.ai chat) — argue why
  2. Design the eval taxonomy from scratch — failure categories, mechanism per category, measurement primitive per mechanism
  3. Document the iteration: first metric → why it was wrong → second metric → mechanism update
  4. Plant a known failure trace. Prepare 10 unseen variants of the same failure mechanism
  5. Build the v1 grader — wall-clock time from "here is the trace" to "here is a runnable eval"
  6. Run the grader on the 10 variants — pass rate visible
  7. Wire the substrate into the surface's CI gate (or instrumentation pipeline)
  8. Write the architectural decision record — taxonomy categories, what's covered, what's NOT, what mechanism each category addresses

What success looks like

  • The taxonomy is mechanism-distinct (a stranger reading it cannot collapse two categories)
  • Time-to-runnable-eval on the planted failure: <1 hour
  • Grader pass rate on 10 unseen variants: ≥7/10
  • The candidate names at least one metric that was wrong and what replaced it (this is the substrate-builder signal)
  • The eval substrate is in production CI or instrumentation pipeline — not parked in a notebook
  • The ADR explicitly identifies what is NOT covered (gap-flagged, not hidden)

References

references.md
Data Science Phase 3 briefing axiom A3 (Eval-Design Fluency Primary-Among-Several), A4 (Substrate-Builder), A5 (Paired Failure-to-Eval Metric), A15 (Four-Layer Eval Stack)
Hamel Husain enterprise eval methodology (700+ engineer corpus)
Eugene Yan three-step methodology
Shreya Shankar criteria-drift methodology
Anthropic April 23 2026 Claude Code postmortem
Existing Product.ai surface code

Constraints

  • Claude Code or Co-work as primary substrate
  • Self-hostable eval substrate only (Phoenix, Langfuse self-hosted, DeepEval) — no Braintrust or LangSmith
  • LLM-as-Judge limited to sub-frontier triage with explicit human-ground-truth validation
  • The candidate MUST report the iteration — what their first metric was, why it was wrong, what replaced it. Skipping this is a fail
  • Privacy-respecting trace handling
  • IP separation: surfaces are application-layer; methodology paths are out of scope
  • The 10 variants must be UNSEEN by the candidate's grader — variant generation by the team or the operator, not by the candidate
Apply
01

Read the Codex (10 min)

The operating principles we work by. If they resonate, the rest of this will land. Open the Codex →

02

12-minute video screen

Hireflix, async. Questions are calibrated to this project specifically.

03

Chemistry call (30-60 min)

Direct call with the CEO. Strategic alignment and mutual fit. No problem-solving exercise.

04

Project begins within 2-3 weeks

1099 contractor agreement, NDA, paid at your stated rate. Day 1 in Santa Monica.

Alpha Team members can take this project without the screen-and-call sequence. Reach out via the Alpha Team channel.

Apply for this trial Browse other projects