You grade the AI judges nobody else can check, building the golden-set exams that catch a miscalibrated LLM verdict before a paying customer sees it.

Product.ai is the verification layer for commerce, the intelligence that tells you what is actually true about a product, including when not to buy. SimplyCodes, its first proof at scale, is the code verification service, at ~$22M a year in revenue. The product is codes that actually work, proven by robots running real checkouts. Founder-owned and profitable since 2009. No outside investors. No board. Fewer than twenty operators outbuilding companies 10x our size.

Why This Role Exists

Everything we sell rests on one question. Is the claim true? Machine verdicts answer it, and they decide what we publish about a product, a price, a code. The constraint on growth is whether the machine verdicts stay right as the volume climbs. You own that, end to end: the eval science behind every automated verdict, from the judges and thresholds to the exams that grade them. We call that whole verdict chain the Truth Engine, and it decides what we publish as true for millions of shopping claims. You work directly with the founder.

You own the arbitration rulebooks, the rules that settle a contested verdict. You also own the structural check that keeps the truth this company ships from resting on any one person's judgment, including yours. That check is a held-out, human-graded exam the machine is measured against, plus a written authority split with the founder. You are not joining a team that does this; you are the person who does it. It is also the most durable engineering seat in the building. Every model generation absorbs another layer of surface skill, but grading a verdict no human can check faster than the machine is an open frontier that grows in value for years.

The System You'll Need to Model

  • Adversarial robustness, end to end. Merchants, bots, and models all have incentives to game a verdict. A merchant wants its expired code to pass as live, and a bot wants to look like a shopper. An LLM judge grades its own model family measurably too generously, a self-preference bias you measure and correct rather than assume away. Your verifier has to be structurally harder to fool than any of them.
  • Calibration with decay. A verdict is more than a yes or a no. It comes with a confidence and a shelf life. A price claim goes stale in days; a materials fact stays true for years. The pipeline has to score how sure the machine is, per claim class, and admit when it is not sure. That means calibrated confidence with principled abstention, because a confidently wrong verdict is worse than no verdict at all.
  • Golden sets: grading the grader. The hardest problem in the seat is meta-evaluation, measuring a judge whose judgments no human can make faster than the machine. The answer is a held-out, human-graded exam the model cannot see, cannot train on, and cannot game. Label quality, inter-rater agreement, contamination control, refresh cadence. Designing that exam is the science.
  • The verification ladder. Every claim climbs one, from unverified to machine-settled to human-graded, and each rung is cheaper than the one above and harder to fool than the one below. What survives feeds Axiomatic Intelligence, our stress-tested knowledge layer, where a proven verdict gets reused instead of re-litigated.
  • Evals as the ship gate. Eval-driven development is how engineering works here. Agent loops write the code and the content; your corpora and gates decide whether a loop's output ships or gets rejected. The eval is the spec.
  • Scale, under model churn. Hundreds of thousands of merchants, millions of claims. An eval method that works on a hundred cases and falls over at a million is a demo. And the ground moves under it. Each model generation resets what judges can do and what your exams still discriminate, so the harness gets rebuilt while it runs.

If reading that energizes you, keep going. If it feels overwhelming or underspecified, this isn't the right fit.

What You Will Own

  • The judge pipeline. The machine verdicts that decide what we claim is true about shopping: the models, thresholds, and rulebooks behind every automated verdict. Your number is the share of verdicts machines settle, taking it from a minority to the large majority while holding a high accuracy floor against the human-graded exam.
  • Calibration and abstention. How sure the machine is, per claim class, each with its own rate of decay. A judge that reports 95% confidence and is right 70% of the time is miscalibrated, and that bug is yours.
  • The golden sets. You grow them and keep them clean: label quality, contamination control, and refreshing before they saturate. When a verdict is disputed, the exam decides, not the loudest engineer in the room.
  • The accuracy floor other people now pay for. Developers and AI agents buy our verified commerce data on metered keys now that anonymous access is closed. A miscalibrated judge is a defect a paying customer sees; you own the floor that makes the data sellable.
  • Your seat charter. Within your first quarter you co-sign a charter for this seat: one machine-checkable number that proves the seat is working, and a written split of what you decide freely versus what you bring to the founder.

You will use the craft you already own (judge design, golden sets, calibration, regression corpora, statistical rigor) and grow into eval-gated agent orchestration, adversarial verification at commerce scale, and adjudication design for judgments no single grader can settle.

Who You Are

You interrogate every green checkmark. A passing eval is a claim, so you ask what the test could not have caught before you ask what it confirmed. You form working models of complex systems on your own, and when the model is wrong, you notice and update fast.

You treat agents as leverage you verify rather than an oracle you trust. Because you can hand-grade a judge, hand-build a corpus, and hand-verify a claim set yourself, you know when to trust or reject the verdict an agent hands back.

You have built an eval harness another team depended on, a regression corpus that caught a real failure before users did, an LLM-judge pipeline where you measured the judge's bias instead of trusting it, or a golden set for a system whose output is never the same twice. Adjacent roads count: scoring systems from fraud, search relevance, or content integrity where ground truth was scarce and the adversary was real. We care about the artifact and the reasoning far more than where you did it.

Who this isn't for. Wrong if you optimize for leaderboard scores or trust vendor-reported evals, or if your instinct when output looks weak is to massage the prompt rather than redesign the verification. Wrong if you'd rather publish a finding than ship a gate, want tightly-scoped tickets, or are comfortable letting an agent grade its own work. You'll be happiest here if you think in corpora and want your verification to be the reason an entire verification layer can be trusted.

How We Hire

We don't run traditional AI engineering interviews. Every stage is demonstrated performance on work-relevant tasks.

  1. Async video screen. About 15 minutes, on your own time, replacing the recruiter screen. We want to see how you think, not how you present.
  2. Calls with company stakeholders. Short conversations with the team.
  3. Conversation with the founder. How you model the system above, where you push back, and whether you can hold the argument live.
  4. Paid work trial (or Fast-Track Portfolio Review). Real verification work in our environment, with live loops, live corpora, and real verdicts. Fast-Track option: For exceptional builders with strong, verifiable shipped evaluation harnesses, published benchmarks, or production repositories, we can fast-track through a live technical architecture and code walkthrough of your shipped work in place of a multi-day trial.

If the work above sounds like yours but your resume is unconventional, apply anyway. We hire on the work and the reasoning, not the pedigree.

Compensation & Ownership

Total first-year compensation: $500,000 – $600,000 (base salary + target first-year ownership and profit-share).

  • Base salary: $350,000 – $420,000, top of market for principal-level machine learning engineering.
  • Target first-year ownership & profit-share: $150,000 – $180,000.

Product.ai is a profitable, founder-owned company generating ~$22M/yr in revenue with zero venture debt. Beyond base salary, senior operators participate in our Profit Interest Unit (PIU) ownership and profit-share program ($0 strike price units sharing in annual free-cash-flow distributions and tender liquidity, earned on performance milestones after joining, never day-one vesting). 100% family medical, dental, and vision coverage, plus an uncapped AI token and compute budget steered by ROI.

Based in Santa Monica, Los Angeles, on our high-trust hybrid rhythm: four focused in-office half-days (Mon/Tue/Thu/Fri 9am–2pm), with Wednesdays and afternoons open to work from wherever you do your best work. Relocation support available.