Product.ai / Join / Projects / Per-PR Cost Ledger + Agent-Policy Metadata — Day 1 substrate for safe agent deployment across the Product.ai repos
Project Open to Alpha Team

Per-PR Cost Ledger + Agent-Policy Metadata — Day 1 substrate for safe agent deployment across the Product.ai repos

Build the two artifacts a backend platform should ship before opening agent deployment to the team: (1) per-module agent-policy metadata declaring agent-allowed / agent-allowed-with-mock / agent-forbidden across each Product.ai backend repo, and (2) a per-PR cost ledger as a GitHub Actions workflow that tags every agent-authored PR with token spend, model breakdown, and a running running-30-day cost rollup per repo.
Project Overview
Discipline
software-engineer-backend · ai-systems-engineer · ai-systems-associate
Duration
2 weeks
Compensation
Your stated freelance rate
Surface
Engineering · Cortex
Kernels
engineering · cortex
Outcomes
team-visible · team-velocity
Tier
Applied
Alpha Team
Open to alpha members who want to take this on
Tooling
Claude Code or Co-work

Why we want this done

Phase 2 Backend research inverted the Phase 1 sequencing: the first-order variables across every successful sub-frontier agent deployment (Stripe Minions, OpenAI Harness Team, Linear Huginn, Replit Agent, LangChain Deep Agents) are scope narrowing, per-task inference economics, and failure-archaeology speed — not substrate quality. Vendors selling substrate have economic incentive to credit the substrate; the empirical record credits scope. ETH Zurich proved auto-generated AGENTS.md files net-degrade agents 3% and inflate cost 20%. Substrate decay is worse than no substrate. The two artifacts named above are the minimum-viable Day 1 scaffolding. Product.ai's backend repos do not have either today. Whoever ships them sets the discipline that lets agent-authored PRs scale safely without burning runway on unbounded retry loops or shipping unauthorized writes to external services.

Scope

  1. Survey Product.ai's backend repos (productai-web, productai-mcp, simplycodes, simplycodes-web, productai-extension, alloy, aios)
  2. Per-module agent-policy classification — argue each module into one of three tiers:
  • agent-allowed — single-file, additive-only, ≥1 existing test on changed path, no production credentials, no external-service writes
  • agent-allowed-with-mock — needs mocked external services for safe iteration, write the mock harness
  • agent-forbidden — touches production credentials, payment paths, customer data, irreversible state
  1. Materialize the classification as a docs/agent-policy.md file in each repo plus a machine-readable .agent-policy.yaml for tooling
  2. Build the per-PR cost ledger — a GitHub Actions workflow that runs on every PR with the agent-authored label, captures token spend from the PR description or a structured comment, computes total $-cost, posts it back as a sticky comment, and rolls up to a 30-day per-repo running cost
  3. Wire a CI freshness validator that fails the build when AGENTS.md/CLAUDE.md references files that no longer exist
  4. Document the ratchet discipline — how the policy gets updated, who reviews changes, what triggers an upgrade or downgrade in tier

What success looks like

  • All seven Product.ai backend repos have docs/agent-policy.md and .agent-policy.yaml
  • Per-PR cost ledger is operational on at least three repos with sticky comments live
  • 30-day rollup is queryable (a JSON or simple dashboard surface)
  • One agent-authored PR is processed end-to-end during the trial — labeled, ledgered, and merged or blocked appropriately
  • The policy doc reads like an engineer who respects scope-narrowing, not "let agents have everything"
  • A second engineer can extend the policy to a new module without re-asking

References

references.md
Backend Engineering Phase 3 briefing axioms A2 (Scope Narrowing), A3 (Substrate Decay), A4 (Failure Archaeology), VERDICT 1 (Substrate Roadmap reversal)
ETH Zurich AGENTbench paper (Feb 2026) on AGENTS.md net-negative effect
Stripe Minions hard 2-CI-rounds-then-escalate rule (canonical bounded-retry)
Cortex CLAUDE.md §5.2 rule 11 (24-hour archaeology ritual)
Product.ai backend repos: productai-web, productai-mcp, simplycodes, simplycodes-web, productai-extension, alloy, aios
Existing GitHub Actions in those repos as reference patterns

Constraints

  • Claude Code as primary substrate
  • The policy classifier must be conservative — ambiguous modules go to agent-allowed-with-mock, not agent-allowed
  • The cost ledger must use model_router.py token telemetry where available; if absent, document the heuristic
  • Freshness validator must fail the build (deterministic), not warn (advisory)
  • IP separation: the policy applies to application-layer repos; methodology paths (aios-methods) are out of scope
  • No auto-generated AGENTS.md / CLAUDE.md content (banned per ETH Zurich finding); human-authored only
Apply
01

Read the Codex (10 min)

The operating principles we work by. If they resonate, the rest of this will land. Open the Codex →

02

12-minute video screen

Hireflix, async. Questions are calibrated to this project specifically.

03

Chemistry call (30-60 min)

Direct call with the CEO. Strategic alignment and mutual fit. No problem-solving exercise.

04

Project begins within 2-3 weeks

1099 contractor agreement, NDA, paid at your stated rate. Day 1 in Santa Monica.

Alpha Team members can take this project without the screen-and-call sequence. Reach out via the Alpha Team channel.

Apply for this trial Browse other projects