An evals engine for people who ship AI features
A small, opinionated harness for testing LLM features against real user tasks instead of leaderboard benchmarks.
The problem
Every team I talk to ships an LLM feature, looks at five outputs, says looks good, and moves on. Two weeks later a prompt tweak quietly breaks something, and nobody notices until a customer does.
Why I'm building it
Off-the-shelf eval tools are either research frameworks or enterprise dashboards. I wanted something in between: a CLI and a tiny web view that a product manager can actually read.
Approach
Tasks are written as plain YAML, close to how a PM would describe a user job.
Every run is a snapshot, so you can diff two prompts or two models side by side.
Graders can be rules, code, or another model, and every grade keeps its reasoning.
Product decisions
The hardest call was saying no to a hosted version. Keeping it local means no accounts, no data leaving the laptop, and a much smaller thing to maintain.
Technical architecture
tasks/*.yaml ──► runner (TypeScript) ──► providers (OpenAI, Anthropic, local)
│
├─► graders (rules · code · model)
└─► runs/<timestamp>.jsonl ──► web view (Next.js)What I've learned so far
Writing good tasks is 80% of the work. The grader matters much less than having twenty honest examples of what good looks like.