Zeros Academy
← All courses

Evals Masterclass for Engineering and Product Success

Enroll

Anyone can get a model to produce something that reads well. Far fewer can say whether it is actually right, and that gap is where AI work quietly fails, in interviews and in production alike. This course builds the measurement layer: a golden set, an LLM-as-judge rubric you have checked against real labels, and an eval harness that tells you your agent got better rather than merely different. It is also the answer to tool churn, because when the framework everyone recommended last month is dead this month, knowing how to measure quality is the part that transfers. You leave able to defend a model choice with numbers rather than preference, and holding a harness that someone else can run.

The outcome

How does this prepare you for a job?

Evals are the clearest case of what interviews actually test. You are handed a problem and watched to see whether you can tell good output from bad and say why. These are the line items that make that legible on a résumé.

  • Built an evaluation harness combining a golden set with LLM-as-judge for an LLM feature, scoring answer accuracy, groundedness and hallucination rate across model versions.
  • Established eval gates in CI that block a deploy when answer quality regresses below a defined threshold.
  • Benchmarked Claude, GPT and Gemini on golden sets specific to the task, and documented the quality, token-cost and latency trade-offs behind the model selection.
  • Calibrated an LLM judge against labels from human experts, measuring agreement before the judge was trusted for release decisions.
  • Wrote adversarial evaluation sets covering prompt injection, jailbreaks and confidently-wrong outputs, each with documented mitigations.
  • Scored a retrieval-augmented generation pipeline end to end, separating retrieval precision from answer quality in order to localize the failures.
  • Instrumented agent runs with structured logging and tracing, to detect drift in output across long-running sessions.
  • Cut the cost of evaluation with sampling and tiered judging, using a cheap model first and a frontier model only where they disagree.
Tools

AI tools learnt

The tools here are the ones an evals job actually runs on: a harness, a judge, a tracer and a gate.

Hands-on skills

  • Turn a vague quality complaint, such as the answers feeling off, into a named metric with a threshold someone can argue about
  • Build a golden set by choosing the cases that represent the work, writing the reference answers, and covering the edge cases that actually break things
  • Write an LLM-as-judge rubric that discriminates rather than flatters, then check it against your own labels before you trust it
  • Measure groundedness and catch unsupported claims, which is the difference between a confident answer and a correct one
  • Score a RAG pipeline in two halves, retrieval and generation, so that you know which one is broken
  • Run a model bake-off on your own task, and read the curve of quality against cost and latency rather than a public leaderboard
  • Gate a merge on eval results, covering thresholds, regression sets, and what to do when a score drops two points
  • Red-team your own system with prompt injection, jailbreaks, and inputs engineered to produce plausible wrong answers
  • Detect drift in a long-running agent and localize the cause to the prompt, the context or the orchestration
  • Sample and tier your judging so that a full eval run costs cents rather than dollars
  • Present eval results to people who do not trust them, which is a calibration conversation with domain experts rather than handing over a dashboard
Syllabus

Modules

Level: Basic

What an eval is, and why it is the whole job

  • Why looks good to me stops working the moment someone other than you uses the system
  • The three questions every eval answers: is it right, is it grounded, and did it get worse
  • Establish a scored baseline on your own task before you change anything

Your first golden set

  • Choose thirty cases that represent the real work, and write the reference answers
  • Label the edge cases that break things: ambiguous inputs, missing context, and out-of-scope asks
  • Score the set by hand once, so that you know what the numbers actually mean

LLM-as-judge, done honestly

  • Write a rubric that discriminates between good and nearly-good, rather than one that rates everything an eight
  • Measure the judge against your own labels, and report the agreement before you trust it
  • Use structured outputs, so that scores are machine-readable, diffable and reviewable

Benchmark the models on your own task

  • Run Claude, GPT and Gemini against the same golden set rather than reading a leaderboard
  • Read quality against token cost and latency, and choose using numbers
  • Write the one-paragraph model-selection memo you can defend when someone challenges it

Level: Advanced

Evals as a merge gate

  • Turn the suite into pytest with explicit thresholds, and wire it into CI
  • Regression sets, where every bug found becomes a permanent test case
  • Decide what blocks a deploy and what only warns, which is the launch-review bar a big-company release has to clear

Truthfulness and adversarial evals

  • Score groundedness, so that every claim is either traced to a source or flagged as unsupported
  • Build red-team sets covering prompt injection, jailbreaks and confidently-wrong output
  • Measure refusal quality in both directions, because over-refusal is a failure mode too

Evaluate a full agent, not just a prompt

  • Score multi-step runs, asking whether it chose the right tool at the right step and recovered when one failed
  • Trajectory evals against final-answer evals, and when each of them lies to you
  • Build a complete eval around your own coding agent, end to end, as the capstone

Drift, observability, and production datasets

  • Trace real runs, and mine production traffic into new eval cases
  • Detect drift before users feel it, and localize it to the prompt, the context or the orchestration
  • Track score history across model upgrades, for the day a provider changes the model underneath you

The economics, and the argument you have to win

  • Tiered judging and sampling, so that a full run costs cents rather than dollars
  • Make the eval-backed case that a more expensive model pays for itself, since spending more is justified only when the stronger model scores better
  • Run the calibration conversation with domain experts who outrank you and distrust the numbers

Students can take the full course (both levels) or either level on its own.

Register

Waitlist

Join the course waitlist

No cohorts are scheduled yet. Join the waitlist and we’ll email you when registration opens.