Study · eval-harness-101 · Lesson

Day 4 — Break & catch

30–45 min · seatbelt drills

Time: 30–45 min

Goal

Feel the harness as a seatbelt: change behavior → suite fails → you decide keep/revert.

Drills (pick ≥2)

  1. Baseline bug: cancel even when status is shipped → policy goldens must fail
  2. Missing refuse: unknown order invents a cancellation → refuse golden fails
  3. Double cancel: second cancel increments a counter / errors loudly → idempotent golden fails
  4. Tool skip: answer “done!” without calling cancel_order → tool-expect fails

For each: run suite before/after; write one line in your study log.

Reflection prompts

  • Which failure mode would users hit first in a real support agent?
  • Would a typical jam “verify green” have caught this? (usually no — different lane)

Log: did: Day 4 broke X caught by G0xx · takeaway = favorite catch.