Day 5 — A/B on behavior

Time: 30–45 min

Rule

Same goldens, two configs, compare scorecards. Tokens/time optional footnotes — primary metric is pass rate (and critical-tag pass rate).

Lab A/B

python3 run_suite.py --agent baseline --out scorecard_a.md
python3 run_suite.py --agent strict   --out scorecard_b.md
python3 compare.py scorecard_a.md scorecard_b.md
# or just diff the JSON summaries if present

Agents in lab:

• baseline — correct-ish policy
• strict — extra-cautious (may over-refuse happy paths)
• sloppy — under-refuses / skips tools

Your write-up (5–8 lines in study log or notes)

1. Pass rate A vs B
2. Which tags flipped (happy / safety / refuse)
3. Which you’d ship and why
4. One risk the winner still has

Interview line to practice out loud

“I freeze scenarios as goldens, score tool use and side effects, and any prompt or model change has to beat or match the suite—or I document an accepted regression.”

Log: did: Day 5 A/B baseline vs strict · takeaway = ship choice + reason.