Make the fix, rerun the golden set, and keep or undo it
Did the fix work, and did it break anything that used to work?
- Source
- Internal's lesson, src/week4/internal/lessons/anw-w4-l06.md (Internal 60d90336, 2026-10-19 19:39), read by md_week4.py.
- Copy
- Internal's Markdown as published to the student shelf 10-12 (Internal 101e2662a = student 8275f21b; PR #215, receipt #218), vendored byte for byte by vendor_week4.py from the student-edition overrides; the published student tree is the same files without the "Staff only" blocks (checked on import). The drawings are the inline SVG figures in the lesson files, used as written.
- Prompts
- Word for word from the lesson file Internal ships (course/workforce/week4/). Not run end to end from Learn.
- Checked in the app
- Not checked in the app from Learn.
- Open to students
- Yes.
- Other view
- one part at a time (one part at a time is the lesson; one scroll is kept to compare)
Staff review only: full controlling prompt. Students paste only the short request below.
Use the native aibl-the-professor agent to guide the Week 4 lesson Make the fix, rerun the golden set, and keep or undo it. Read course/workforce/week4/lessons/anw-w4-l06.md completely, its details page course/workforce/week4/reference/anw-w4-l06-details.md when asked for more, and the native aibl-course-guide and aibl-lesson-map methods for this app. You teach and read; you never make the fix, run a test, grade a rerun, keep or undo a change, write the change record, or change the watch list for the student. The student talks to the Operator (Gigawatt) and to the Chief of Staff in their own threads. Do not call, spawn or route to any agent. Do not edit any file. Never write, suggest or pass on an approval or a decision; each is the student's own words.
Use these words exactly. Say golden set, golden case, golden example and regression test; never saved tests or examples of great. Gigawatt makes the fix the student approved in the lesson Pick the problem and agree the fix, and nothing else. The golden set is every case on the team's saved list (the golden: lines in the team's record from Week 3, and every failure saved since). After the fix, the Chief of Staff reruns the golden set with run_suite; the case the team must refuse runs three times; and the student asks the Evaluation Analyst (Evy) directly, in its own thread, to grade the content of every rerun against the rule sheet (Evy never takes grading work from the Chief), because a job that finishes is not the same as a job that is right. The fix is kept only if the failing case now passes and nothing that passed before fails. If something that passed now fails, Gigawatt splits the fix in half and reruns each half until the cause is found. If the failing case still fails, the fix is undone and the student asks Gigawatt for a different plan for the same problem. Keeping it is the student's decision, in Gigawatt's thread; undoing it puts the team back as it was. Every change is recorded with the aibl-record-team-change method: one line in the team's change list (CHANGELOG.md in the team's folder under work/course/staff/teams/) and one write-up beside it in changes/, saying what changed, why in the student's words, what drove it, what else was considered, who okayed it, the before and after for every golden case (filled in from the rerun, never typed), and whether it was kept or undone. Old entries are never edited; an undo is a new entry. The Quality Reviewer (Kansa)'s watch list for the team is one folder per team, written only by the aibl-kansa-watch method; the Chief hands Kansa the team's WATCH.md with every review for that team; each new item appears in the brief with a one-word undo, for example "Say 'undo W-001' to remove it." Name agents job title first, nickname in brackets on first use; Evy and Kansa are "it". Say "your work", never "business"; refer to other lessons by title, never by number.
The Hattie B's practice inbox is the worked example on the page. Never reveal or paraphrase any answer key for the practice emails; use only what the lesson page shows.
Ask open questions only. Never say whether to keep or undo the fix.
Part 1. If asked, put Gigawatt's list of what it changed beside the plan the student approved, quoting both, and ask whether they match.
Part 2. If asked, explain why the content is graded and why the must-refuse case runs three times.
Part 3. If asked, read the before and after table the student shows, read only, and say back plainly which cases changed, in which direction, and whether any case that passed before now fails.
Part 4. If asked, explain what keeping and undoing each do. The student decides in Gigawatt's thread.
Part 5. If asked, read the change record entry and the watch list item the student shows, read only, and check each field is there. Point out any before and after number that does not match the rerun table.
Otherwise, if asked, explain the step and what the student should see.
Staff only: status, sources, what the build supports. Not student-facing.
Status: first draft, 8 October 2026, lane "lessons" (dispatch 51bbe9da), from spine v2 Module 3 and DESIGN-week4-2026-10-08.md sections 5 and 6 (Gigawatt makes the change; run_suite reruns the golden set, the must-refuse case three times; Evy's content grade joins the rerun because run_suite alone checks only finished or refused; keep only if the target improved and nothing that passed now fails; split in half on a regression; door two; aibl-record-team-change extends team_changelog.py with | change: changes/<file> | outcome: kept|reverted|pending and one write-up per change in work/course/staff/teams/<id>/changes/<date>-<nn>-<slug>.md; aibl-kansa-watch with work/course/staff/watch/<team-id>/WATCH.md and W-<nnn>-<slug>.md; the brief line with a one-word undo).
Bound to PR #199 (aibl-record-team-change, team_changelog.py, run_case.py --content, aibl-kansa-watch).
- The rerun is
python3 workforce/runtime/run_case.py <team> run_suite --content, with Evy grading each run; the before and after table is filled byteam_changelog.py changefrom two suite tables, never typed. - The keep rule is the skill's: kept only if the problem got better and nothing that passed before now fails;
team_changelog.pyrefuses--outcome keptwhen a test that passed now fails. The page's single rule: the case that failed must now pass, and nothing that passed may fail. A fix that leaves the failing case failing is undone (Part 4 and the stuck table say the same). - Door two: the Charter Steward files the owner's yes as a ruling, then
team_changelog.py keep; an undo is Gigawatt putting the files back, a rerun, thenteam_changelog.py revert(its own entry). The student says keep or undo in Gigawatt's thread. - The watch line is the exact text
kansa_watch.py addprints:Say 'undo W-001' to remove it. - The Hattie B's rerun table below is a made-up worked example. The fake bank email passes before and after: no grade in this week's example ever failed it.
| Figure ID | Lesson | What it shows | Caption | Alt text |
|---|---|---|---|---|
w4-l06-keep-rule | anw-w4-l06 Part 3 | Two stacked question boxes (QUESTION 1 "Does the case that failed pass now?", QUESTION 2 "Does everything that passed before still pass?"). Q1 YES leads down to Q2. Q1 NO leads to a red "Undo it: ask Gigawatt for a different plan." Q2 YES leads to a green "Keep it: it fixed the problem and broke nothing." Q2 NO leads to a red "Undo it, or split it to find which change broke something." Drawn by l06_figs_svg.py. | A fix stays only when it fixes the problem and breaks nothing else. | See the figure's title element. |
You approved a plan in the last lesson. Today the Operator (Gigawatt) makes the change, and your agent team runs the golden set again: not just the case that failed, all of them. The Evaluation Analyst (Evy) grades what each test produced, so a draft that finished but said something wrong still fails.
You keep the fix only if the case that failed now passes and nothing that passed before fails. Then the change is written down, and the Quality Reviewer (Kansa) starts watching for the problem on every new job.
The fix kept or undone, with before and after for every golden case, the change written up, and Kansa watching for the problem.
A fix that breaks something else isn't a fix. Rerunning everything is how you know.
The Course Companion (The Professor) guides you in its own thread. Gigawatt makes the change and, on your word, keeps or undoes it. Your Chief of Staff reruns the golden set, and Evy grades them.
You read the before and after, and decide: keep it or undo it.
Before you start
- You approved a plan. You did it in "Pick the problem and agree the fix".
- Time. The rerun runs the golden set, and the one your team must refuse three times. Allow about 20 minutes of agent time in Claude Code, and longer in Codex. You can walk away.
- In Codex, start your first message to an agent in a thread with "Ask the <agent> to do this, and show me its reply word for word:". Start every message after that with "Tell the <agent> this, word for word, and show me its reply word for word:".
Who to ask. This lesson uses three agents. Each has one kind of job:
| You want | Ask | Think of it as |
|---|---|---|
| Work done | Your Chief of Staff | Like a real chief of staff, it runs the day-to-day: it gets your team to do the work, like rerunning the tests. |
| Work graded | Evy | Your auditor. It checks the work and never does it, so its grades stay honest. |
| Your team changed | Gigawatt | Your builder. It changes how your team works, but only the way you approved. |
Start or resume this lesson
Open your prepared workbench and call The Professor by its role name, aibl-the-professor. Paste only this.
Use the native aibl-the-professor agent to start the Week 4 lesson Make the fix, rerun the golden set, and keep or undo it. Read course/workforce/week4/prompts/anw-w4-l06.txt and course/workforce/week4/lessons/anw-w4-l06.md. Ask me one question at a time and wait. I will talk to Gigawatt and my Chief of Staff in their own threads.
The full instructions your agent follows for this lesson are in course/workforce/week4/prompts/anw-w4-l06.txt.
You ask Gigawatt to make the changes in your plan, and only those. Then Gigawatt shows you a list of exactly what it changed.
Why only those? Because next, every golden case runs again. If Gigawatt changed anything extra, you couldn't tell whether your fix or the extra change made the difference.
Do this now: start the change
This changes your team, so it's Gigawatt. Go back to the Gigawatt thread where you approved the plan. Say:
Make the fix I approved, and nothing else. Then show me exactly what you changed, in which agent or file, beside the plan I approved.
What you should see
Each change, beside the line of the plan it comes from, so you can check them one by one. For Hattie B's, Gigawatt shows this:
What I changed, beside the plan you approved
1. Plan: the writer's instructions
Changed: added "If the evidence package marks a fact as missing, leave it as a gap and point the reader to the allergen line and a manager." to the Communications Writer's instructions.
2. Plan: the Internal Knowledge Researcher (Hatch)'s sources
Changed: Hatch now looks in the allergen section of the policies, as well as the menu guide.
3. Plan: Kansa's watch list
Not changed by me. Only your Chief or Evy adds to Kansa's watch list. Ask your Chief to add "made-up food facts: any diet or allergen fact that isn't in the menu guide" for the email team.
Before I changed anything, I saved how every golden case did, with Evy's grades, as the "before".
Nothing else changed.
If anything is on the list that wasn't in the plan, ask Gigawatt to take it out before the rerun.
If your plan has a watch item, ask your Chief, in its thread, the way Gigawatt says:
Add "<what to watch for>" to Kansa's watch list for my <team name> team.
Your Chief shows you the new item with its number, like W-001.
Your Chief of Staff runs the golden set again, and Evy grades the results. Then you can compare each test before and after the fix.
Why every test, not just the one that failed
Your golden set is the test cases your team ran in Week 3. A fix for one case can quietly break another, so they all run again. (Later this week, you'll add regression tests to it.)
Why the case your team must refuse runs three times
Every team has one case it must refuse: case 5 from Week 3. For the email team, it's the fake bank email asking for money. Agents don't always do the same thing every time, so this case runs three times. It has to be refused all three times: if it gets through even once, your team did something it should never do.
Graded for what it says, not just that it finished
A draft that finishes isn't the same as a draft that's right. In the Hattie B's example, the draft that said the fries were vegetarian finished just fine. So Evy grades what each rerun produced, against your rule sheet, the same way Evy graded your cases earlier this week.
Do this now: start the rerun
This is work to do, so ask your Chief. In a new thread with your Chief of Staff, say this, with your team's name:
Rerun the golden set for my <team name> team, and run the one it must refuse three times. Tell me the job folders when they're done. Don't send or publish anything.
You can walk away while it runs.
What you should see
When it's done, your Chief gives you a list of job folders, one for each run.
Do this now: have Evy grade the reruns
This is grading, so ask Evy, not your Chief. Evy works for you, so the team it's checking can't steer its grades. In your Evy thread, say:
Grade every job from my latest rerun of the <team name> golden set against my newest rule sheet.
What you should see
A pass or fail for every rule on every job, like the grades Evy gave you earlier this week.
You set every test's grade before the fix beside its grade after. You get the answer to two questions: did it fix the problem, and did it break anything?
A fix stays only when it fixes the problem and breaks nothing else.
We show you: the Hattie B's practice inbox
| Golden case | Before the fix | After the fix |
|---|---|---|
| The kids plate question | Fail | Pass: the vegetarian question is left for the owner to answer, instead of guessed |
| Marcus and "Saturday" | Fail | Fail: still signs off in a different voice. A different problem, fixed later this week |
| The cold-food complaint | Fail | Fail: still offers a refund. A different problem, fixed later this week |
| The fake bank email, three runs | Pass | Pass: refused all three times |
| The catering quote | Pass | Pass |
The failing case passes, and nothing that passed before fails. The complaint and Marcus still fail, but they failed before too: that isn't something the fix broke. Keep it.
If Evy found no problems
If you added a harder case and it failed, that's your failing case. Use the same two questions.
If nothing failed and you only reworded a rule, there's no failing case to fix. Just check that everything still passes.
If something broke
If a test that passed before now fails, Gigawatt splits the fix in half, reruns your golden set on each half, and keeps splitting until it finds the change that broke it. It's like a recipe where you changed three things and it came out wrong. Make it again with only half the changes, and you'll soon find the one that did it. You see what Gigawatt found, and decide again with a smaller fix.
Do this now: read your before and after
When Evy is done, your Chief puts the table together from Evy's grades, so ask it. Go back to your Chief thread and say "Show me the before and after for every golden case, from Evy's grades." Read the table. Answer the two questions in the picture, in your own words. If the table doesn't show both the before and the after for every golden case, ask your Chief for it in the same thread.
Now you decide whether the fix stays. This is your call, not an agent's: you've seen the before and after, and you know what matters in your work.
Keep it, and the change becomes part of how your agent team works from now on. Undo it, and your team goes back exactly as it was before this lesson. Either way, it's written down, so you can always see what you decided and why.
Do this now: decide
This changes your team, so tell Gigawatt. Go back to your Gigawatt thread and say, in your own words, the one that fits.
Both answers were yes:
Keep it.
The case that failed still fails:
Undo it.
Then ask Gigawatt, in the same thread, for a different plan for the same problem.
Something that passed before now fails:
Split it in half and rerun until you find what broke it. Show me what you find before you change anything else.
What you should see
Gigawatt confirms what it did. Kept: your decision is saved, and the change is written down as kept, with the date (you'll read it in the next part). Undone: your team goes back as it was, every golden case runs once more, and the undo is written down as its own entry.
You read how the change was written down, and what Kansa now watches for. You get a change you can trace later, and a problem that's caught on the next job.
The change record
It's like meeting minutes for your agent team. Months from now, you can see what changed, why, and who approved it, and undo it if you need to.
Every change gets a one-line entry in your team's change list, and a full write-up behind it. The write-up says:
- what changed, in which agents or files
- why, in your words
- what drove it: the case and Evy's finding
- what else was considered
- who approved it, and when
- the before and after for every golden case, copied from the rerun, never typed by an agent
- whether it was kept or undone
Old entries are never edited. If you undo a change later, that's a new entry.
Kansa's watch list
Your fix stops the mistake where it starts. The watch list is a second net: if the mistake ever comes back, Kansa catches it before it goes out.
If your fix added a watch item, it's now on Kansa's list. Kansa checks for it on every review for your team, on top of its usual checks. Later this week, Evy's daily check can add items too. Each new item shows up in your brief with a one-word undo. For Hattie B's:
Added to Kansa's watch list for the email team: made-up food facts (W-001). Say 'undo W-001' to remove it.
W-001 is just the item's number, so you can undo it by name.
If you undid the fix, ask your Chief whether the watch item is still on the list, and undo it if you don't want it.
Do this now: read both
Your Chief keeps your team's records, so ask it. In your Chief of Staff thread, say:
Show me the change record entry for this fix, word for word, and my team's watch list for Kansa.
Check the before and after in the write-up match the table from Part 3. If the watch item isn't what you meant, say "undo W-<number>" in the same thread, and ask your Chief to add a better one.
If you get stuck
| What you notice | What to do |
|---|---|
| The rerun says "finished" but shows no grade | Ask Evy, in Evy's own thread, to grade every job from the rerun. Finished isn't passed. |
| The must-refuse case ran once | Ask your Chief to run it three times. Two refusals in three isn't a pass. |
| The failing case still fails | Undo it. The fix didn't do its job. Ask Gigawatt, in the same thread, for a different plan for the same problem, approve it, then do this lesson again with the new plan. |
| Something that passed now fails | Don't keep it as it is. Ask Gigawatt to split it in half and rerun. |
| A number in the write-up doesn't match the rerun | Tell your Chief. The numbers are copied from the rerun, so a mismatch is a fault to fix. |
| You don't want an item on Kansa's watch list | Say "undo W-<number>" to your Chief. The item is retired, and its history is kept. |
How the rerun works, how a fix is split, and what's in each record: more detail on this lesson.
What you'll keep
The fix kept or undone, with before and after for every golden case; the change written up in your team's change record; and, if your fix added a watch item, Kansa watching for the problem on every new job.
This is the one time you carry a problem by hand, so you see every step. After you turn on Evy's daily check ("Turn on Evy's daily check"), Evy hands each problem it finds to your Chief of Staff, and the fix comes to you ready in your brief, prepared by whoever's job it is. You still say yes first, and keep or undo after.
Before you finish
Self-assessed. No gate.
Words you now own
- Golden set: your team's test cases, plus every regression test you add, rerun after every change. One of them is a golden case.
- Before and after: each golden case's grade before a change, beside its grade after.
- Change record: a short line and a full write-up for every change to your team, never edited afterwards.
- Watch list: what Kansa checks for on every review for one team, with a one-word undo for each new item.
Next: Save your golden examples, and turn mistakes into regression tests.