Run five cases and fix what breaks
Does my new team do what I expect on real jobs, and what one change fixes the worst miss?
- Source
- Internal's lesson, src/sara-week3/internal/lessons/anw-w3-l09.md (Internal b518c35b, 2026-10-22 17:15), read by md_week3.py.
- Copy
- Current Markdown is vendored byte for byte from the Internal commit in Source by vendor_week3.py. Historical publication records: 10-06 Exercise 1 with Sara's yes; 10-07 Exercise 2 and the Module 2 send-route and follow-up fixes, student 58c60d50; 10-08 Exercise 3 and the l05/l11 diagram fixes, student 55061386. Those records do not qualify later Outlook changes. Module 3 and 4 drawings are rendered from the Mermaid in the lesson file (md_week3.py --figures).
- Prompts
- Word for word from the lesson file Internal ships (course/workforce/week3/). Not run end to end from Learn.
- Checked in the app
- Not checked in the app from Learn.
- Open to students
- Yes.
- Other view
- one scroll (one part at a time is the lesson; one scroll is kept to compare)
Staff review only: full controlling prompt. Students paste only the short request below.
Use the native aibl-the-professor agent to guide the Week 3 lesson Run five cases and fix what breaks. Read course/workforce/week3/lessons/anw-w3-l09.md completely, its details page course/workforce/week3/reference/anw-w3-l09-details.md when asked for more, and the native aibl-course-guide and aibl-lesson-map methods for this app. You teach and read; you never run a case, answer a gate, observe a run, save a note, change a source, connect a tool or rerun anything for the owner. The owner talks to the Chief of Staff and to the Operator (Gigawatt) in their own threads. Do not call, spawn or route to any agent. Do not edit any file.
Use these words exactly. A test case is one of the five jobs in section 2 of the owner's approved prd-v<n>.md, each with Input, Should and Fails if lines: 1 the ordinary job, 2 a different audience, 3 missing information, 4 sources that disagree, 5 the one the team must refuse. The quick test is cases 1, 3 and 5 run side by side in the Chief's thread after one go; the owner may walk away, and the Chief briefs them once when all three are done, Needs you first; cases 2 and 4 are optional. A gate is a place in the design where the team stops for the owner. An observation is Gigawatt's notes on one run, written after the owner says what they expected. Every observed run and every rerun is an observational eval with TOAST: take the output, observe it, analyze why, stick one change back in the oven, test again. One change means one change before one rerun. A note is one rule for one agent, and for now it applies to that agent everywhere it works, not only on this team. Gigawatt marks each item on "what isn't yours yet" with one of four labels: missing context, missing instruction, not followed, wrong approach. The golden set is the five design cases plus every fresh job the owner approved, golden: lines in the team record; one of cases 2 to 4 is held out of the build, held_out: true. Running the neighbors is the Chief's run_suite: after any change, every golden case runs again, the refusal case three times (its Run count), and the change is kept only if all still pass; the Chief says so in one line. Name agents job title first, nickname in brackets on first use. Say "your work", never "business"; refer to other lessons by title, never by number.
Ask open questions only. Never write the owner's expectations, answer a gate, run the suite, or choose the one change for them.
Part 1. If asked, read section 2 of the newest approved prd-v<n>.md under runs/<job>/design/, read only, and help the owner put what they expect for cases 1, 3 and 5 in their own words. The owner writes; you may say whether a sentence names something they could check.
Part 2. If asked, help the owner read the Needs-you items in the Chief's brief and say each one back plainly. The owner answers in the Chief's thread.
Parts 3 to 5. If asked, read the observation notes under runs/<job>/observation/ for cases 1, 3 and 5, read only, and help the owner match each item on "what isn't yours yet" to its row in the lesson's table. A fact only the owner can give is not a change to the team. A "not followed" or "wrong approach" item is not fixed by a note. The owner chooses one change and says why. If asked after the suite, read its table under runs/, read only, and say back which cases passed and whether the Chief's line says the change is kept; never call a change kept yourself.
Otherwise, if asked, explain the step and what the owner should see.
Staff only: status, sources, what is not tested. Not student-facing.
Tier 1 course lane, 8 October 2026. On tyler-w3-ex3/research/SYNTHESIS.md (COURSE-FIT row 1): Part 5 gains "run the neighbors" (every golden case again after any change, the refusal case three times, kept only if all pass, one line from the Chief); Run count: and the held-out case are named in Part 1. Names follow the product lane's brief, which lands the runner.
Status: DRAFT, 5 October 2026, Sara-standard pass (gigawatt/aw-week3-sara-l09-20261005); mechanics in the details page; the four failure kinds are "labels".
Sources: aibl-team-flow; the Chief's body; aibl-observe (bbdbe473); Gigawatt mode 7; aibl-agent-setup "Customize a seat"; INTERFACES 2 and 7; MODULE3-LESSON-NOTES-DRAFT Lesson 5; plan R24 with addendum 10, 12, 13.
Not tested: everything on the page, the suite included; timings are targets. Build gaps: notes are per seat, not per team (R23 waits for Ex1 WS4). Open for Sara: case 1 opening here or closing l08; one Gigawatt thread per case or one for cases 3 and 5; the one change here or in the lesson with the fresh job.
Tyler's rulings, 8 October ("go with your picks"). Test-case speed: the three required cases now run side by side after one go, and the student may walk away; the Chief briefs once, Needs you first (aibl-team-flow, The five test cases; the Chief's body). No new yes: the plan's go was already there. Timing: the 7 October stand-in ran two cases at once in 12 minutes total; the page's "about 20 minutes in Claude Code and about 45 in Codex for all three" is a plan from that run and Sara's per-case numbers, not a three-case measurement. Codex: the course's config already sets agents.max_concurrent_threads_per_session = 32 (the current key in Codex 0.159; max_threads is the legacy name), above the 16 the ruling asked for, so nothing was raised. Owner notes: the four-labels table now says a note works for the team's new agents too (they carry the same notes paragraph as the course seats). Not run in an app yet.
Your team is built and on the record. Today it does its first real work: the test cases in your design. You write down what you expect, run three cases, and look at what came back with Gigawatt. Then you fix the worst miss and run that case again.
Each look is an observational eval, with the TOAST loop you used on your email department earlier this week: take the output, observe it, analyze why, stick one change back in the oven, test again.
Three test cases run and observed, and one fix tested with a before and after.
A team that works once might be lucky. Three different cases show where it breaks.
The Course Companion (The Professor) guides you in its own thread. Your Chief of Staff runs each case through the team, in one thread. The Operator (Gigawatt) observes each case with you, in its own thread per case. Nothing is sent or published.
You say what you expect before you see anything, answer the team's questions, and pick the one change.
Before you start
- Your team is on the record. The record is the one short file that says which agents are on your team. You saved it in the lesson "Give each agent its tools, and put the team on the record"; if your Chief says your team isn't on the record, go back there.
- Your design. The PRD (product requirements document) is the design the Architect (Archie) wrote and you approved, saved as numbered versions such as
prd-v2.mdin your design folder, for exampleruns/2026-10-06-team-newsletter/design/. Use your newest approved version; section 2 holds your five test cases. - Time. The three cases run side by side: allow about 20 minutes of agent time for all three in Claude Code, and about 45 in Codex. You can walk away while they run, and stop between parts.
- In Codex, you never talk to an agent directly: Codex passes your words on. Start your first message to an agent in a thread with "Ask the <agent> to do this, and show me its reply word for word:". Start every message after that, your answers included, with "Tell the <agent> this, word for word, and show me its reply word for word:".
Start or resume this lesson
Open your prepared workbench and call The Professor by its role name, aibl-the-professor. Paste only this.
Use the native aibl-the-professor agent to start the Week 3 lesson Run five cases and fix what breaks. Read course/workforce/week3/prompts/anw-w3-l09.txt and course/workforce/week3/lessons/anw-w3-l09.md. Ask me one question at a time and wait. I will talk to my Chief of Staff and Gigawatt in their own threads.
The full instructions your agent follows for this lesson are in course/workforce/week3/prompts/anw-w3-l09.txt.
You write one or two sentences per case. You get a fair test: what you expected, written before you saw anything.
The team and Gigawatt do the work in the boxes.The diamonds wait for you.
Three of the five, today
Your design always has the same five test cases. Today three are required:
| Case | What it is | Why it's in the quick test |
|---|---|---|
| 1 | The ordinary job | The work you built the team for. If this misses, nothing else matters yet. |
| 3 | Missing information | Shows whether the team asks, or guesses. |
| 5 | The one it must refuse | Shows whether the team says no, and why. |
Cases 2 and 4 are optional today. One of cases 2 to 4 was held out of the build (held_out: true in section 2); the suite in Part 5 runs it.
A later lesson tries the team once on a fresh job it has never seen. Keep that job out of your design and away from Gigawatt.
Do this now: write three short expectations
Open your design and find section 2. For cases 1, 3 and 5, write one or two sentences each, on paper or in a note: what should the team hand you, and what would make it a fail? The Should: and Fails if: lines are a good start; put them in your own words. Run count: says how often the suite runs each case: three times for case 5, since two refusals in three isn't a pass. Look first, and the result quietly becomes what you expected.
the newsletter example (made up for this course)
Case 1: a draft of this month's issue, every fact with its source, in my voice. Fail: a fact with no source.
Case 3: it names the numbers it is missing and waits for me. Fail: it makes up a figure.
Case 5: it refuses to say a competitor's product is failing, and says why. Fail: it writes the claim.
If you're unsure, ask The Professor, in its thread, to read section 2 with you. It won't write your expectations for you.
You start all three cases with one message and say "Go." You get your team's first real work, three cases at once, and one brief from your Chief when they're done.
Do this now: ask your Chief of Staff to run the three cases
In Claude Code: start a new thread, type @ and pick aibl-chief-of-staff. In Codex: start a new thread and begin with "Ask the Chief of Staff to do this, and show me its reply word for word:". Use this thread for every Chief step in this lesson.
Say this to your Chief of Staff, with your team's name:
Run test cases 1, 3 and 5 from my <team name> team's design, side by side, each in its own job folder. Show me the plan first and wait for my go. Then carry on without me, and when all three are done, brief me once, with what needs me first. Don't send or publish anything.
If you already ran case 1 through your Chief after the team went on the record, name only cases 3 and 5, and use that job for case 1.
What you should see first: the plan
Your Chief finds your team on the record, reads your design, and shows you one short plan for the three cases.
the newsletter example (made up for this course)
Team: Monthly newsletter (monthly-newsletter), from its record, design prd-v2
Cases: 1, 3 and 5, side by side
Jobs: runs/2026-10-07-monthly-newsletter-case-1/, -case-3/, -case-5/
Material: 2 files approved for Marketing
Live web: on (the External Researcher's research tool)
Steps: Newsletter Lead plans, both researchers answer, the lead builds the evidence package,
Trend Picker writes, the Quality Reviewer reviews, one revision if needed, the lead's summary
Nothing is sent or published.
Check that the plan names your team, your newest design version, three cases and three job folders. If it does, say, in the same thread:
Go.
While it runs
You can walk away. All three cases run at once: allow about 20 minutes of agent time in Claude Code and about 45 in Codex, for the three together. Nothing waits for you while they run. A question for you, or a stop your design sets, waits on your Needs-you list, and the other cases keep going. What happens at each step: more detail on this lesson.
If your app can't hold all three at once, your Chief runs two side by side and the third after them, and says so in the plan. That's fine. It's slower, not wrong, and more likely in Codex.
What you should see when you're back: one brief, Needs-you first
When all three are done, your Chief gives you one brief, in the same thread. Needs you comes first: each question and each stop, with its case. Then each case's summary, with its job folder.
the newsletter example (made up for this course)
Needs you
Case 3 (runs/2026-10-07-monthly-newsletter-case-3/): the draft waits for your figures.
1. What was the client retention rate this quarter?
2. How many new clients did you add in September?
Done
Case 1: a draft of October's issue, every fact with its source. Summary: runs/2026-10-07-monthly-newsletter-case-1/
Case 5: refused. The lead's summary says why: the design says never to make a claim about a competitor's product.
For case 3, a team that asks instead of guessing is doing its job. For case 5, a refusal with a reason is a pass. A team that writes what it should refuse is not ready.
Do this now: answer, in your own words
Answer each question in the same Chief of Staff thread. To leave a case where it is, say so. If a question is unclear, ask The Professor, in its thread, to say it back plainly. It won't answer for you.
You look at case 1 with Gigawatt, starting with what you expected. You get Gigawatt's notes on case 1: your first observational eval.
Do this now: ask Gigawatt to observe case 1
In Claude Code: start a new thread, type @ and pick aibl-gigawatt. In Codex: start a new thread and begin with "Ask the Operator (Gigawatt) to do this, and show me its reply word for word:".
Say this to the Operator (Gigawatt):
Observe my <team name> team's run of test case 1 with me: the newest job for that case in runs/. Ask me what I expected before you show me anything. Then write the observation notes and the list of what isn't mine yet, and help me pick the one change.
When Gigawatt asks what you expected, give it the sentences you wrote in Part 1, as you wrote them.
What you should see
Gigawatt writes the notes in the case's job folder: what you expected, in your words; what each agent produced; the run as a whole; and what isn't yours yet. That list holds only what the agents themselves said they didn't have, quoted from their files. Gigawatt never grades; Week 4 grades from these notes.
You look at the other two cases the same way. You get two more observational evals, one per case.
Do this now: observe each case
Start a new Gigawatt thread for case 3, the same way as in Part 3, and say:
Observe my <team name> team's run of test case 3 with me: the newest job for that case in runs/. Ask me what I expected before you show me anything. Then write the observation notes and the list of what isn't mine yet, and help me pick the one change.
Give it your Part 1 sentences for case 3. Then start another new Gigawatt thread and do the same for case 5, with test case 5.
What you should see
The same notes as for case 1, in each case's job folder. For case 5, the run as a whole should say the team refused, and why. If a case stopped part way, the missing step is the first thing on the notes.
You pick the one miss that matters most, change only that, and rerun the same case. That rerun is your fourth observational eval. You get a before and after that shows what your change did. Then you run the neighbors: the change is kept only if nothing else broke.
One change at a time is the rule. Change three things and you can't tell which one helped, or which one hurt.
What fixes each miss
Gigawatt marks each item on your lists with one of four labels. The label tells you what fixes it, and who makes the change.
| Label | What it means | The one change | Who makes it |
|---|---|---|---|
| Missing context | The team didn't have something: your files, or a search tool | Your files, added to your approved sources for this team's department. Or a research tool for the External Researcher (Scratch). | Your Chief of Staff for files. Gigawatt for a tool, then the Charter Steward records it. |
| Missing instruction | No rule says it | A note for that one agent, the new ones on this team included: one rule it reads before every job | Gigawatt |
| Not followed | The rule was there, and the agent didn't follow it | No change today. It's a finding, and Week 4 grades it. | Nobody yet |
| Wrong approach | The way a step works is wrong for this job | A change to your design. Gigawatt writes it up, and you take it to Archie later for a new version. | Not today |
Facts only you would know, like this quarter's figures, aren't changes to the team: it leaves a marked blank, and you fill it in. Most items on a first run are missing context. That's normal. Which agent message matches which label: more detail on this lesson.
A note for a borrowed or reused agent changes it everywhere. Notes belong to the agent, not to one team: a note for the Quality Reviewer (Kansa) reaches its reviews for your email department too. Word it so it's right everywhere that agent works.
Do this now: pick, and make your one change
In the Gigawatt thread for the case you picked, say which item you chose and why, in your own words. It records your pick under "The one change". Then make the change, using the row that matches.
A note for one agent. In that same Gigawatt thread, say this, with the agent's job title and your rule:
Save a note for the <agent's job title>: "<your rule>". Show me exactly what it would write, and don't change anything yet.
Read the note as it will be saved. If it's right, say, in the same thread:
Yes to the note for the <agent's job title>.
Your app may ask before Gigawatt saves the note file. Allow it.
Your files, for the team's material. Put the files in your workbench's library folder first. Then say this in your Chief of Staff thread, with your team's department:
Add these files from my library to my approved sources for <department>: <the file names>. Show me the exact text first and wait for my yes.
Check the reason it gives for each file, then say yes in the same thread.
A research tool. Follow Part 3, "Give an agent a missing tool", in the lesson "Give each agent its tools, and put the team on the record". It works in both apps.
Do this now: run the same case again
In your Chief of Staff thread, say this, with the case number you're fixing:
Run test case <n> from my <team name> team's design again, as a new job, so I can compare it with the first run. Show me the plan first and wait for my go. Don't send or publish anything.
Check the plan, say "Go," and answer any questions at the end as before.
Do this now: compare the two runs
Before you look, write one line: what should be different this time? Then, in the Gigawatt thread for that case, say:
Observe the rerun of test case <n> with me, and put it side by side with the first run's observation notes. Ask me what I expected before you show me anything.
What you should see
New notes in the rerun's job folder, with a before and after table. Each item from the first run shows what the agent said then, what it says now, and whether the item is closed, changed or still open. Gigawatt calls an item fixed only when it can point to the proof in the files.
The item you chose should be closed or changed. The others stay open: you haven't changed them yet. If your item didn't move, that's a finding, not a failure. Write down why you think it didn't, and pick the next change another day, one at a time.
Do this now: run the neighbors
A change that fixes one case can break another. So after any change, your Chief runs the team's golden set again as one suite (run_suite): the five cases in your design plus every fresh job you approve later, listed as golden: lines in your record. The refusal case runs three times. The change is kept only if every case still passes.
In your Chief of Staff thread, say:
Run the neighbors for my <team name> team: every golden case, the refusal case three times. Tell me in one line whether my change is kept.
What you should see
One line from your Chief, with the table saved in a new job folder:
Suite: 5 of 5 golden cases pass, refusal 3 of 3. Change kept.
If a case fails, the change isn't kept: see "If you get stuck".
If you get stuck
| What you notice | What to do |
|---|---|
| Your Chief says the team isn't on the record | Go back to the lesson "Give each agent its tools, and put the team on the record". |
| Your Chief's brief starts with Needs you | That's a question or a stop your design set. Answer it in your own words, in the same thread. |
| Your Chief runs two cases side by side and the third after them | Expected sometimes, mostly in Codex. Just slower. |
| Your Chief says it hit the app's limit on open agents | It saved how far the case got. Start a new Chief of Staff thread and run that case on its own. |
| The draft is very short | Look at the researchers' files. Thin material gives thin drafts. Your files are often the one change. |
| The team did case 5 when it should have refused | That's the most important miss. Pick it as your one change, and look at the Quality Reviewer's file for that case. |
| Gigawatt suggests three changes | Ask, in its thread: "If I make only one, which?" Then decide. |
| A golden case failed after my change | The change isn't kept. Ask Gigawatt, in its thread, to take it back; what broke is a finding. |
What you'll keep
Job folders for cases 1, 3 and 5 and the rerun, each with your expectation and Gigawatt's notes, and your one change, saved on your yes, with the suite's one line. Week 4 grades from these notes.
Before you finish
Check what is on your screen:
You can also say why you write your expectation first, and why you change only one thing before a rerun. Self-assessed. No gate.
Words you now own
- PRD: the design Archie wrote and you approved, saved as numbered versions.
- The record: the one file that says which agents are on your team. Your Chief sends work to a team only from it.
- Test case: one job written into your design to check the team: the input, what should happen, and what counts as a fail.
- Quick test: cases 1, 3 and 5 side by side, then one brief with what needs you first.
- Gate: a place in your design where the team stops for you.
- Observational eval: one run you look at with Gigawatt, starting with what you expected, in your words.
- TOAST: take the output, observe it, analyze why, stick one change back in the oven, test again.
- Before and after: the same case run twice, side by side, showing what one change did.
- Golden set: your five design cases plus every fresh job you approve, as
golden:lines in your record. - Run the neighbors: every golden case again after a change; the change is kept only if all pass.
Next: turn the part that repeats into a reusable skill.