More detail: Run five cases and fix what breaks
Optional reading for this lesson. Everything you must do or decide is in the lesson.
This page goes with the Week 3 lesson Run five cases and fix what breaks (course/workforce/week3/lessons/anw-w3-l09.md). You don't need it to finish the lesson. Read it when you want the longer version.
How one case runs
The plan. Your Chief of Staff finds your team on the record, reads your newest approved design, and shows you a plan before anything starts. The plan lists your team and its record, the design version, the job folder, the material the team may use, whether live web search is on, the steps from your design, and that nothing is sent or published. Nothing runs until you say "Go."
The steps. Your Chief calls each agent in turn, in your Chief's thread, and waits for each one. Before the team's first job, it also has the Knowledge Librarian (Sallee) file the files your design approved, and names them in the plan. Your lead plans the job. Both researchers answer the lead's questions. The lead builds the evidence package. Your specialist writes. The Quality Reviewer (Kansa) reviews the finished piece against section 2 of your design, and your Chief decides whether one revision is needed. Each agent writes its own file in the job folder, so you can always see who did what.
The end. Your Chief gives you the lead's summary, with the one decision left for you, and offers the observational eval with Gigawatt. The job folder also holds a receipt, with a section called "What the agents said they did not have". Gigawatt starts its list from there.
Running three cases side by side
Cases 1, 3 and 5 each get their own job folder. After your go, your Chief carries each step for all three at once, in its thread, and waits for every agent before the next step. You don't need to watch. When an agent on any case needs you, or a case reaches a stop your design set, that case waits on your Needs-you list with its work saved, and the others keep going. When all three are done, your Chief briefs you once: what needs you first, then each case's summary.
Your Chief runs two cases side by side and the third after them, and says so in the plan, when your app can't hold all three at once. Codex limits how many agents can be open at once, so this happens more there; the course's Codex settings allow 32, which holds three cases.
If your Chief hits the app's limit on open agents part way, it saves how far the case got. Start a new Chief of Staff thread and run that case on its own.
What Gigawatt's notes hold
Gigawatt reads only that case's job folder, your team's record, and the Input:, Should: and Fails if: lines for that case in section 2 of your design. It writes the notes to observation/observation-v1.md in the case's job folder, in this order:
- The files it read.
- What you expected, in your words, word for word. The design's
Should:line goes beside it, marked as the design's, never in place of yours. - What each agent produced, one plain line each.
- The run as a whole. For case 5, a refusal with a reason counts as going as designed.
- What isn't yours yet, each item quoted from an agent's own file, with the file and one of the four labels.
- The one change, once you pick it.
Gigawatt never grades, never gives its own opinion of your draft, and never changes your design, your team's record or an agent. You make each change, on your yes, in its own step.
Which message gets which label
| What an agent wrote | Label | What it means |
|---|---|---|
| The Internal Knowledge Researcher (Hatch) lists a gap, or "Material: none approved" | Missing context | The team doesn't have your material for this. |
| The External Researcher (Scratch) names a search it couldn't run | Missing context | The team can't look outside for this yet. |
| An agent says it has no notes on file, or no agent knew a rule you have | Missing instruction | That agent has no rule of yours yet. |
| The rule is in the agent's notes, and its file shows it did something else | Not followed | A finding for Week 4. A second note won't fix it. |
| A step, a role or a handoff in your design is wrong for this work | Wrong approach | A change request for Archie: a new design version, with your new yes. |
| The evidence package lists facts it needs from you | None | Only you would know these. Fill in the marked blank. It isn't a change to the team. |
More on notes
A note is one rule for one agent. The agent reads its notes before every job, in every team it works on. So a note for a borrowed agent, such as the Quality Reviewer, also reaches its work for your email department. A note for a reused agent reaches the rest of its department. Word each note so it's right everywhere that agent works. Gigawatt shows you the exact note before it saves anything, and saves it only on your yes.
More on a research tool
You give the External Researcher a research tool by following Option 4 in the lesson "Make the team yours, then run the same email again". Gigawatt connects it, and the Charter Steward records it. In Codex, once a tool is connected, every agent in the thread can reach it. Your records still say only the External Researcher uses it, and that it can only search.
More on the before and after
The rerun is a new job with its own folder. Gigawatt starts new notes there and adds a before and after table. For each item from the first run, it quotes what the agent said then and what the same agent says now, each with its file, and marks the item closed, changed or still open. It puts the two drafts' paths side by side. It never repeats a send or an approval to make a comparison.
Why expectation first, and why one change
Expectation first. If you look at the result before you say what you wanted, the result quietly becomes what you wanted. Writing it down first keeps the eval honest.
One change. Two changes and one better result teach you nothing about which change worked. One change, one rerun, and you know.
Running the neighbors
A change that fixes one case can break another. So after any change to a team (a note, a file, a tool, later a skill or a new design version), your Chief runs the team's golden set again as one suite: the five cases in your design plus every fresh job you approve later, listed as golden: lines in your team's record. The refusal case runs three times (its Run count:), because a refusal that holds twice in three is not a pass. The held-out case is in the set: the build never saw it, so it is the fairest case of the five.
Your Chief runs the suite as run_suite, in a new job folder, and writes one table: each case, how many runs passed, and the receipt for each run. It reports one line, such as Suite: 5 of 5 golden cases pass, refusal 3 of 3. Change kept. A change is kept only if every golden case still passes. If one fails, the change is not kept: Gigawatt takes a note or a tool back on your yes, your Chief takes a file out of the team's approved sources on your yes, and what broke is written down as a finding. The suite sends and publishes nothing.
The fresh case
A later lesson in this module runs your team once on a job it has never seen. That job is a fair test only if it stays out of your design and out of every thread with Gigawatt until then. A fresh job that passes can join the golden set on your yes.