More detail: Turn what you found into the rules for this team
Optional reading for this lesson. Everything you must do or decide is in the lesson.
This page goes with the Week 4 lesson Turn what you found into the rules for this team (course/workforce/week4/lessons/anw-w4-l02.md). You don't need it to finish the lesson. Read it when you want the longer version.
The starter checks, in full
Every team starts with six checks. They come from how the AI Build Lab's other courses grade work.
| Level | Check | Fails if |
|---|---|---|
| Whole job | Correct | anything in it is wrong |
| Whole job | Complete | it leaves a question unanswered without saying so |
| Whole job | Ready to use | you'd have to change it before sending or using it |
| Each agent | Correct | anything in its piece is wrong, given what it was handed |
| Each agent | Complete | it skipped part of its piece |
| Each agent | Right shape | the next agent couldn't use it without asking back |
You keep the ones that matter to your team and sharpen them. You can drop one that doesn't fit.
How Gigawatt sharpens a rule
Gigawatt uses its rubric method (aibl-rubric-forge) for this. For each rule it asks:
- Is it yes or no? "Mostly accurate" isn't. "States a fact that isn't in the menu guide" is.
- Does it say what makes it fail? Each rule names the exact thing that fails it.
- Would two people agree? Gigawatt tries the rule on one of your cases and asks whether a second person would grade it the same way.
- Is it one thing? A rule that checks two things is two rules.
How many rules
Start with three to five for the whole job, and three to five for each agent that needs them. Not every agent needs its own rules at first: only the ones that failed, or that your design's "never" lines cover. As you grade more work, add rules. Never go past eight on one level. If a step needs more than eight, it's doing too much, and that's a design question for Archie.
Versions
Your rule sheet is kept in versions in your team's folder, work/course/staff/teams/<team>/rules/. Version 1 is what you approve today. Every change makes a new version, and every version is kept, so you can always see which rules a grade was given against. Changing your rules while you grade is normal: you learn what you care about by grading.
Sealing
Your grades for each case are saved by the Professor as one grade file per case, in your team's folder under owner-grades/, holding only the rules you graded, with your reason for each. Your Chief of Staff seals each file with a fixed script, on your ask, before Evy grades: the script takes a fingerprint of the file, a short code worked out from every character in it, and refuses to seal a case Evy has already graded. Evy never seals your grades: the agent being compared can't hold the lock on your answers. If anything in the file changed later, the fingerprint wouldn't match, and the comparison would refuse to count it. A grade you'd change later goes in the comparison as "Evy caught something I missed", never as an edit.
Who saves the rule sheet
Gigawatt sharpens the wording with you and shows it. The Professor saves it, as a new version, only after you say yes to the exact text, and reads it back. Gigawatt never saves it.
Pass or fail only
No scores out of ten and no "partly". A score invites arguing over a 6 or a 7. Pass or fail, with a reason, invites asking what would make it pass.