Compare your grades with Evy's and teach Evy
How far can I trust Evy on this team, and how do I teach Evy to grade the way I do?
- Source
- Internal's lesson, src/week4/internal/lessons/anw-w4-l04.md (Internal 60d90336, 2026-10-19 19:39), read by md_week4.py.
- Copy
- Internal's Markdown as published to the student shelf 10-12 (Internal 101e2662a = student 8275f21b; PR #215, receipt #218), vendored byte for byte by vendor_week4.py from the student-edition overrides; the published student tree is the same files without the "Staff only" blocks (checked on import). The drawings are the inline SVG figures in the lesson files, used as written.
- Prompts
- Word for word from the lesson file Internal ships (course/workforce/week4/). Not run end to end from Learn.
- Checked in the app
- Not checked in the app from Learn.
- Open to students
- Yes.
- Other view
- one part at a time (one part at a time is the lesson; one scroll is kept to compare)
Staff review only: full controlling prompt. Students paste only the short request below.
Use the native aibl-the-professor agent to guide the Week 4 lesson Compare your grades with Evy's and teach Evy. Read course/workforce/week4/lessons/anw-w4-l04.md completely, its details page course/workforce/week4/reference/anw-w4-l04-details.md when asked for more, and the native aibl-course-guide and aibl-lesson-map methods for this app. You teach and read; you never count agreement, decide a disagreement, word a rule, write in Evy's notebook or set how much the student checks by hand, and you never count patterns. One save is yours: when the student brings a sharpened rule sheet from Gigawatt, show the exact file and, on their yes in their own words, save it as the next version, work/course/staff/teams/<id>/rules/rules-v<n>.md, a new file that never replaces an earlier version. The student talks to the Evaluation Analyst (Evy), to the Operator (Gigawatt) and to the Chief of Staff, which sends the Evidence Pattern Analyst (Mitch Please) its job, in their own threads. Do not call, spawn or route to any agent. Edit no other file.
Use these words exactly. The comparison sets the student's sealed grades beside Evy's, case by case, at the job level and for each agent. How often they agree is counted by a fixed script, never by Evy and never by you, only over the rules both graded, for each team, each level and each rule, and saved as agreement.json in the team's folder under work/course/staff/teams/. The script also reports agreement on the cases the student failed, separately, because a team that almost always passes can show high agreement while Evy is only waving work through. Every disagreement is one of three kinds: the student's rule was unclear (Gigawatt sharpens the rule with the student, and you save the new version), Evy was wrong (it goes in Evy's notebook for this team), or Evy caught something the student missed (the student says so; the sealed grade is never changed, and the count stays honest). Evy's notebook for a team is evy-notebook.md in the team's folder: every case both graded, agreements included, with both grades and the student's one-line reason; Evy reads it before grading that team. How much the student checks by hand steps down as agreement holds: every case, then half, then one in five, then only what Evy flags as unsure or serious. The thresholds are fixed in the script, and a disagreement on a serious rule steps it back up. Name agents job title first, nickname in brackets on first use; Evy is "it". Say "your work", never "business"; refer to other lessons by title, never by number.
The Hattie B's practice inbox is the worked example on the page. Never reveal or paraphrase any answer key for the practice emails; use only what the lesson page shows.
Ask open questions only. Never sort a disagreement into a kind for the student, and never say whose grade was right.
Part 1. If asked, read the comparison the student shows, read only, and say back plainly what each number counts.
Part 2. If asked, put the student's reason and Evy's story for one disagreement side by side, quoting both, and walk the page's three questions in order (did Evy read the work correctly; could the rule be read two ways; otherwise Evy caught something the student missed). The student decides.
Part 3. If asked, read Evy's notebook for the team, read only, and help the student with the page's two checks: are their reasons in their own words, and would they want each correction applied to every future job on this team.
Part 4. If asked, read the script's line on how many cases the student still grades themselves, and explain the steps down. Grading it yourself means the student grades and seals a case themselves so it can be compared. A serious rule is one whose line on the rule sheet says "serious"; a disagreement on one moves the steps back up at once.
Part 5. If asked, read the Evidence Pattern Analyst (Mitch Please)'s reply the student shows, read only, and explain plainly what observed, inferred, a count and a single signal mean. Mitch counts and quotes; it never fixes, regrades or picks a fix, and a cause is always inferred. Never confirm, reject or revise a pattern for the student, and never pick the problem to fix.
Otherwise, if asked, explain the step and what the student should see.
Staff only: status, sources, what the build supports. Not student-facing.
Status: first draft, 8 October 2026, lane "lessons" (dispatch 51bbe9da), from spine v2 Module 2 and DESIGN-week4-2026-10-08.md section 4.4 (notebook at work/course/staff/teams/<id>/evy-notebook.md; agreement counted by agreement.py, never by Evy, per team, level and rule, written to work/course/staff/teams/<id>/agreement.json; fails reported separately; steps all, half, one in five, only unsure or serious; a serious-rule disagreement steps back up; agreements kept as well as disagreements).
Bound to the Evy lane (PR #202). Evy runs agreement.py compare and count and quotes the summary line; count writes agreement.json (aibl.agreement.v1). Only rules both the owner and Evy graded are compared (rows_for takes the intersection), so the page says so. Evy writes its notebook itself (its may_write). A rule change is sharpened by Gigawatt and saved by the Professor as a new version (the Professor's Week 4 may_write, this PR); Gigawatt never saves it.
- The spine's title says "teach her"; the page says "teach Evy" (Sara's rule 4, no gendered pronouns for an agent). The lesson map carries the same title.
- "Evy's notebook" is the student-facing name for the spine's "Evy's notes on how you grade" (spine open item 3, Sara to confirm).
- The Hattie B's comparison below is a made-up worked example built on the version 1 rule sheet shown in "Turn what you found into the rules for this team" (J1 to J4, A1 to A12). Its numbers are illustrative, not a script's output. None of its disagreements restates the internal answer key.
Part 5, Mitch Please (lane new-agents, W4 push 3, card a46c2bd2, 13 October 2026). Placement proposed to Sara (handoff/new-agents-placement.md); publish waits on her OK. Checked against Mitch's agent file in .claude/agents/ and the aibl-evidence-pattern-analysis skill: Mitch reads Evy's runs/<job>/evaluation/ files that a work order names, counts out of a named total, labels OBSERVED, INFERRED (low, medium or high, another explanation, a settling question) or UNKNOWN, never states a rate or cause as observed, needs two distinct records for a pattern, never fixes, regrades or decides, and returns a confirm, reject or revise queue to the Chief. With no work order there is no job, so the page sends the student to the Chief, whose Week 4 seats line routes this question to Mitch with the records named. Mitch writes only under runs/<job>/patterns/. The Hattie B's reply is a made-up example built on the rule sheet and grades shown on this page; its shape is abridged and it does not restate the internal answer key.
| Figure ID | Lesson | What it shows | Caption | Alt text |
|---|---|---|---|---|
w4-l04-three-kinds | anw-w4-l04 Part 2 | One box "a disagreement" splitting into three arrows: "my rule was unclear" to "change the rule, new version"; "Evy was wrong" to "goes in Evy's notebook"; "Evy caught what I missed" to "say so; my sealed grade stays" | Every disagreement teaches something, and each kind goes to its own place. | When your grade and Evy's disagree, it is one of three things. If your rule was unclear, you change the rule and the rule sheet gets a new version. If Evy was wrong, it goes in Evy's notebook for this team. If Evy caught something you missed, you say so, and your sealed grade stays as it was. |
w4-l04-check-less | anw-w4-l04 Part 4 | Four steps going down left to right: every case, half, one in five, only what Evy flags; a curved arrow from the last step back to the first labelled "a disagreement on a serious rule" | You check less as you and Evy keep agreeing, and more the moment you don't. | How much you check by hand steps down as you and Evy keep agreeing: every case, then half, then one in five, then only what Evy flags as unsure or serious. A disagreement on a serious rule steps it back up. |
You and the Evaluation Analyst (Evy) have both graded the same cases. Today you put the grades side by side and see how often you agree.
Where you disagree, you decide which of three things happened:
- Your rule was unclear. You reword it.
- Evy was wrong. That goes in Evy's notebook for this team, its notes on how you grade, so it grades your way next time.
- Evy caught something you missed. You say so. Your sealed grade stays as it was.
Agreements go in the notebook too: they show Evy what you count as good.
The more you and Evy agree over time, the fewer cases you grade yourself.
Last, the Evidence Pattern Analyst (Mitch Please) looks across all of Evy's grades for this team and counts what keeps happening, so you can pick what to fix next.
How often you and Evy agree, a clearer rule sheet, Evy's notebook for this team, how many cases you still grade yourself, and Mitch's count of what keeps happening.
This is how you stop checking everything. You trust Evy where Evy has earned it, and only there.
The Course Companion (The Professor) guides you in its own thread and saves a changed rule sheet on your yes. Evy shows you the comparison and keeps its notebook. The Operator (Gigawatt) helps you reword a rule. Your Chief of Staff sends Mitch the job of counting patterns.
You decide what each disagreement means. Nobody decides for you.
Before you start
- Evy has graded your sealed cases. You did it in "Have Evy grade the same work".
- Nothing to open. Evy reads both sets of grades.
- In Codex, start your first message to an agent in a thread with "Ask the <agent> to do this, and show me its reply word for word:". Start every message after that with "Tell the <agent> this, word for word, and show me its reply word for word:".
Start or resume this lesson
Open your prepared workbench and call The Professor by its role name, aibl-the-professor. Paste only this.
Use the native aibl-the-professor agent to start the Week 4 lesson Compare your grades with Evy's and teach Evy. Read course/workforce/week4/prompts/anw-w4-l04.txt and course/workforce/week4/lessons/anw-w4-l04.md. Ask me one question at a time and wait. I will talk to Evy and Gigawatt in their own threads.
The full instructions your agent follows for this lesson are in course/workforce/week4/prompts/anw-w4-l04.txt.
Evy shows you your grades and its grades side by side, rule by rule, with a count of how often you agree.
Counted by a script, never by Evy
- A fixed script counts how often you agree, the same way every time. Evy shows you the result, but never decides the number.
- Why not Evy? Evy is one of the two graders. Letting it count would be like letting it mark its own homework.
- Only the rules you graded count. A rule you left blank never counts as agreeing.
- Your grade and Evy's are matched by the rule's label, the part before the colon:
J1: No made-up food factsis rule J1. Write each rule as its label, a colon, then a short name. - A rule is serious when its line on your rule sheet says "serious". A disagreement on a serious rule moves you back up a step at once (Part 4).
- The count is kept at each level and for each rule, so you can see exactly where you differ.
- The cases you failed are counted again, on their own. If a team almost always passes, you and Evy can agree a lot while Evy is only waving work through. Agreeing on fails is what shows Evy catches what you catch.
We show you: the Hattie B's practice inbox
The example student failed four job rules: two on the kids plate question, and two on the cold-food complaint. Evy failed the same four, plus one more: J4 (nothing I'd have to change) on the catering quote.
The script's count, word for word:
Agreement for team email-team: 5 job(s) counted; incomplete, not counted: 0; not graded by you: 0. Agent level: 18 of 20 agree. Job level: 19 of 20 agree. On your fails Evy also failed 7 of 7. Hand checks: all (check every job by hand). Wrote work/course/staff/teams/email-team/agreement.json
Then the comparison for each case. Here is the catering quote's:
Compared 2026-10-07-support-catering-quote for team email-team (owner sealed first, then Evy)
| level | agent | rule | owner | Evy | agree | the owner's reason |
|---|---|---|---|---|---|---|
| agent | aibl-hatch | A5 | PASS | FAIL | NO | The date is tagged as missing, which is right. |
| agent | aibl-ygm | A7 | PASS | PASS | yes | Every fact has its source. |
| job | whole job | J1 | PASS | PASS | yes | No made-up food facts. |
| job | whole job | J4 | PASS | FAIL | NO | I'd send it as it is. |
2 of 4 agree
How to read your count
- "Job level: 19 of 20 agree." Four job rules across five cases makes 20 grades. You and Evy gave the same grade on 19.
- "On your fails Evy also failed 7 of 7." Every time you failed something, Evy failed it too. This is the most important number: it shows Evy catches what you catch.
- "Agent level: 18 of 20 agree." The same count for the agent rules you graded.
- "Hand checks: all" is how many cases you still grade yourself (Part 4).
- Each case's table shows every rule you both graded, with your reason. A row that says NO is a disagreement. You work through them in Part 2.
There's no pass mark today. What matters is the count over time: in Part 4 you'll see how it decides how many cases you still grade yourself.
Do this now: ask Evy for the comparison
In a new thread with Evy, say this, with your team's name:
Compare my sealed grades with yours for my <team name> team, at the job level and for each agent. Show me the counting script's result word for word, then list every disagreement.
What you should see
The script's count, word for word, in the same shape as the example, then each case's table, ending with how many rules agree, and a numbered list of every disagreement.
For each disagreement, you decide which of the three things happened. Each kind goes to a different place.
Every disagreement is one of three things, and each goes to a different place.
How to tell which kind it is
Ask three questions, in this order:
- Did Evy read the work correctly? If not, Evy was wrong.
- Did Evy read it right, but could your rule be read two ways? Use the two-people test. If yes, your rule was unclear.
- Did Evy read it right, and is your rule clear? Then Evy caught something you missed.
We show you: the Hattie B's practice inbox
The example student's three disagreements, side by side:
| 1. The catering quote's blanks | 2. The catering quote's date | 3. Marcus's sign-off | |
|---|---|---|---|
| The rule | J4 · Nothing I'd have to change | A5 · Missing means tagged, not guessed | A9 · The practice voice's greeting and sign-off |
| Evy said | Fail: the reply has blanks for the owner to fill in | Fail: the researcher gave a date that wasn't in the sources | Fail: the reply signs off "Cheers!" |
| The student had said | Pass | Pass | Pass |
| Did Evy read it right? | Yes | No: the researcher tagged the date as missing, and Evy read the line below it | Yes |
| Could the rule be read two ways? | Yes: "anything I'd have to change" might or might not include filling in a blank | No need to ask: Evy misread the work | No |
| So | My rule was unclear | Evy was wrong | Evy caught what I missed |
| Where it went | Reworded to "...anything other than a blank left for me", saved as version 2 | Evy's notebook: "On this team, a missing tag is the researcher doing its job." | Said so to Evy. The sealed grade stays a pass. |
Why does a wrong grade stay? Your sealed grade stays as it was, so the count shows how often you and Evy really agreed. What you learned goes in Evy's notebook instead.
Do this now: decide each disagreement
Take the disagreements one at a time. For each, in the same Evy thread, say:
Show me disagreement <number>: my reason and your story, side by side, word for word.
Then say which kind it is, in your own words:
- Evy was wrong, or Evy caught something I missed: tell Evy in the same thread, with why, in as much detail as you need. Evy adds it to its notebook for this team and reads it back.
- My rule was unclear: go to your Gigawatt thread from "Turn what you found into the rules for this team", or start a new one, and say:
My rule <label and short name> was unclear on <case>. Here is what I meant: <your words>. Help me reword it and show me the whole sheet with the change. Don't save it.
Copy the whole sheet from Gigawatt's reply. Then, in your Professor thread, say "Save this as the next version of my rule sheet" with the sheet pasted under it, and say yes when it shows you what it will save. The earlier version is kept.
If you can't tell which kind it is, ask The Professor, in its thread, to put the two side by side with you. It won't decide.
What you should see
For each disagreement, Evy shows your reason and its story side by side. Then either Evy reads back the line it added to its notebook, or the Professor shows you the new version of your rule sheet before it saves it.
What the notebook is
Evy keeps a notebook for each of your teams: one file, saved with your team's records, where it writes down how you grade.
- Every case you and Evy both graded, with both grades and your reason, whether you agreed or not.
- Every correction you've made to Evy, like the one about the researcher's missing tags.
- From later this week, small rulings from your everyday work. If you tell your Chief "Always say which location you mean", Evy's daily check notices it and adds it here.
Why it matters. Before Evy grades anything for this team, it reads the notebook first. So every line changes how Evy grades from then on. Your rule sheet says what counts as a pass or a fail. The notebook shows how you actually apply it, case by case. Think of a new marker reading the head marker's past papers before marking their own.
The better the notebook, the more often Evy's grades match yours, and the fewer cases you grade yourself (Part 4). Evy writes it. You change it by telling Evy.
We show you: the Hattie B's practice inbox
Part of the example student's notebook:
Kids plate question, J1 · No made-up food facts: both FAIL.
"Says the fries are vegetarian, and the menu guide doesn't. A pass leaves that question blank for me."
Marcus, J4 · Nothing I'd have to change: both PASS.
"I'd send it as it is."
Catering quote, A5 · Missing means tagged, not guessed: owner PASS, Evy FAIL.
Evy was wrong: "On this team, a missing tag is the researcher doing its job."
Marcus, A9 · The practice voice's greeting and sign-off: owner PASS, Evy FAIL.
Evy caught what I missed: "Use the practice voice's sign-off."
How to check it
Read every line and ask:
- Are your reasons in your words, not Evy's summary of them?
- Would you want each correction applied to every future job on this team? "A missing tag is the researcher doing its job" will shape every grade Evy gives this team. If a line is too broad or too narrow, tell Evy how to word it.
Do this now: read it back
In the same Evy thread, say:
Show me your notebook for my <team name> team, word for word.
If a line doesn't say what you meant, tell Evy what to change, in your own words.
What you should see
The notebook, word for word. When you ask for a change, Evy reads the changed line back to you.
You won't have to grade every case yourself forever. As you and Evy keep agreeing, you grade fewer yourself, and leave the rest to Evy.
Grading a case yourself works the way it did this week: you grade it and seal it, so it can be compared with Evy's grade. Only cases you've graded and sealed count.
Everyone starts by grading every case themselves.Each step down takes more jobs and more agreement, including on the cases you failed.
Everyone starts by grading every case themselves. The same counting script decides when you move down a step, using the fixed steps in the picture, so nobody talks you into it, not even Evy.
The "cases you failed" part is the safety check. If you only ever agree on passes, Evy might just be waving work through, so you have to agree on fails too.
Do this now: ask how many cases you still grade yourself
In the same Evy thread, say:
From the counting script: how many of this team's cases do I still grade myself, and what would move it down?
What you should see
After this lesson, the script still says to grade every case yourself: moving down a step takes at least 10 jobs you and Evy have both graded, and you have 3 to 5 so far. That's expected. The steps only move as you grade more of your team's work yourself, so there's more to compare with Evy's grades.
So far you've looked at one disagreement at a time. Now you look across all of Evy's grades for this team at once, and ask what keeps happening. You get a short list of patterns to take into "Pick the problem and agree the fix".
Who does this: the Evidence Pattern Analyst (Mitch Please)
- Mitch counts and quotes. It reads Evy's grades for every case you name, and says what repeats, what changed, and what is missing. Every pattern it observed comes with a count, like "seen in 3 of 5 jobs", and a quote from each grade that shows it.
- It labels every pattern. Observed means the grades show it. Inferred means it's a possible reason, with how sure Mitch is (low, medium or high), another way to explain it, and the one question that would settle it.
- Counts, never percentages or causes. Mitch never says "60 percent" and never says why something happened as if it were a fact.
- One grade is not a pattern. A pattern needs at least two cases. With only three cases, Mitch may say "not enough signal", and that's a correct answer.
- You decide. Mitch hands you a review queue: confirm, reject or revise each pattern. It never fixes anything, regrades anything or picks what to fix.
Your Chief of Staff sends Mitch the job. It writes the order with your question and the grade files for each case, then shows you Mitch's reply.
We show you: the Hattie B's practice inbox
The example student asked: "Which problem should I fix first on my email team?" Part of Mitch's patterns file and reply, for the five practice emails:
Records read: 5 (Evy's grades for 5 jobs, team email-team). Could not read: none.
P1 OBSERVED J4 · Nothing I'd have to change failed in 3 of 5 jobs
R1 kids plate: "the fries line would need changing"
R3 cold-food complaint: "the refund line would need changing"
R5 catering quote: "blanks are left for the owner to fill in"
P2 OBSERVED in 2 of those 3 jobs, another job rule failed too
R1 kids plate, J1: "says the fries are vegetarian; the menu guide doesn't"
R3 cold-food complaint, J2: "offers a full refund"
P3 INFERRED confidence low, because only two jobs show it:
the writer fills a gap with something only you can give.
Another explanation: the evidence the writer got already held the guess.
Would settle it: does the researcher's evidence for R1 and R3 hold those lines?
Single signal, not a pattern: A9 · sign-off, on R4 Marcus only.
Review queue: P1, P2, P3. Confirm, reject or revise each.
These are patterns for you to confirm, reject or revise, not facts.
Nothing here changes a seat or a rule until you decide.
The student revised P1: "R5 is the one I decided was my unclear rule. The team's own mistakes are R1 and R3." They confirmed P2, and kept P3 as a question to check. The count stays 3 of 5, because Mitch counts Evy's grades exactly as they were recorded. Deciding what they mean was the student's call.
Do this now: ask your Chief for Mitch's count
In your Chief of Staff thread, say this, with your team's name:
Have Mitch Please look across Evy's grades for every case on my <team name> team. My question: which problem should I fix first? Name each grade file in the order, and show me Mitch's reply and its patterns file word for word.
Then, in the same thread, tell your Chief what you make of each pattern, in your own words: confirm, reject, or revise it with your correction.
What you should see
Mitch's reply, word for word: how many grade files it read, each pattern with its label and its count, and the review queue. Its patterns file has the rest: a quote from each case that supports an observed pattern, the reason, another explanation and a settling question for every inferred one, and anything seen only once. If Mitch says "not enough signal", that's fine: take your own pick into the next lesson.
If you get stuck
| What you notice | What to do |
|---|---|
| You and Evy agree on everything | Good, but look at the line for the cases you failed. If you failed none, the count can't tell yet whether Evy would catch a mistake. That's what the harder cases later this week are for. |
| You want to change your sealed grade | Don't. Say "Evy caught something I missed" instead. The sealed grade keeps the count honest. |
| You can't tell whether your rule was unclear or Evy was wrong | Go through the three questions in Part 2, in order. The second one is the two-people test: would a second person, reading your rule, grade it the way Evy did? |
| Evy argues with your decision | Your decision stands. Evy writes it down; it doesn't overrule you. |
| Evy gives you a number without the script | Ask for the script's result word for word. Evy never decides the count. |
| Mitch says "not enough signal" | That's a correct answer with only a few cases. Pick your problem yourself in the next lesson, and ask Mitch again once Evy has graded more. |
| Mitch's reply gives a percentage or a cause as fact | Ask your Chief to send it back to Mitch: counts only, and a cause is always inferred. |
How the count works, what's in the notebook, and when you step down: more detail on this lesson.
What you'll keep
How often you and Evy agree on this team, a clearer rule sheet, Evy's notebook for this team, how many cases you still grade yourself, and Mitch's patterns with your decision on each.
Before you finish
Self-assessed. No gate.
Words you now own
- Agreement: how often you and Evy give the same grade, counted by a script for each level and each rule.
- Evy's notebook: Evy's notes on how you grade one team: every case you both graded, your reasons and your corrections. Evy reads it before grading that team.
- Pattern: something that shows up in more than one case, with a count like "2 of 5 jobs". One case alone is a single signal.
- Grading less yourself: every case, then every second case, then one in five, then only what Evy flags as unsure, as you keep agreeing over time. Evy grades the rest.
Next: Pick the problem and agree the fix.