AI Build Lab / Learn · Agent Workforce · Week 4 · Module 02

Have Evy grade the same work

How does an agent grade my team's work, and can I see exactly why it decided what it did?

Staff only · this page and its checks

Source: Internal's lesson, src/week4/internal/lessons/anw-w4-l03.md (Internal 60d90336, 2026-10-19 19:39), read by md_week4.py.
Copy: Internal's Markdown as published to the student shelf 10-12 (Internal 101e2662a = student 8275f21b; PR #215, receipt #218), vendored byte for byte by vendor_week4.py from the student-edition overrides; the published student tree is the same files without the "Staff only" blocks (checked on import). The drawings are the inline SVG figures in the lesson files, used as written.
Prompts: Word for word from the lesson file Internal ships (course/workforce/week4/). Not run end to end from Learn.
Checked in the app: Not checked in the app from Learn.
Open to students: Yes.
Other view: one scroll (one part at a time is the lesson; one scroll is kept to compare)

Start hereWhat this lesson is, and how to begin

Staff review only: full controlling prompt. Students paste only the short request below.

Use the native aibl-the-professor agent to guide the Week 4 lesson Have Evy grade the same work. Read course/workforce/week4/lessons/anw-w4-l03.md completely, its details page course/workforce/week4/reference/anw-w4-l03-details.md when asked for more, and the native aibl-course-guide and aibl-lesson-map methods for this app. You teach and read; you never grade, start Evy's grading, change a grade, or compare grades for the student. The student talks to the Evaluation Analyst (Evy) in its own thread. Do not call, spawn or route to any agent. Do not edit any file.

Use these words exactly. Evy grades the student's sealed cases against the team's newest approved rule sheet, without seeing the student's grades. Each grade has two sides. The number side is PASS or FAIL for each rule, for each agent and for the whole job, never added together. The story side comes in this order: what Evy looked at (a short quote from the work), why it passes or fails against the rule, for a fail which agent introduced it and how Evy traced it, what would fix it, how sure Evy is, and anything Evy couldn't check ("not verified" rather than a guess). Evy writes each grade in the case's job folder under runs/<job>/evaluation/: evaluation.md for the story and evaluation-binary.json for the number side. Evy holds every tool and connection any agent in the organization has, so it can redo a step to check it, for example opening a source a researcher cited or running a search again. Evy never holds the actions that send anything outside, post publicly, delete, move money or change account authority, and writes only in its own places. Name agents job title first, nickname in brackets on first use; Evy is "it", never a gendered pronoun. Say "your work", never "business"; refer to other lessons by title, never by number.

The Hattie B's practice inbox is the worked example on the page. Never reveal or paraphrase any answer key for the practice emails; use only what the lesson page shows.

Later this week, Evy grades every job, including jobs the student grades. For those jobs, Evy's answer stays hidden from the student until the student's grade is saved and sealed. Remind the student: "Don't open Evy's folder for a job you're about to grade."

Ask open questions only. Never say whether Evy's grade is right, and never tell the student how their own sealed grade compares; that is the next lesson.

Part 1. If asked, explain what Evy reads and what it can and can't do, from the page.

Part 2. If asked, help the student find their sealed cases and newest rule sheet by name so they can name them to Evy. Never start the grading.

Part 3. If asked, read one evaluation.md the student opens, read only, and help them find each part of the story in order and ask the page's three checks: is the quote real words from the work, does why point at one of their rules, and for a fail does it name the agent and how Evy found out. Say back plainly which agent Evy traced the fail to and how.

Part 4. If asked, point to the line in the grade where Evy redid a step, and say what it opened or ran and when.

Otherwise, if asked, explain the step and what the student should see.

Staff only: status, sources, what the build supports. Not student-facing.

Status: first draft, 8 October 2026, lane "lessons" (dispatch 51bbe9da), from spine v2 Module 2 and DESIGN-week4-2026-10-08.md sections 4.1 to 4.3 (on demand, the owner asks Evy directly; every tool any seat has minus the smoke alarm; the number side and the six-part story; runs/<job>/evaluation/evaluation.md and evaluation-binary.json).

Bound to the Evy lane (PR #202).

Evy takes the owner's requests directly and never takes work from the Chief ("On demand", "Never for the Chief" in its body). It holds Read, Glob, Grep, Write, Edit, Bash, WebFetch, WebSearch, the browser tools, Agent and SendMessage, minus the smoke alarm, so the redo-a-step claims in Parts 1 and 4 match the body's step 5.
Evy writes runs/<job>/evaluation/evaluation.md, evaluation-binary.json and return-evy.md, and watch items through aibl-kansa-watch. Evy runs agreement.py compare and starts the notebook only in the next lesson, when the owner asks to compare (walk fix 2026-10-09).
No trace copy: Evy writes "not verified: no trace copy for this job" (read item 4), so a student without Langfuse can do this lesson from the job folders alone. Part 1 says so.
Evy passes everything: Part 3 has the branch (read the first case instead). Module 3 has the all-pass path.
The Hattie B's grade below is a made-up worked example. Its trace is the design's own example (researcher tagged it couldn't find; the evidence package kept the gap; the writer filled it). It does not restate the internal answer key.

Figure ID	Lesson	What it shows	Caption	Alt text
`w4-l03-grade-card`	`anw-w4-l03` Part 3	One grade drawn as a card with two halves: on the left, a short column of PASS and FAIL marks (each rule, each agent, the job level); on the right, six numbered lines of the story (what it looked at, why, which agent and how Evy found out, the fix, how sure, not verified)	Every grade has a number side and a story side, and the story is what you can check.	One of Evy's grades has two sides. The number side lists PASS or FAIL for each rule, at the agent level and the job level. The story side gives, in order, the quote Evy looked at, why it passes or fails, which agent introduced a fail and how Evy traced it, what would fix it, how sure Evy is, and what it couldn't check.

In plain English

In the last lesson you wrote your team's rules and graded your cases against them. Your grades are now sealed.

Today the Evaluation Analyst (Evy) grades the same cases, against the same rules. It never sees your grades. Think of a second marker grading the same exam papers with your marking guide.

Why? Later this week, Evy grades your team's work every day on its own. Before you trust it with that, you need to see how it grades.

Later this week, Evy grades every job, including the few the counting script picks for you to grade yourself. For those jobs, you won't see Evy's answer until you've saved your grade and your Chief has sealed it. Don't open Evy's folder for a job you're about to grade.

Evy doesn't just read the work. It can redo a step to check it. For example, it can open the file the researcher used and search it again.

Every grade Evy gives shows what it looked at and why, so you can check its thinking.

Today you only read Evy's grades. Comparing them with yours is the next lesson.

What you get

Evy's grade for every case, each one quoting the work Evy judges.

Why it matters

An agent that grades the way you do can check far more work than you have time for. First you need to see how it grades.

How it works

The Course Companion (The Professor) guides you in its own thread. Evy grades your cases in Evy's own thread and saves each grade beside the work.

Your part

You start the grading, then read one grade through, all the way down.

Before you start

Your rule sheet is saved, and your grades are sealed. You did both in "Turn what you found into the rules for this team".
Nothing to open. Evy reads your cases and your rule sheet directly.
In Codex, start your first message to an agent in a thread with "Ask the <agent> to do this, and show me its reply word for word:". Start every message after that with "Tell the <agent> this, word for word, and show me its reply word for word:".

Start or resume this lesson

Open your prepared workbench and call The Professor by its role name, aibl-the-professor. Paste only this.

Use the native aibl-the-professor agent to start the Week 4 lesson Have Evy grade the same work. Read course/workforce/week4/prompts/anw-w4-l03.txt and course/workforce/week4/lessons/anw-w4-l03.md. Ask me one question at a time and wait. I will talk to Evy in Evy's own thread.

The full instructions your agent follows for this lesson are in course/workforce/week4/prompts/anw-w4-l03.txt.

Part 1 of 4See what Evy reads, and what Evy can't do· 7 min

Before you ask Evy to grade, you see what it reads and what it's allowed to do. Then you know what to trust in its grades, and what to question.

What Evy reads, and why

For each case, Evy reads everything your team left behind, plus your rules:

Evy reads	So it can check
Each agent's instructions for the job	whether the agent did what it was told
What each agent handed back	where a mistake first came in
The researcher's notes: what it found, and what it tagged as missing	whether a fact was really missing
The draft, and the Quality Reviewer (Kansa)'s review	what the customer would get, and whether Kansa caught the problems
Your rule sheet	what counts as a pass or a fail
The record of the run in Langfuse, the recorder from Week 2	what each agent actually did, step by step

Evy never sees your sealed grades. That's what keeps the comparison fair.

If you didn't set up Langfuse, that's fine. Evy grades from everything else. Anything only the recording would show, Evy marks "not verified" instead of guessing. You set the recorder up later this week.

What Evy can do, and what Evy never does

Evy can	Evy never
Open a file the researcher used	Sends anything outside your organization
Run a search again	Posts anything in public
Run the same script another agent ran	Deletes anything
Read the recording of a run in Langfuse	Moves money or changes an account

Why? Evy holds the same tools as your other agents, so it can redo their work and check it, not just read about it. It's kept away from anything that can't be undone.

Evy writes only its own grades, and a few notes you'll meet later this week.

Do this now: ask Evy what Evy holds

In Claude Code: start a new thread, type @ and pick aibl-evy. In Codex: start a new thread and begin with "Ask the Evaluation Analyst (Evy) to do this, and show me its reply word for word:". Use this thread for the rest of the lesson. Say:

Before you grade: which tools do you hold, which ones did you leave out on purpose, and where will you save my grades?

You should see a short list, with sending, posting, deleting, money and account changes left out.

Part 2 of 4Ask Evy to grade your sealed cases· 7 min, plus agent time

You send Evy one message to start. Evy then grades every case on its own. It takes a while, so you can do something else while it works.

What Evy does while you're away

For each case, Evy works in the same order every time:

Checks nothing is missing from the job: every file the team should have left is there and readable.
Reads your rules first, then each agent's instructions, then what that agent handed back.
Grades every rule, at the agent level and the job level.
Writes the story behind each grade: what it looked at, why, and the rest of the six steps you'll read in Part 3.
Redoes a step itself when the record isn't enough to be sure, like searching the menu guide again.
Saves the grade beside the work it judges.

Do this now: start Evy

In the same Evy thread, say this, with your team's name:

Grade my sealed <team name> cases against my newest rule sheet for this team. Grade at the agent level and the job level, pass or fail for every rule. Don't look at my grades. Tell me when every case is done.

Why "newest"? Your rule sheet has versions. Evy must grade against the one you graded with and sealed.

You can walk away. Allow a few minutes per case.

What you should see

One short line per case as it finishes, with where the grade is saved, then "done".

Your agents, rules and drafts don't change. For each fail, Evy may also note something for Kansa to check next time. You'll see those notes later this week.

Part 3 of 4Read one grade, all the way down· 10 min

You read one of Evy's grades from top to bottom. Every grade is built the same way, so once you can read one, you can read them all quickly.

Every grade Evy gives has two sides.The number side is a pass or fail for every rule. The story side says, in the same six steps every time, why.

We show you: the Hattie B's practice inbox

The card above is Evy's grade for the kids plate question.

The number side, on the left, gives a pass or fail for every rule. Five failed:

J1 · No made-up food facts
J4 · Nothing I'd have to change
A7 · Draft uses only the evidence (the writer)
A8 · No guessing to fill a gap (the writer)
A10 · Kansa catches unsupported facts

Every other rule passed, including A9, the greeting rule the student didn't grade. Evy grades every rule anyway.

The story side, on the right, says why, in six steps. Read it top to bottom: the quote about the fries, why that breaks J1, how Evy traced it to the writer and Kansa, the fix, how sure Evy is, and what it couldn't check.

How to check a grade

Don't just read Evy's grade. Check it. For each grade, ask:

Is the quote real words from the work? If not, Evy may be guessing.
Does "why" point at one of your rules? If not, Evy is grading against something you didn't decide.
For a fail, does it name the agent and say how it found out? If not, you can't tell what to fix.

Do this now: read your first fail

In the same Evy thread, say:

Show me your grade for the first case you failed, both sides, word for word. If you failed none, show me your first case.

Read it in the same order as the card, and ask the three questions. A pass has a story too: the quote that shows each rule was met. If a part is missing, ask Evy for it in the same thread.

What you should see

Evy's grade, both sides, word for word: a pass or fail for every rule, then the six steps of the story, in the same order as the card.

If Evy failed nothing

That's a result, not a dead end. Read one pass all the way down and check each rule has a quote behind it. A pass with no quote is a guess. If every case passed for you too, later this week you add a harder case.

Part 4 of 4See Evy redo a step to check it· 6 min

Evy doesn't just read what your other agents wrote. Sometimes it goes back and does part of their work again, to see if they got it right. In this part, you ask Evy whether it did that for your grade, and where.

Why it matters. An agent that only reads another agent's notes can repeat that agent's mistakes. One that goes back to the source can catch them. It's like an editor checking a quote against the original, instead of trusting the writer's notes.

Evy isn't Kansa. You met Kansa, the Quality Reviewer, last week. Evy checks differently:

	Kansa	Evy
When it checks	Before you send a reply	After the job is finished
What it checks	The draft	Every agent's piece, and the whole job
Where it looks	Only at what the team handed it	It can also go back to the original files and check them itself

We show you: the Hattie B's practice inbox

For the kids plate question, Evy didn't just take the researcher's word that the vegetarian answer was missing. It opened the menu guide the researcher used and searched it for anything about vegetarians. It found nothing. That's how it could say the gap was real and the writer filled it with a guess.

Do this now: find the step Evy redid

In the same Evy thread, say:

In that grade, which step did you redo yourself to check it? Show me what you opened or ran, and when.

What you should see

The file Evy opened, or the search it ran again, and when it did it.

If Evy redid nothing, that's fine when the files were enough. Evy only redoes a step when the record isn't enough to be sure. Ask it why it trusted the files.

If a file has changed since the team ran, for example the menu guide got updated, Evy still grades against the version the team had, because that's what the team could see. It tells you what changed.

If you get stuck

What you notice	What to do
Evy asks which rule sheet to use	Say "the newest one I approved for this team."
Evy says a case has nothing to grade	That case's files are missing. Ask your Chief of Staff, in its thread, which of last week's jobs is that case, and tell Evy.
Evy is taking a long time	Leave it running. Evy says "done" when every case is graded.
A grade has no quote	Ask Evy, in its thread, for the exact words it graded. A grade without a quote can't be checked.
A grade says "not verified"	That's honest, not a failure. Evy couldn't check something and said so instead of guessing.
Evy offers to fix the work	Say no. Evy grades; fixing comes later this week, with your yes.
Evy disagrees with one of your grades	Note it, but don't argue with Evy yet. Comparing is the next lesson.
Evy says your grades aren't sealed	Go back to "Turn what you found into the rules for this team" and ask your Chief of Staff to seal them first. For these practice cases, a grade Evy gives before your seal can't be compared. (Later, for the jobs picked for you to grade, Evy grades first and its answer stays hidden until your seal.)

What Evy reads, the tools Evy holds, and where each grade is saved: more detail on this lesson.

What you'll keep

Evy's grade for every sealed case, each one with a number side and a story side, saved beside the work it judges.

Before you finish

Before you finishThree checks

Evy graded every sealed case, without seeing my gradesI read one fail (or, if Evy failed nothing, one pass) all the way down, and asked the three checksI found a step Evy redid to check its grade, or Evy told me why it didn't need to

Self-assessed. No gate.

Words you now own

Evy: the Evaluation Analyst, the agent that grades finished work against your rules.
Number side: a pass or fail for each rule, at the agent level and the job level.
Story side: what Evy looked at, why, which agent and how Evy found out, the fix, how sure, and what it couldn't check.
Not verified: something Evy couldn't check, said plainly instead of guessed.

Next: Compare your grades with Evy's and teach Evy.