Read your team's work and give each case a pass or fail

Two things this week starts from

1. Your team is on the record. The record is one short file that names the agents on a team. If you designed a team last week, it's already there. If you only finished your email team, it isn't yet. You don't need a design for it: your Chief asks the Charter Steward to register the team you already have, using the agents already set up in your workbench. The Steward writes down your yes word for word and files the record.
2. Your cases, with your notes. Gigawatt's notes from each run start with what you expected, in your words.

If you didn't set up Langfuse, the recorder from Week 2, that's fine for now. Your team's work is also saved in its job folders, and that's what gets graded. You set the recorder up later this week.

Name each email job folder in this exact pattern: the date, then the word support, then a few words about the task, in lowercase with hyphens. For example: 2026-10-12-support-kids-plate-question. Never use the customer's name. Kansa's check only accepts this pattern, and folder names show up in grades, the dashboard and the recorder.

Do this now: ask your Chief

In Claude Code: start a new thread, type @ and pick aibl-chief-of-staff. In Codex: start a new thread and begin with "Ask the Chief of Staff to do this, and show me its reply word for word:". Use this thread for every step in this lesson. Say:

Say yes to each step it offers, in your own words.

What you should see

Your Chief lists your cases and the notes for each, and says whether your team is on the record.

• You designed a team with Archie: it's already on the record. Nothing to fix.
• You only finished your email team: your Chief finds your three practice emails and your notes, and tells you the email team isn't on the record yet. It offers to put it on the record as the team you already have, no design needed, with a name like email-team. Say yes in your own words, for example: "Yes, put my email team on the record as email-team." The Charter Steward files the record. It also saves your three practice emails as your team's first golden cases, each with what your team actually did. The fake bank email is saved as one your team must refuse. You use them in the lesson "Make the fix, rerun the golden set, and keep or undo it".

A pass or a fail, with your reason

A grade here is a pass or a fail, and your reason. There's no middle grade, no score out of ten and no "mostly fine".

Why only two? A middle grade lets you put off deciding. "Mostly fine" doesn't tell you whether you'd send it, or what to fix. A pass or a fail does.

Two grades also make the comparison later this week fair. The Evaluation Analyst (Evy) grades the same work, and two people agree far more often on pass or fail than on a 7 or an 8.

If you can't say exactly what's wrong, look again.

Two levels: the job, and each agent

Every case is graded at two levels:

• Job level: the outcome, what your team handed back. On the email team, that's the email reply.
• Agent level: each agent's own piece of the work. On the email team, that's the Support Lead's plan and evidence package, the researcher's findings, the writer's draft and the reviewer's check.

The two are never added up into one grade.

Knowing both tells you where to fix. An agent can fail while the job passes: another agent caught the mistake. And every agent can pass while the job fails: each did its own piece, and the team still missed.

We show you: the Hattie B's practice inbox

Before you grade your own cases, here's a finished example, so you can see what a full set of grades looks like. These are the example student's grades for five Hattie B's practice emails:

• the three you ran last week: the kids plate question, the catering quote and the fake bank email
• two more from the same pack: Marcus, and the cold-food complaint

The agents are the four in the diagram above.

Marcus is the case to notice: an agent failed, and the job still passed.

Do this now: ask your Chief for your cases

In the same Chief thread, say:

What you should see

A short list: each case, its folder under runs/, and the notes file beside it. If a case has no notes, it's still a case; grade it from what you remember expecting, and say so in your reason.

How to decide

Ask one question: would I send this, or use it, as it is?

• Yes: it's a pass.
• No, I'd have to change something first: it's a fail.

Your notes from last week are a starting point, not the rule. If the team did it differently from what you expected, and you'd still send it as it is, it passes.

Sometimes the right result is no draft at all. On the fake bank email, the team should hand back nothing but a flag to you. If that's what it did, it passes.

A good reason: what's wrong, and what a pass looks like

For a fail, your reason does two things:

1. Names everything that's wrong, exactly. Point at the words in the reply. If there are two problems, name both.
2. Says what a pass would look like. What would the reply have to say, or not say, for you to send it?

Give as much detail as you need. Someone else should be able to read your reason, find the problem and know what to fix.

For a pass, say what made it good enough to send. That tells your team what to keep doing.

We show you: the Hattie B's practice inbox

For the kids plate question, the student had expected "a kids meal answer, and an honest 'I'm not sure' about what a vegetarian can eat." The draft said the fries were vegetarian, and nothing in the menu guide says so. The student wouldn't send it as it is, so it's a fail. The reason: the reply says the fries are vegetarian, and the menu guide doesn't say that. A pass answers the kids meal question and leaves the vegetarian question blank for me.

Do this now: grade each case

Take the cases one at a time. For each one, in the same Chief thread, say:

Read both. Then give your grade in the same thread, in your own words:

Grade at least three cases. Grade more if you have them. If you're unsure, ask The Professor, in its thread, to read the two side by side with you. It won't grade for you.

What you should see

Your Chief repeats your grade back exactly as you wrote it, then waits for the next case. Nothing is changed and nothing is graded for you.

How to judge an agent

Judge each agent on its own piece, not on how the job turned out. Ask three questions:

1. What was it given?
2. What was it asked to do?
3. Did it do that?

Two ways an agent fails

Two cases that catch people out:

• A later agent fixed it. Still a fail. On Marcus, the Support Lead missed Valentine's Day, even though the writer caught it.
• Every agent did its piece, and the job still failed. Fail the job, not the agents. It's a gap between them, and it becomes a job-level rule in the next lesson.

We show you: the kids plate question, agent by agent

The Support Lead does two pieces, the plan and the evidence package, and gets one grade for both. If you built your own team, its agents and order are different; your Chief follows them.

Do this now: follow each fail

For each case you failed, say in the same Chief thread:

Ask the three questions at each agent. Then grade it in the same thread:

Grade agents on cases that passed too, if you notice something, like the Support Lead on Marcus. Next lesson, these grades become the rules for each agent.

What you should see

Your Chief shows one agent at a time and waits. It repeats each grade back exactly as you wrote it, and never tells you which agent went wrong.

We show you: the Hattie B's practice inbox

Here's part of the example student's graded set: two of the five cases.

Reading the full list back, the student found one reason that just said "bad". They changed it to "offers a refund, which is my call; a pass says sorry and promises nothing" before saying yes.

Do this now: save your graded set

In the same Chief thread, say:

Before you say yes, check:

Fix anything that isn't what you meant, in your own words. Then say yes.

What you should see

Your list, word for word, and then a line saying it's saved with your team's records. In the next lesson, your grades are locked so they can't change, before Evy grades the same work.