More detail: Compare your grades with Evy's and teach Evy
Optional reading for this lesson. Everything you must do or decide is in the lesson.
This page goes with the Week 4 lesson Compare your grades with Evy's and teach Evy (course/workforce/week4/lessons/anw-w4-l04.md). You don't need it to finish the lesson. Read it when you want the longer version.
How agreement is counted
A fixed script reads your sealed grade files and Evy's grades and counts how often they match, over the rules you both graded. A rule you left blank isn't counted at all. It counts:
- for the whole job
- for each agent
- for each rule
- once more for only the cases you failed
It saves the result as agreement.json in your team's folder. Evy shows you the result; Evy never works out the number. The dashboard reads the same file.
Why the failed cases get their own count
If your team passes almost everything, you and Evy can agree on almost everything just because Evy passes almost everything. The count on the cases you failed shows whether Evy catches what you catch. If you failed none yet, the count says it can't tell.
The three kinds of disagreement
| Kind | What it means | What happens |
|---|---|---|
| My rule was unclear | Someone else, reading your rule, could fairly grade it Evy's way | Gigawatt sharpens the rule with you, and the Professor saves the new version on your yes |
| Evy was wrong | Your rule was clear, and Evy misread the work or the rule | It goes in Evy's notebook for this team, with your reason |
| Evy caught something I missed | Evy's right and you'd grade it that way now | You say so. Your sealed grade stays, and it goes in Evy's notebook |
Evy's notebook
evy-notebook.md, one per team, in your team's folder. It holds every case you and Evy both graded, agreements as well as disagreements, each with both grades and your one-line reason. Later, corrections you make while working with your Chief go in it too. Evy reads it before grading that team.
Evy adds examples to Evy's notebook without asking. Evy never changes your rules: Evy suggests a change, and you decide.
Checking less over time
| Step | You check by hand |
|---|---|
| Start | Every case |
| Agreement holds | Half |
| Holds longer | One in five |
| Holds longer still | Only what Evy marks as unsure, or as touching a serious rule |
The thresholds are fixed in the script, so nobody argues you into checking less. Any disagreement on a serious rule, like a made-up allergen fact, steps you back up.
Why agreements count too
Every case where you and Evy agree tells Evy what you count as good. That's as useful to Evy as a correction.
What the Evidence Pattern Analyst (Mitch Please) reads, and what it never does
Mitch works from an order your Chief of Staff writes: your question, the team, and the grade files for each case. It reads each file in full, gives each one a short id (R1, R2 and so on), and counts every pattern out of that total. Its notes go in the job's own folder, beside the order.
- Observed: the grades show it. You get the count and a quote from each grade that shows it.
- Inferred: a possible reason. You get how sure Mitch is (low, medium or high), another way to explain it, and the one question or record that would settle it.
- Unknown: something missing that could change the answer.
If a grade looks wrong to Mitch, it says so as its own finding and still counts the grade as Evy gave it. It never fixes the work, regrades anything, changes a rule or picks what to fix. Nothing it finds becomes something your organization relies on until you confirm it.