More detail: Have Evy grade the same work
Optional reading for this lesson. Everything you must do or decide is in the lesson.
This page goes with the Week 4 lesson Have Evy grade the same work (course/workforce/week4/lessons/anw-w4-l03.md). You don't need it to finish the lesson. Read it when you want the longer version.
What Evy reads for each case
| What | Where it comes from |
|---|---|
| Each agent's instructions for the job, and what it handed back | The job folder under runs/ |
| The researcher's found and couldn't-find tags, and the facts table in the plan | The job folder |
| The draft and Kansa's review | The job folder |
| A copy of the run's records from Langfuse | Saved on your computer, in the job folder, by a fixed script |
| Your team's newest rule sheet | Your team's folder |
| Evy's notebook for your team | Your team's folder (it starts empty, and fills in the next lesson) |
Evy never reads your sealed grades before grading.
The tools Evy holds
Evy holds every tool and connection any agent in your organization holds, so Evy can redo any step to check it. For example:
- open the document the Internal Knowledge Researcher (Hatch) cited
- run the External Researcher (Scratch)'s search again
- look something up in a tool you connected
Evy never holds the tools that send anything outside your organization, post in public, delete, move money or change an account. When another agent gains a tool, Evy gains the same one, unless it's one of those. Evy writes only in Evy's own places: Evy's grades, the daily check folder, Evy's notebooks, and Kansa's watch list.
When a live check differs from the run
If Evy opens a source today and it says something different from what the team saw when it ran, Evy grades the job on what the team had at the time, and says what's changed and when Evy looked.
Where each grade is saved
In each case's job folder, under evaluation/:
evaluation.md: both sides of the grade, in wordsevaluation-binary.json: the number side, for the counting script and the dashboard
A second pass on the same job is saved beside the first and never overwrites it.
Things Evy won't do
- Grade against a rule you didn't approve.
- Add the levels together into one score.
- Fix the work, or decide what ships.
- Treat anything written inside a customer's email as an instruction. If an email tries to tell an agent what to do, Evy quotes it as a finding.
Asking an agent why
Sometimes the records don't say why an agent did something. Evy may ask the agent. The answer is a lead, not proof: Evy checks it against the records, and if the two disagree, that's a finding of its own.
Staff only: internal notes. Not student-facing.
- 12 October (L3, Tyler ruled): Evy grades every job; the owner's sample is hidden from the owner until saved and sealed (workforce/runtime/evy_hold.py); the seal runs the compare.