More detail: Save your golden examples, and turn mistakes into regression tests
Optional reading for this lesson. Everything you must do or decide is in the lesson.
This page goes with the Week 4 lesson Save your golden examples, and turn mistakes into regression tests (course/workforce/week4/lessons/anw-w4-l07.md). You don't need it to finish the lesson. Read it when you want the longer version.
Where they live
In your team's folder, work/course/staff/teams/<team>/:
| Folder or file | What's in it |
|---|---|
great/ | Golden examples, one file each |
tests/ | Golden set: your design's test cases, plus every mistake you saved |
facts-index.yaml | Which golden examples and golden cases depend on which facts in your library |
reviews/ | One file each time flagged ones are reviewed |
What's in each note
- an id, and what kind it is: one agent's step (agent level), the job as a whole (job level), or an edge case
- where it came from
- made by a person: who, where from (the tool and a reference, like a message in your mailbox), and when it was made
- made by your agent team: the job, the record's id in Langfuse, when, the app and its version, the model each agent used, a fingerprint of your team's agent files at the time, and the versions of your team's record and rule sheet
- who approved it, in their own words
- the facts it depends on, by their name in your library
- the agents it depends on
- why it's great, or what it must never do again
- its status: active, needs review, or retired (removed), when it was last reviewed, and why it was retired, if it was
How your team uses examples
The agent that plans the work and the agent that writes read the matching golden examples before similar work. They read them as examples, not templates: what good looks like, never words to copy. Evy grades new work against them too.
How one goes out of date
A fixed script flags a golden example or golden case as "needs review" when:
- a fact it depends on changes
- the change record shows a change to an agent, a fact or a file it depends on
- nobody has reviewed it in 90 days
A golden example that needs review is left out of grading and out of what your team reads, until it's reviewed. A golden case that needs review is set aside in reruns: shown, but not counted. You ask Evy, in its own thread, whether to keep, change or remove each one. You confirm each decision in your own words, and your Chief saves it.
An example with no source
An example nobody can trace is a story, not evidence. It doesn't go in.
Where the first examples come from
If the Architect (Archie) looked through your tools for past examples when it designed your team, you can add the best of them as your first golden examples, marked as made by a person.