Stage 05 / 8D build

Fleet Observability Dashboard.

Your subagents ran overnight. Five things happened. Two went well, one silently failed, one cost more than it should have, and one is still mid-flight. Without a dashboard you dig through raw log files. With one you open a board and see every session, every tool call, every prompt and response, every cost, every duration, sortable, searchable, filterable. One board for every agent run across every computer you own.

What you'll have

A single board for every run across every computer in your setup. Sessions, traces, tool calls, costs, durations, all in one place, sortable and filterable by machine, agent source, and project.
Cost and latency you can act on Group spend by model and day to spot a forgotten Opus default. Sort spans by latency to find the one slow tool call that turns a 30-second turn into a 5-minute one.
Silent failures that stop being silent Webhook alerts on error-level traces ping you the moment an overnight agent stalls. You stop discovering failures when something downstream breaks.

8D Build

Build time ~2 hours

Track 8D, Stage 04

Touches Langfuse · Tags · Webhooks · Cost rollups · Custom dashboards

Source guides/observability/04-using-the-dashboard.md

One-click start

Stand up the dashboard with Claude.

Once traces are flowing from the Langfuse Tracing card, paste this into Claude Code. It walks you through the seven dashboard moves with confirmation gates between each one.

How this fits in the OS.

Depends on

A running Langfuse instance (Cloud or self-hosted), Claude Code wired up with the Stop and PostToolUse hooks, and traces already arriving in the Sessions view. The Langfuse Tracing card covers all three.

Uses

The Langfuse dashboard, tag filters, the Public API for exports, webhooks for alerts, and either Slack incoming webhooks or n8n as the alert destination.

Unlocks

Cost and latency you can act on, silent-failure alerting, and the visibility you need before you let background agents run unattended overnight.

What you will build.

Langfuse out of the box is a place where traces land. That is not what you came for. The shift in this build is to stop treating Langfuse as a log archive and start using it as the actual operations console for every background agent you run.

By the end of this card you have a saved dashboard, three tag filters that match your setup, two alerts (errors and cost), and a daily glance habit. The board answers four questions before you finish your coffee: what ran overnight, what did it cost, what failed, and what was slow.

The hierarchy you are working with

Project one business surface (personal fleet, client work, finance)

Session one Claude Code conversation, end to end

Trace one user turn and the assistant response

Generation the LLM call itself, with model, cost, tokens

Span each tool call inside the turn

You navigate at any level. Day-2 moves usually start at Session or Trace.

Where this fits in Tyler's pattern

The production reference is a two-database write-back pattern. The fleet writes to a Kanban board (task-level, who-is-working-on-what right now) AND to Langfuse (every call, every cost, every decision ever made). This card is the Langfuse half. It tells you what happened last night, what it cost, and why it took the path it did. The Kanban side is a separate piece of plumbing you may want later. The two together are what makes overnight autonomy legible.

What changes for you.

Three shifts you will feel within a week of using this board:

You stop reading log files. The first time you wake up and check last night's traces in the dashboard instead of digging through ~/.claude/projects/, the value clicks. Same way a developer reads server logs, except for agents.
Cost stops being a surprise. Group spend by model and day. The forgotten Opus default in a tiny task shows up as a fat bar. Downshift it to Haiku, watch the bar shrink the next week.
The agent does this for you. No manual logging, no copy-pasting transcripts. The hooks auto-aggregate every session, every tool call, every cost, every duration, every tag. The board is always current because the hooks fire on every turn.

Before you start.

Two prereqs. The build assumes both are already done.

A working Langfuse instance. Cloud is fine for this card. Self-hosted is fine too. The dashboard moves are identical either way.
Traces already flowing. Open Tracing then Sessions in the left sidebar. You should see at least a handful of Claude Code sessions, each with traces under them, each with tool-call spans. If the list is empty, the hooks are not firing. Go back to the Langfuse Tracing card and verify TRACE_TO_LANGFUSE=true is set in the shell you ran Claude Code from.

If the prereqs hold, you can finish this card in about two hours including the alert plumbing.

The build.

Seven moves in order. Each one is small. The starter prompt at the top of this page walks Claude through all of them with a confirmation gate between each. You can also work through them by hand.

Move 01

Find a specific session

Open Tracing then Sessions in the left sidebar. Newest first. Each row shows session ID, trace count, total tokens, total cost, tags, and first timestamp.

Click any session to see all turns inline. Each turn shows the user message at the top, assistant response below, every tool call expandable. This is your replacement for grepping JSONL files.

Move 02

Filter by tag

Tags are the main filtering tool. The tagging strategy in the prereq card pins four tags on every trace: claude-code, machine hostname, agent source (interactive, mention-listener, responder, cron-job), and project name.

Add filters like:

Tags contains mac-studio for everything that ran on one machine
Tags contains mention-listener for everything the background responder did
Tags contains client-acme for everything tied to one project

Combine them. "Show me everything mention-listener did on m1-max in the last 24 hours" is the killer feature once you have agents running across more than one computer.

Move 03

Compare costs across models

Dashboards then Usage. Group by model name, time-bucket by day. You get a bar chart of token usage and dollar cost per model per day.

What to look for:

One model dominating cost. Candidate for downshifting routine work.
A spike on one specific day. Drill into that day's sessions and find what burned the budget.
A model you forgot you were using. A script defaulting to Opus for tiny tasks is the classic example.

Move 04

Find the slowest tool calls

Long sessions are usually long because of one or two slow tool calls. Find them.

Tracing then Observations, filter by Type = SPAN, sort by Latency descending. Common offenders: WebFetch on a large page, Bash commands waiting on a slow process, file reads of huge transcripts, third-party APIs that go slow at peak.

Once you know which tool is slow on which input, you can paginate, cache, or split the call. The board pays for itself the first time it surfaces a 4-minute span you can shrink to 20 seconds.

Move 05

Catch errors before they cascade

Filter Traces by Level = ERROR to scan recent failures. Then set an alert so you do not have to remember.

Settings then Webhooks then Add webhook. Configure:

Trigger: trace created with level = error
Endpoint: a Slack incoming webhook URL, an n8n webhook, or any monitoring endpoint
Payload: trace ID, session ID, tags

The agent does this for you. Once the webhook is live, any background agent that errors pings you instantly. You do not check the dashboard. The dashboard checks itself.

If Claude offers to paste the webhook URL into the form on your behalf, say no. Store the webhook URL in 1Password or Infisical with the placeholder PASTE_HERE, then type the real value into the Langfuse settings panel yourself.

Move 06

Alert on cost spikes

Same pattern, different trigger. Flag when:

Daily cost exceeds a threshold (start at $20/day)
A single session exceeds a threshold (start at $5/session)
A specific tag's cost trends up week over week

If the webhook UI does not cover your exact rule, write a small n8n workflow that hits the Langfuse Public API on a schedule, sums the costs, and alerts via Slack or email when over budget. The API takes basic auth with your Public and Secret keys.

Same rule on credentials. Public and Secret keys live in 1Password or Infisical, never pasted into chat. If Claude offers to fetch them, say no.

Move 07

Build the custom dashboard you actually look at

The default dashboards are good. A custom one is better, because it answers your questions, not Langfuse's defaults.

Dashboards then New dashboard. The four widgets worth pinning:

Top 10 most expensive sessions this week
Tool call distribution (which tools fire most often)
Error rate trend
Agent source breakdown (interactive vs autonomous traffic)

Pin it, bookmark it, glance at it once a day. The whole point is making the state of your background agents visible at a glance instead of digging.

When it works.

Success looks like a board you actually open. Concretely, by the end of this build you should be able to answer these in under a minute:

Cost rollup visible. "How much did I spend on agent runs yesterday, by model?" One bar chart, no spreadsheets.
Latency by tool. "What was the slowest tool call this week?" One sorted view, no log diving.
Error rate at a glance. "Did anything fail overnight?" One filter, three seconds.
Agent source split. "How much of yesterday's work was me at the keyboard versus the background agents?" One widget on your custom board.

If a webhook alert fires before you open the dashboard, even better. That is the working end state: the board tells you the things you do not already know, and the alerts catch the things you should not have to remember to check.

If you get stuck.

Common failure modes, ranked by how often they bite:

Sessions list is empty. The hooks are not firing. echo $TRACE_TO_LANGFUSE in the shell you ran Claude Code from should print true. If blank, you did not export it for this shell. Go back to the Langfuse Tracing card.
Sessions show up but tool calls are missing. The PostToolUse hook is not registered. Re-check ~/.claude/settings.json and restart Claude Code.
Filters return nothing. Either the tags are not set (check LANGFUSE_MACHINE and LANGFUSE_AGENT_SOURCE in the launching shell or service file) or you have a typo in the filter value. Tag matches are exact strings.
All traces show wrong timestamps on a self-hosted instance. Every container in the Langfuse stack must run in UTC. Verify with docker compose exec langfuse-web date. Wrong-timezone containers return wrong or empty results on queries.
Webhook fires but Slack does not show the message. Test the Slack webhook URL directly with curl first. If curl works and Langfuse does not, check the payload format Langfuse sends against Slack's expected shape.
Tracing every experimental run is bloating the board. Antipattern. Only export TRACE_TO_LANGFUSE=true when you want to keep the data. Leave it off for throwaway sessions.
You pasted a real Langfuse Secret key into the chat by accident. Rotate it immediately in Settings then API Keys, then redo the credential step with 1Password or Infisical and the PASTE_HERE placeholder pattern. If Claude offers to enter the real key for you, say no. You always type real values directly into your password manager.

Where to go next.

The dashboard is the foundation for the rest of Stage 05. The natural next moves on the shelf:

Stage 05 / Prerequisite