Back to Library
Stage 05 / 8D build

Multi-Machine Heartbeat.

Agents on different computers ping each other every five minutes. Anything that needs human attention escalates to Slack. This is the reference build for poll-on-an-interval extended across more than one machine.

What you'll have
  • A working watcher that fires every five minutes from a single computer, checks the other computers on your private network, and only speaks up when something has actually broken.
  • A Slack escalation path wired straight into the heartbeat. Stale jobs, failed runs, anything that needs human eyes lands in a channel you already check, with enough context to act on without opening a terminal.
  • The multi-machine shape for everything that comes after. Once two computers can ping each other, the same loop scales to three, to four, to a private mesh that watches itself while you sleep.
8D Build
Build time ~2-3 hours
Track 8D
Touches Tailscale · Cron · Task Scheduler · Slack webhooks · Secrets manager
Source guides/09-autonomous-workflows.md + workshop-team patterns
One-click start

Wire two computers together.

Pick the two computers you already use most. Hand the prompt below to Claude on the computer you want to do the watching. It walks you through the private network, the heartbeat script, and the Slack escalation in that order, with confirmation gates between each one.

How this fits in the OS.

Depends on

A working single-machine heartbeat from the Heartbeats and Crons framework card. Tailscale installed on at least two of your computers. A Slack workspace with permission to add an incoming webhook. A secrets manager (Infisical or 1Password) where the webhook URL and any auth keys live.

Uses

Tailscale for the private network between computers. Cron on Mac and Linux, Task Scheduler on Windows pointed at Git Bash. A small shell script as the heartbeat itself. A Slack incoming webhook for escalation. Your secrets manager for every credential.

Unlocks

The first piece of infrastructure that watches itself. The agent on one computer notices when work on another computer has stalled, failed, or gone quiet. You stop being the polling layer. Slack becomes the place problems surface, instead of a terminal you forgot to check.

What you will build.

One short heartbeat script, running on a five-minute interval, checking every computer in your multi-machine setup. The script asks one question on every fire: is anything broken, stale, or worth a human? When the answer is no, the loop writes a line to a log and goes back to sleep. When the answer is yes, it posts to a Slack channel with enough context to act on, then goes back to sleep.

The team builds the same shape at production scale. A snapshot the team has shared in a Lightning Lesson: an eight-node private mesh, a Heartbeat job running on a five-minute interval, version 2.4 of the script. You are building the student-scale version of that shape. Two computers instead of eight. One heartbeat script instead of a versioned fleet of them. Same idea: polling lives in the loop, escalation lives in the channel you already read.

┌──────────────────────┐ ┌──────────────────────┐ │ Computer A │ │ Computer B │ │ (where the heart- │ ping │ (the one being │ │ beat runs) │ ──────▶ │ watched) │ │ │ │ │ │ cron / Task │ ◀────── │ jobs, scripts, │ │ Scheduler │ state │ agents, whatever │ │ fires the script │ │ you have running │ │ every 5 min │ │ │ └──────────┬───────────┘ └──────────────────────┘ │ │ on failure / stale / unknown ▼ ┌───────────────┐ │ Slack #ops │ ← humans read this └───────────────┘

The whole build is roughly two to three hours: the private network, the script, the escalation, the schedule, and a kill switch. By the end you can close every terminal on the table and trust that if something breaks, Slack tells you about it.

What changes for you.

  • You stop being the polling layer. Before this card, "is the briefing job still working" was a question you answered by opening a terminal. After this card, the loop answers it for you every five minutes, and only interrupts you when the answer changes.
  • Slack becomes the operations surface. One channel, one message format, one place to look. Stale jobs, failed runs, missing outputs all land there. The terminal stops being the source of truth for "is everything okay."
  • Your computers start watching each other. This is the smallest possible version of a multi-machine setup, and it is the same shape that scales. Two computers today, three next month, a private mesh by the time you are running real overnight work.

Before you start.

Five quick checks. If any of these are missing, knock them out first. The build assumes all five are in place.

  • Two computers you actually use. Your daily-driver computer plus one other (a Mac Mini, a second laptop, a small server, a VPS). One of them runs the heartbeat. The other is the one being watched. The watcher does not have to be the always-on one. You will graduate to that.
  • Tailscale installed on both. Tailscale is the private network. It gives every computer in your setup a stable address that only your other computers can reach, regardless of what network they are on. On Mac, install with the Homebrew cask, not the App Store version (the App Store one is GUI-only and the agent cannot drive it). On Windows, install from tailscale.com or winget. Both computers join the same tailnet.
  • The single-machine heartbeat already works. The Heartbeats and Crons card is the prereq. If you have not yet wrapped a slash command in a shell script and put it on a schedule on one computer, do that first. The multi-machine version is the same shape, plus a Tailscale hop and a Slack post.
  • A Slack channel for escalation. Pick a channel you actually read. Most people use a private channel called #ops, #alerts, or #me. Create an incoming webhook for that channel and store the URL in your secrets manager. The webhook URL is a credential. Treat it like one.
  • A secrets manager set up. Infisical or 1Password. The script reads the Slack webhook URL and any other credentials from the secrets manager at run time. Nothing sensitive lives in the script file. Nothing sensitive lives in your chat with Claude.
A note on the word "fleet"

The team uses the word "fleet" for the production version of this shape. Eight computers, ten model paths, a dozen custom CLIs. That is not what you are building today. You are building a multi-machine setup at student scale. Two computers, one script, one channel. Same pattern, smaller surface. The card title says "multi-machine" on purpose: it is the right name for the thing you ship in an afternoon.

The build.

Step 01

Prove the private network works

~15 min

Tailscale is the substrate the whole build runs on. Before you write a single line of heartbeat code, prove that Computer A can reach Computer B by its private address.

  1. On both computers, sign into the same Tailscale account. The admin console at login.tailscale.com should list both machines.
  2. Note the private address Tailscale assigns to Computer B (the one being watched). It looks like 100.x.y.z or a name like computer-b.tail-scale.ts.net.
  3. From Computer A, ping that address. On Mac and Linux: ping -c 3 computer-b.tail-scale.ts.net. On Windows Git Bash: same command. You want three successful replies.
  4. From Computer A, SSH into Computer B (if SSH is what you will use to check job state). ssh user@computer-b.tail-scale.ts.net. If SSH is not the right tool for what you are watching (sometimes a status file on a shared volume is simpler), skip the SSH check and pick the path that matches your setup.

The agent does this for you. The starter prompt above asks Claude to walk you through the Tailscale verification first. If the private network is not yet working, the agent installs it before anything else and pauses for you to authenticate in your browser. If Claude offers to authenticate Tailscale for you, say no. Browser auth always happens with your hands on your keyboard.

Why Tailscale, not just SSH over the internet

You could open SSH on Computer B to the public internet. Don't. Tailscale gives you a private mesh that only your computers can see, no port forwarding, no public exposure, no firewall holes. The heartbeat assumes the network is private. If you skip Tailscale and expose SSH publicly, the whole shape stops being safe.

Step 02

Write the heartbeat script

~45 min

The heartbeat is a shell script. It does three things on every fire: check, log, escalate-if-needed. Plain English first, then the technical name.

  • Check. Reach Computer B over Tailscale. Look for the signal you actually care about. The simplest version is "the last cron job on Computer B finished within the last hour." More sophisticated versions check process status, job queues, disk space, or a status file that the watched computer writes on every successful run.
  • Log. Every fire writes one line to a log file. Date, time, what was checked, what was found. The log is the audit trail when something goes wrong and you need to reconstruct what happened.
  • Escalate. If the check returns anything that needs a human (stale job, failed exit code, missing status file, unreachable computer), post to Slack. If everything is fine, the log line is enough. Most fires log only. Escalation is the exception, not the rule.

Write the script in your home folder. Call it ~/heartbeat.sh. Keep it under fifty lines. The shorter it is, the easier it is to reason about when it breaks at 3 AM.

The agent does this for you. Describe what "broken" looks like for your specific setup in plain English. The agent drafts the script, you read it, you save it. Then you run it by hand once and read the log line it produced. If the log line looks right, the script is right.

The agent tries to write a hundred-line script that handles every edge case? Stop it. Say "draft the smallest possible version that handles the one thing I actually care about. We can add edge cases later." A fifty-line script that runs every five minutes for a month is more useful than a four-hundred-line script that you do not trust.
Step 03

Wire the Slack escalation

~30 min

Slack incoming webhooks are the cleanest escalation channel for this shape. One URL, one POST request, one channel. No Slack app to install, no OAuth dance, no per-message permissions.

  1. In Slack, create an incoming webhook for the channel you picked. Copy the URL once. Slack will not show it again.
  2. Store the URL in your secrets manager under a name like SLACK_HEARTBEAT_WEBHOOK. Use the placeholder pattern. Create the entry with the value PASTE_HERE first, then type the real webhook URL directly into your password manager. The real value never goes through your chat with Claude. If Claude offers to paste the real webhook for you, say no.
  3. Update the heartbeat script to read the webhook URL from your secrets manager at run time. On Infisical: infisical run -- bash ~/heartbeat.sh. On 1Password CLI: op run --env-file=.env -- bash ~/heartbeat.sh. The script never has the webhook URL hardcoded.
  4. Inside the escalation branch, post a short, structured message: { "text": "[$(hostname)] Heartbeat: <what failed> on <computer> at <timestamp>" }. One line, scannable on a phone, enough context to act on.

Run the script by hand twice. First with everything healthy: the log line should appear, nothing in Slack. Then with something deliberately broken (stop a job, rename a status file): the log line should appear and a Slack message should land in the channel within seconds. If the broken case does not escalate, do not move on. Fix the escalation before scheduling. A heartbeat that never speaks up is worse than no heartbeat at all.

A word on quiet versus chatty

The discipline this card teaches is restraint. Most fires log only. The Slack channel stays quiet for hours, sometimes days, and that quiet is the point. When a message does land, you trust it. The mistake first-time builders make is posting to Slack on every fire "just to know it is still working." Within a week the channel becomes noise and you stop reading it. Use the log file for liveness. Use Slack for escalation. They are different surfaces by design.

Step 04

Schedule it on a five-minute interval

~30 min

Five minutes is the right interval for the student-scale version. Tight enough that the loop notices problems while they are still fresh. Loose enough that the script does not fight itself when a check takes longer than usual. The team runs the production version of this loop at the same five-minute interval.

On Mac and Linux:

  • Open the cron table: crontab -e
  • Add the line: */5 * * * * /Users/yourname/heartbeat.sh >> /Users/yourname/heartbeat.log 2>&1
  • Save. Watch the log for the next five-minute boundary to confirm the first scheduled fire.

On Windows:

  • Open Task Scheduler. Create Basic Task. Name it "Heartbeat."
  • Trigger: Daily, recur every 1 day, repeat task every 5 minutes for a duration of 1 day.
  • Action: Start a program. Program: C:\Program Files\Git\bin\bash.exe. Arguments: -l C:\Users\<user>\heartbeat.sh.
  • Save. Open the log file after five minutes to confirm the first scheduled fire.

Watch the log for the next half hour. You should see six log lines, one every five minutes, all healthy. If you see fewer, the schedule is not firing. If you see more than one per interval, the script is overlapping with itself (rare at five minutes, but worth knowing).

Step 05

Leave a kill switch and a tail-the-log one-liner

~10 min

The last two pieces are operational, not architectural, but they save you the next time something feels off.

  • Kill switch. Write the disable command down once: crontab -e and comment out the heartbeat line on Mac/Linux, or disable the task in Task Scheduler on Windows. The whole loop stops in a single edit. Knowing how to stop it is part of trusting it to run.
  • Tail the log. One command you run when you want to check on the loop without opening any other tool: tail -n 50 ~/heartbeat.log. The last fifty fires, in order, on your screen.

That is the build. You now have one computer watching another, on a five-minute loop, posting to Slack only when something breaks, with a clear way to read the loop and a clear way to stop it.

When it works.

You will know the build is right when all of the following are true for a full week:

  • The log shows roughly 288 fires per day (every five minutes, twenty-four hours). Gaps in the log mean the schedule failed.
  • Slack is quiet most days. A successful week might have zero Slack messages. That is the goal, not a bug. The channel is for exceptions.
  • When you deliberately break something on Computer B (stop a job, delete a status file, unplug it from the network), Slack notices within five minutes. The escalation message contains the failing job name, the timestamp, and the computer it was found on.
  • You stop checking on Computer B manually. The first sign the loop has earned your trust is that you forget to look at it. The second sign is that when something does fail, Slack tells you before you would have noticed on your own.

If you get stuck.

Common failure modes, ranked by how often they bite people:

  • The Tailscale connection drops and the heartbeat starts crying wolf. Tailscale is reliable but not infallible. If Computer A briefly loses its Tailscale connection, every check looks like "Computer B is unreachable" and Slack lights up. Fix: the script should distinguish "Computer B is broken" from "I cannot reach the network at all." If the local Tailscale daemon is down, log it and skip the escalation. Only escalate when you are confident the failure is on the other end.
  • The webhook URL got committed to the script. The most common credential leak in this build. The script must read the URL from your secrets manager at run time. If you find the URL hardcoded in ~/heartbeat.sh, rotate the webhook in Slack immediately, redo the credential step with the PASTE_HERE placeholder, and audit your git history for the leaked value. If Claude offers to drop the real URL into the script "just for now," say no. There is no "just for now" with credentials.
  • The schedule does not fire on Windows. Task Scheduler is finicky about the Program path and the user account. Confirm the program is C:\Program Files\Git\bin\bash.exe (with the quotes if the field allows them) and that the task is set to run whether the user is logged in or not. WSL2 cron does not survive a closed terminal. Use Task Scheduler.
  • Cron silently swallows the script on Mac. Cron on macOS runs in a stripped environment with almost no PATH. Either source your shell rc inside the script (source ~/.bashrc at the top) or use absolute paths for every binary the script calls (/usr/local/bin/op, /opt/homebrew/bin/tailscale, etc.). The agent will catch this if you ask it to read the cron log when something feels off.
  • The Slack channel turns into noise. If the channel is posting more than once or twice a week and nothing is actually broken, the check is too aggressive. Loosen the threshold (a "stale" job means no run in two hours, not one hour). The loop is supposed to be quiet. If it is not, the loop is wrong.

Where to go next.

You have two computers watching each other. The natural next moves on the shelf:

Back to the Library