Back to Library
Stage 04 / 8D build

YouTube Cairns Blueprint.

Watch 40 channels at once. Every new video gets transcribed, tagged, embedded, and dropped into your knowledge vault overnight. The 8D build that turns a list of YouTube channels into a source-grounded second brain, with the same three-tier shape every other Cairns build sits on.

What you'll have
  • A working YouTube vault with L1 waypoints, dense cards in the L2 Chain of Density Card Catalog Layer (Card Catalog Layer for short) per source, and L3 raw transcript notes, generated from public captions on channels you actually care about.
  • A searchable retrieval layer via Supabase pgvector, so your agent can ask questions across every transcript and cite the exact source instead of bluffing.
  • An overnight ingestion pattern you can later wire to a cron job, so new videos flow in while you sleep and the vault stays current without you babysitting it.
8D Build
Build time ~1 Sunday session
Track 8D build (4D students inspect the demo read-only)
Touches yt-dlp · Obsidian · Supabase pgvector · Chain-of-Density · cron (later)
Source blueprints/youtube-cairns/README.md
One-click start

Hand the blueprint to Claude.

Open Claude Code in your local workshop repo. Paste this, and Claude will read the blueprint end to end, ask you which channels you want to follow, and walk you through the build with confirmation gates between phases.

How this fits in the OS.

Depends on

The Cairns Three Tiers framework. This build is one concrete instance of that contract, applied to public YouTube channels.

Uses

yt-dlp for captions, Python for the pipeline scripts, Obsidian as your local vault surface, Supabase pgvector for retrieval by meaning, and Infisical or 1Password for credentials.

Unlocks

A second brain that grows by itself overnight, source-cited answers across hundreds of hours of video, and a pipeline shape you can later point at podcasts, blogs, or any messy source pile.

What you'll build.

By the end of this build you have a local vault with three folders: cairns/L1/, card-catalog/L2/, and raw/transcripts/. You pick a public YouTube channel. The pipeline pulls a few captions, drafts a dense card per video, links the cards to ten durable waypoints, and embeds all three layers into Supabase so your agent can answer questions and cite the exact source.

The reference vault behind the public Build Lab lessons holds 56,621 transcripts across 48 channels at $0 external API cost, ingested over years using this same shape. You will not match that on a Sunday. That is fine. The Sunday goal is the shape working end to end on a small slice you actually care about. Scale is just more channels and more nights.

L1 / MapTen durable waypoints. The agent reads these first to decide where to look next.
L2 Chain of Density Card Catalog Layer (Card Catalog Layer for short)One dense Chain-of-Density card per video. Summary, retrieval hooks, linked Cairns, source path.
L3 / RawImmutable transcript notes. The source of truth the agent cites for exact evidence.
Retrieval cascades top down. Storage grows bottom up. You hand the raw layer to the pipeline once. The cards and waypoints update as new videos land.

What changes for you.

The pipeline itself is straightforward Python. The bigger thing is the shift in how you treat YouTube as a knowledge source. Three internal shifts you'll feel:

  • You stop bookmarking videos you'll never rewatch. The captions go into a vault you can query. "What did that channel say about Cairns retrieval?" becomes a real question with a sourced answer, not a search through your watch history.
  • You stop dumping raw transcripts into chat windows. The agent reads waypoints first, picks a card, opens the raw transcript only to verify. Your context window stays small. Your answers stay grounded.
  • You start thinking of overnight as compute time. Once the shape works, the next move is a cron job. Channels you follow get checked nightly. New videos land, get transcribed, get embedded, get summarized, get linked. By morning, the vault is current. You wake up to a brief, not a backlog.

Before you start.

This build assumes a few things are already in place. If any of these are unfamiliar, work through the prereq cards first.

  • You understand the Cairns three-tier shape. If L1, L2 Chain of Density Card Catalog Layer, L3 doesn't ring a bell yet, read The Cairns Three Tiers first. The blueprint assumes you know the contract.
  • Python 3.10 or newer. Standard library only. The pipeline scripts have zero dependencies beyond yt-dlp.
  • yt-dlp installed locally. brew install yt-dlp on macOS, pip install yt-dlp works on Windows. The pipeline calls it as a subprocess, so all it needs is a yt-dlp command on your PATH. If Homebrew starts compiling rust or deno from source (it does that on a macOS version it no longer ships pre-built packages for), stop it and run pipx install yt-dlp instead. You do not need Xcode.
  • A Supabase project with the vector extension available. Free tier is fine for a Sunday build. You will run one SQL migration to create namespaced tables.
  • An OpenAI API key for embeddings, stored in your password manager. The blueprint defaults to text-embedding-3-small. You can swap in a local model later, but the starter uses OpenAI so you can finish in one session.
  • Optional, only if you want graph traversal. Neo4j Desktop, Aura, or a local Neo4j server. The Sunday build does not require it.
  • Public content only. Public YouTube channels, or channels you have rights to process. Do not point this at private uploads or content with usage restrictions.
Credential rule, repeated here because it matters

Real API keys and service tokens never get pasted into Claude Code. Store them in Infisical or 1Password, with a placeholder like PASTE_HERE, and you type the real value directly into your password manager or your local .env file. If Claude offers to enter a real key for you, say no. You always do that part yourself.

The build.

Five phases. The starter prompt at the top of the page walks Claude through all of them with confirmation gates between each one. What follows is each phase in order, with what to paste into Claude Code and how to know it worked.

Phase 01

Configure your channel list and env

~15 min

Copy the example files into the real ones. Pick one channel. Three videos max. You can scale up after you trust the shape.

Paste into Claude Code

In the local repo, copy blueprints/youtube-cairns/pipeline/channels.example.yaml to channels.yaml and pipeline/.env.example to pipeline/.env. Then ask me which public YouTube channel I want to start with. I'll give you the URL and a stable slug (the slug becomes part of the L2 Chain of Density Card Catalog Layer and L3 file paths, so pick something I won't want to rename). Edit channels.yaml with one channel and max_videos: 3. Do not write any real credentials into pipeline/.env. Leave the keys as YOUR_OPENAI_API_KEY_HERE style placeholders. I will type the real values in myself after we finish the dry-run pass.

Claude offers to enter your real Supabase or OpenAI key? Say no. Real keys go into your password manager and into pipeline/.env directly. They do not pass through this chat.
Phase 02

Pull captions and write the L3 raw layer

~10 min for 3 videos

The build.py script calls yt-dlp for each channel in your config, pulls public captions, and writes one immutable markdown note per video into vault/raw/transcripts/<channel-slug>/. These are your L3 source-of-truth files. Once written, you do not edit them. If a transcript is bad, you record the correction in L2 Chain of Density Card Catalog Layer later.

Paste into Claude Code

Run python3 blueprints/youtube-cairns/pipeline/build.py --config blueprints/youtube-cairns/pipeline/channels.yaml --vault blueprints/youtube-cairns/vault --max-videos 3 from the repo root. Show me what landed in vault/raw/transcripts/. I want to see three markdown files, each with a header block (video ID, channel slug, source URL) and a transcript body. If yt-dlp failed on any video, show me the stderr and we'll decide whether to skip it or troubleshoot.

The agent does this for you. You don't hand-download captions. The script handles the channel listing, the per-video caption fetch, the file naming, and the metadata header. You just confirm the L3 files look right.

Channel has captions disabled, or videos are members-only? Skip that channel for the first pass. Public auto-captions are the easiest path. Once the shape works on a clean channel, you can add fallback transcription later (MacWhisper or Parakeet on a local audio file).
Phase 03

Draft Chain-of-Density cards in the L2 Chain of Density Card Catalog Layer

~10 min for 3 videos

Phase 2 gave you raw evidence. Phase 3 gives you the middle layer. The enrich.py script reads each L3 transcript and drafts one L2 Chain of Density Card Catalog Layer card per source. The card holds a dense summary, retrieval hooks, key entities, linked L1 Cairns, and the path back to L3.

Quick definition. Chain-of-Density is a writing pattern. The first pass is a sparse summary. The next pass keeps the same length but packs in more specific signal: entities, numbers, named ideas. You repeat until the card is dense without being longer. That dense card is what the agent reads at retrieval time.

Paste into Claude Code

Run python3 blueprints/youtube-cairns/pipeline/enrich.py --vault blueprints/youtube-cairns/vault. Then open vault/card-catalog/L2/sources/<channel-slug>/ in Obsidian and walk me through one card. I want to see: the dense summary, the retrieval hooks (the questions this video could answer), the linked L1 Cairns, the source path back to L3, and the promotion note (does this card suggest any L1 waypoint should change). The starter version is deterministic, so this is a draft I will review, not a finished card.

The starter cards are drafts. The production reference uses richer enrichment. For the Sunday build, the deterministic version is the right starting point because you can inspect what it did and adjust the prompt later. Treat every L2 Chain of Density Card Catalog Layer card as a draft until a human has read it.

The L2 Chain of Density Card Catalog Layer card looks thin or wrong? Good signal. The deterministic enricher is the floor, not the ceiling. Mark the card as a draft, note what's missing, and revisit after you've seen a few. Pattern-matching across a handful of cards is what tells you where to tune the prompt.
Phase 04

Create your Supabase tables and embed the layers

~20 min

This phase moves your vault from local markdown to retrieval by meaning. Supabase pgvector is a column type in Postgres that lets you search by what something means, not just by the words it contains. You'll create namespaced tables for your vault, then embed every L1 waypoint, L2 Chain of Density Card Catalog Layer card, and L3 chunk into those tables. After this, your agent can ask a natural-language question and get back the matching cards and chunks with source citations.

What "embed" actually means. Every chunk of text gets converted into a list of numbers (an embedding) that represents its meaning. Two chunks about the same idea sit close together in that number space, even if they don't share any of the same words. The agent searches the number space, finds the closest matches, and reads the original text back.

Paste into Claude Code, then handle the SQL step yourself

Open blueprints/youtube-cairns/pipeline/supabase/migrations/0001_youtube_cairns_blueprint.sql. Replace __CAIRNS_PREFIX__ with the same value I'll set in pipeline/.env as CAIRNS_PREFIX (something like wade_youtube_cairns). Show me the edited SQL. Do not run it. I will copy it into the Supabase SQL editor in my browser and run it there myself. That keeps the service key out of this chat.

After the SQL runs in your browser, fill in the real values in pipeline/.env directly (Supabase URL, service key, OpenAI key, the matching CAIRNS_PREFIX). Then come back to Claude Code:

Paste into Claude Code

I've run the SQL in my Supabase SQL editor and filled in pipeline/.env with real values. Now run set -a; source blueprints/youtube-cairns/pipeline/.env; set +a followed by python3 blueprints/youtube-cairns/pipeline/ingest-rag.py --vault blueprints/youtube-cairns/vault --apply. Show me the row counts written to <prefix>_sources, <prefix>_chunks, and <prefix>_cairns. If any row count is zero, stop and we'll figure out why.

Supabase says the vector extension isn't enabled? In the Supabase dashboard, go to Database, Extensions, and enable vector. Then re-run the SQL migration. The match_* retrieval functions in the migration depend on the extension being live.
Phase 05

Query the vault and confirm citations

~10 min

The whole build exists for this moment. You ask a real question. The agent walks L1 to L2 Chain of Density Card Catalog Layer to L3 and answers with source citations. If the cascade works, the blueprint works.

Paste into Claude Code

Run python3 blueprints/youtube-cairns/pipeline/query.py "What are the strongest reusable ideas from this channel?" against my vault. Show me the answer plus the L1 waypoints it picked, the L2 Chain of Density Card Catalog Layer cards it pulled, and the L3 chunks it cited. I want to see source paths and video URLs in the citations. If the answer is generic or has no citations, that's a retrieval failure, not a good answer. Tell me which layer fell down.

Optional, only if you want graph traversal too. ingest-graph.py writes a Cypher file you can apply to Neo4j to get channels, videos, sources, entities, and Cairns as a navigable graph. Skip it on the Sunday build unless graph queries are why you started.

Five phases shipped. You have a working L1 to L3 vault, embeddings in Supabase, and a query path that cites real sources. Add a second channel. Then a third. Then wire build.py + enrich.py + ingest-rag.py to a nightly cron and let the vault grow while you sleep.

When it works.

Success looks specific. If all of these are true, the blueprint is doing its job:

  • L3 transcripts are immutable. You can re-run build.py and the same video does not get rewritten. The raw layer is a one-way ratchet.
  • L2 Chain of Density Card Catalog Layer cards each cite one L3 source. Every card has a path back to the transcript that generated it. No card invents content the transcript doesn't contain.
  • L1 waypoints stay small. Ten Cairns. New videos can propose an L1 update through the card's promotion note, but L1 doesn't silently rewrite itself.
  • Queries cite cards and sources. The answer to "what did this channel say about X" comes back with the L2 Chain of Density Card Catalog Layer card name and the L3 video URL. No citations means the retrieval cascade fell down somewhere, not that the answer is bad.
  • Adding a channel is cheap. Edit channels.yaml, re-run the three pipeline commands, the new content lands in the same three layers. No new code.

If you get stuck.

Common failure modes, ranked by how often they bite people:

  • Tried to ingest 40 channels on the first pass. Most common. Start with one channel, three videos. Confirm the shape. Then scale.
  • Edited an L3 transcript to "clean it up." Don't. The raw layer exists so the agent can cite evidence and you can audit the answer. Record corrections in L2 Chain of Density Card Catalog Layer or create a reviewed derivative.
  • Skipped the L2 Chain of Density Card Catalog Layer review. Treated the deterministic enricher's output as final. The starter cards are drafts. Read them. Fix the obvious misses. The prompt improves from seeing the first few outputs.
  • Pasted a real Supabase service key into the chat. Rotate the key immediately, then redo the env step with Infisical or 1Password and the placeholder pattern. If Claude offers to enter the real key for you, say no. Real values go into your password manager and into pipeline/.env directly.
  • Query returns generic answers without citations. The cascade didn't reach L2 Chain of Density Card Catalog Layer or L3. Check that ingest-rag.py wrote rows to all three tables (sources, chunks, cairns). If any count is zero, embedding failed for that layer.
  • Channel has weird caption coverage. Some channels have great auto-captions, some have nothing. If the channel you care about has poor captions, this blueprint is the wrong starting tool. You'd want a transcription pipeline (MacWhisper or Parakeet) feeding into build.py instead.

Still stuck? Post the channel slug, the phase you're on, and the actual error, in the workshop Slack. Someone in the cohort hit the same thing.

Where to go next.

You shipped the first vault. The pattern compounds from here. The natural next moves on the shelf:

Back to the Library