Back to Library
Stage 04 / 8D build

Cairns Ingest Pipeline.

Drop in an article, a transcript, a note. The pipeline preserves the raw, generates a dense card, and proposes new waypoints. Every step you approve. One slash command runs the whole flow, the agent does the writing, and your second brain compounds on every source you feed it.

What you'll have
  • A single ingest path for every source transcripts, articles, meeting notes, papers, voice memos. Same skill, same vault layout, same human approval gate at the end.
  • Raw evidence preserved first, always L3 raw is written before any summarization runs. The dense card cites the raw. Provenance never breaks.
  • Proposed waypoints, never silent writes the agent flags when a source should change your always-on context and asks before touching L1. Your standing memory stays curated.
8D Build
Build time ~45 min
Track 8D required, 4D optional intro
Touches Cairns L1 / L2 Chain of Density Card Catalog Layer (Card Catalog Layer for short) / L3 · /cairns-ingest skill · Chain of density
Source .claude/skills/cairns-ingest/SKILL.md
One-click start

Ingest your first source.

Hand the skill spec off to Claude and ingest one real source you actually want to remember. The agent does the writing. You approve each step.

How this fits in the OS.

Depends on

A second brain vault on your computer with the Cairns three-tier skeleton already in place (cairns/L1/, cairns/L2/, cairns/L3/), and the cairns-ingest skill installed in your workshop repo at .claude/skills/cairns-ingest/SKILL.md.

Uses

One skill, plain markdown files, your existing vault folder layout, and the chain-of-density technique. No external services, no credentials, no API keys.

Unlocks

A single ingest path for every source you ever want to remember, immutable evidence preservation at L3, dense L2 Chain of Density Card Catalog Layer (Card Catalog Layer for short) cards the agent can scan instead of grepping raw files, and curated L1 waypoints proposed by the agent and approved by you.

What you'll build.

The ingest pipeline is the entry point to your second brain. If Librarian is how knowledge comes out of Cairns, ingest is how knowledge goes in. One skill takes any source you point it at, classifies the type, writes the raw evidence to L3 first, runs a chain-of-density pass to produce an L2 Chain of Density Card Catalog Layer card, and only then proposes whether the source should change your always-on L1 context.

The whole point of this build is that the agent does the work. You describe what you want to remember in plain English. The agent classifies, writes, summarizes, links, and proposes. You approve each step. The structural discipline of the pipeline (raw preserved first, dense cards cite raw, L1 never silently updated) is enforced in the skill itself, not left to your willpower.

What changes for you.

The nine-step flow is the easy part. The bigger thing is the shift in how you treat capture. Three internal shifts you'll feel:

  • Sources stop evaporating. You read an article worth remembering, you hand it to /cairns-ingest, and it lands in your vault with a dense card and a backlink to the raw. You stop bookmarking and forgetting.
  • Raw evidence becomes load-bearing, not optional. The pipeline writes L3 before it writes anything else. Your dense summary is never the only record. Six months later when you want to audit what was actually said, the verbatim is there.
  • Your standing context stays curated. The agent doesn't write to L1 silently. It proposes a waypoint only when a source changes how you operate, and asks before touching the always-on layer. Memory drift gets gated at the door.

Before you start.

Two prerequisites:

  • Cairns is set up. You've worked through The Cairns Three Tiers and you have at least the L1, L2 Chain of Density Card Catalog Layer, and L3 folder skeleton in your vault. Ingest writes into folders that have to exist.
  • The skill is installed. The cairns-ingest skill ships in the workshop repo at .claude/skills/cairns-ingest/SKILL.md. Once it's there, Claude Code picks it up automatically when you ask to ingest, capture, add, or process a source.

No credentials, no API keys, no external services. This is local markdown only.

Walk through the build.

Step 01

Verify the skill is installed

~5 min

Confirm the skill spec exists locally before you run anything. The skill file is the source of truth. This card is the orientation pass.

Paste into Claude Code

Open .claude/skills/cairns-ingest/SKILL.md in our local workshop repo. Read it end to end and tell me back: (1) what the skill does in one sentence, (2) the nine steps in the workflow, (3) the atomic write pattern. I want to confirm the spec is current before we run it.

The agent does this for you. You don't parse the spec by hand. Claude reads the file, summarizes it, and confirms the behavior you should expect. If the file is missing, your workshop repo clone is stale; pull the latest from upstream and try again.

Spec on this card disagrees with the local file? The local file wins. It's the source of truth. This card is an orientation pass that can lag the spec by a release.
Step 02

Pick a real source to ingest first

~5 min

Don't use a test file. Pick a real source you actually want to remember: a transcript from a call this week, an article that shifted how you think about something, a meeting note, a Slack thread, a research paper, a voice memo. The point of the first run is to feel the value of structured capture, not to validate the skill.

Have the source on your computer or have a URL ready. The skill will fetch URLs or read local files. Either path lands the same shape of L3 raw, L2 Chain of Density Card Catalog Layer card, and L1 proposal.

Step 03

Confirm vault mode and classify the source

~5 min

The skill opens by confirming the shape of your vault. If only cairns/L1/ exists, you're on the 4D fast path and the skill won't create L2 Chain of Density Card Catalog Layer or L3 without asking to upgrade first. If the full L1, L2 Chain of Density Card Catalog Layer, and L3 skeleton is present, the skill writes into all three.

Once vault mode is confirmed, the skill classifies the source. Type is one of: transcript, meeting, paper, article, Slack thread, note, URL, decision context, or other. Classification drives which L3 subfolder gets the raw and which L2 Chain of Density Card Catalog Layer naming pattern gets the card.

Decisions take a different path

If you're capturing a decision rather than a source, the right tool is the /log-decision slash command, not cairns-ingest. Decisions live under cairns/L2/decisions/ and get the atomic five-file write described on the Decision Logging card. The ingest skill points you at /log-decision when it detects a decision-shaped source.

Step 04

Write L3 raw first, always

~5 min

This is the structural rule the skill enforces. The complete source is saved to cairns/L3/<type>/ before any summarization runs. No edits, no compression, no chain of density at this step. Just the verbatim.

If the source is an existing L3 file, the skill does not overwrite or edit it. Raw is immutable. The only allowed change is adding frontmatter or provenance, and only if you approve.

L3 RAWVerbatim source, complete, written first. Provenance preserved. Never edited after first write.
L2 CARDDense chain-of-density summary that cites L3 by backlink. Append-only after first write.
L1 PROPOSALSingle short waypoint line, proposed to you. Never silent. You approve before it lands.
Raw evidence is the load-bearing layer. Summaries cite raw. Waypoints cite summaries. The provenance chain never breaks.
Step 05

Run the chain-of-density loop for the L2 Chain of Density Card Catalog Layer card

~10 min

The L2 Chain of Density Card Catalog Layer card lives at cairns/L2/cards/YYYY-MM-DD-<source-type>-<slug>.md. The skill writes a sparse first summary, lists the salient entities and details it left out, then rewrites at the same length, denser. It loops until the final 120 to 180 word dense summary is written.

The card stays short. The signal goes up. Same compression technique walked through on the Chain-of-Density Cards page, now wired into the ingest pipeline so you don't run it by hand.

The agent does this for you. You don't write the passes. You don't do the entity extraction. You read the final card on the read-back step and approve, edit, or rerun.

Step 06

Add retrieval hooks and the L3 backlink

~5 min

A dense card the agent can't find later is dead weight. The skill adds explicit retrieval scaffolding to every L2 Chain of Density Card Catalog Layer card:

  • Questions this card should answer. The questions you'd want a future session to surface this card on.
  • Tags. Topic anchors that L1 INDEX entries can route to.
  • Entities. People, organizations, tools, projects, dates, concepts.
  • Related L1 categories. Which always-on waypoint files this card connects back to.
  • Source notes. The L3 backlink, the evidence level, and a one-sentence L1 promotion candidate (or none).

The backlink to L3 is non-negotiable. Every L2 Chain of Density Card Catalog Layer entry has a pointer back to the raw it summarizes. Lose the path and the card becomes an opinion.

Step 07

Propose an L1 waypoint, never write one silently

~5 min

This is the human-in-the-loop gate. L1 is the always-on memory layer the agent loads at session start. Updates to L1 change how every future session behaves. The skill never updates L1 on its own.

It proposes one short waypoint only when the source changes standing context: active work, decisions, people, preferences, constraints, or durable operating shape. If approved, the skill appends a single line under cairns/L1/personal/<category>.md:

L1 append shape

- YYYY-MM-DD: <waypoint under 220 characters> [L2 Chain of Density Card Catalog Layer: <card filename>]

If the target waypoint file doesn't exist yet (your first project capture, your first people capture, your first learning goal capture), the skill creates it with L1 frontmatter, a heading, and the first dated pointer. L1 stays short and navigational. Detail belongs in L2 Chain of Density Card Catalog Layer and L3.

Tempted to approve every L1 proposal? Don't. The agent will over-propose at first. L1 is for standing context, not for interesting-but-transient items. If the source doesn't change how you operate, the L2 Chain of Density Card Catalog Layer card is enough. Decline the L1 promotion and move on.
Step 08

Commit the atomic write

~5 min

Every file the skill produces lands in a temp stage outside the vault first. The skill validates frontmatter, paths, and backlinks before any file touches your real vault. Only then does it copy staged files into the vault, and only if their target paths don't already exist.

Atomic means all or nothing. If any write fails, the skill removes every file copied during this run and reports rollback. You either get the complete ingest or you get a clean rollback. You never get a half-ingested source rotting in your vault.

Same epistemic posture as decision logging

The atomic write pattern here mirrors the one on the Decision Logging build. Stage outside the vault, validate, commit as a transaction, roll back on any failure. The discipline is the same; only the file set differs. Capturing knowledge and capturing decisions both deserve transactional writes.

Step 09

Read back every path

~5 min

The skill closes by reporting the L3 path, the L2 Chain of Density Card Catalog Layer card path, and any pending L1 proposal. Open the L2 Chain of Density Card Catalog Layer card. Confirm the dense summary actually reflects the source. Edit it in place if the agent compressed away something load-bearing. The L3 raw stays untouched.

The diagnostic question after your first ingest: open a fresh Claude Code session in your vault folder and ask "what did I capture about <topic> this week?" If the agent surfaces the new L2 Chain of Density Card Catalog Layer card by reading L1 routing or the card itself, the pipeline is wired. If it has to grep the filesystem blindly, something didn't land where it should have.

Why this shape, not a bigger one.

The ingest pipeline looks minimal. Raw plus dense card plus optional waypoint. That's by design. The shape works at small corpus sizes without fancy retrieval, and it scales by adding layers later instead of rewriting the foundation.

External validation

Andrej Karpathy described an essentially identical workflow on X: a raw/ directory of source documents plus an LLM-compiled wiki plus Obsidian as the editing frontend plus LLM-maintained index files and brief summaries. At his working corpus size (around 100 articles, around 400,000 words), he explicitly noted he didn't reach for fancy RAG. The LLM auto-maintained the indices and read related data well enough.

"I thought I had to reach for fancy RAG, but the LLM has been pretty good about auto-maintaining index files and brief summaries of all the documents and it reads all the important related data fairly easily at this small scale." That's the anchor. You don't need a vector store on day one. You need a clean ingest path that preserves raw, writes dense cards, and gates standing-context updates. Add layers when you cross the threshold, not before.

Karpathy also flagged the write-side complement that the ingest pipeline alone doesn't solve: outputs of agent queries should be filed back into the wiki, so each exploration enhances retrieval for the next one. That loop closes elsewhere in the OS, on the consolidation passes and the decision logger. Ingest handles the input side. Closing the loop is on the shelf alongside it.

When it works.

You'll see three things land in your vault on a successful ingest:

  • A new file under cairns/L3/<type>/ with the complete source, verbatim. Open it. The content should match the source exactly.
  • A new file under cairns/L2/cards/ with the dense 120 to 180 word summary, the retrieval hooks, the key entities, and the backlink to the L3 file. Open it. The summary should feel tight, not fluffy.
  • Either an L1 proposal you approved (one new line in the right waypoint file) or no L1 change at all. The default outcome is no L1 change. That's correct. Most sources don't shift standing context.

The compounding test: ingest five sources over a week. At the end, ask your agent "summarize what I've captured this week and surface anything that connects." If the agent answers by scanning the L2 Chain of Density Card Catalog Layer cards (not grepping raw), the pipeline is doing its job.

If you get stuck.

Common failure modes, ranked by how often they bite people:

  • Vault path was ambiguous and the skill guessed wrong. Most common on the first run. The skill defaults to ${HOME}/Documents/second-brain on macOS and the user's Documents folder on Windows. If the default is wrong for you, tell the agent explicitly: "the vault root is <path>, use that for every cairns-ingest from now on."
  • The L1, L2 Chain of Density Card Catalog Layer, or L3 folders didn't exist yet. The atomic write needs targets. If the folders are missing, the skill can create them, but it's cleaner to set up your Cairns skeleton first via the Cairns Three Tiers card.
  • The L2 Chain of Density Card Catalog Layer card lost something load-bearing. Chain of density is lossy by design. The read-back step is your moment to catch a missing entity or a softened claim. Edit the L2 Chain of Density Card Catalog Layer card in place. Never edit the L3 raw; that's the evidence layer.
  • The agent tried to write to L1 silently. If you see an L1 file change you didn't approve, stop and ask the agent which step skipped the proposal gate. The spec is explicit: never silently update L1. If a session ever does, treat that as a bug in the skill setup, not a feature.
  • You ingested a source that was actually a decision. The skill should redirect you to /log-decision when it detects decision-shaped input. If it doesn't, redirect yourself. Decisions belong under cairns/L2/decisions/ with the atomic five-file write, not under cairns/L2/cards/.
  • A write failed mid-run and you're not sure about rollback. The spec says rollback is automatic; if rollback itself fails, the skill stops and tells you the exact paths needing manual review. If you ever see a partial ingest, do not re-run the skill on the same source until you've cleaned up the partial files using the paths the agent reported.

Still stuck? Post the prompt, the agent's response, and what you expected, in the workshop Slack. Someone in the cohort is solving the same thing.

Where to go next.

You ran the pipeline end to end on one source. The compounding starts when the pattern becomes habit. The natural next moves on the shelf:

Back to the Library