Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →vellum-ai/memory-corpus-ingest
Ingest a large dataset into memory as a skimmed map. Cold-store the raw files under a workspace imports directory, census them into a slice plan, skim each slice into compact map pages that point back at the raw files, ingest the map with the memory ingest CLI, and author a drill-in retrieval skill so the corpus stays searchable on demand. For recording archives, transcript collections, document dumps, and any corpus too large to hold in memory directly.
Passed 13 of 17 test cases with the skill, and 4 of 17 without it.
and 15 more in the full report
12 fixed5 passed either way5 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 10 test cases the skill fixed. Same model, same prompt, with and without the skill.
After creating staged map pages in `imports/docs/.staging`, what command line call must be executed first before committing the pages to long-term memory?
| Checked | Without skill | With skill |
|---|---|---|
| Runs assistant memory ingest with the --dry-run flag first | Missed | Passed |
The instructions your model reads when the skill loads.
Bring a large dataset into the assistant's working knowledge without stuffing it into memory. The model is a library: the workspace holds the stacks (the raw files, cold and complete), memory holds the card catalog (a small set of map pages that say what exists, when it is from, and where to look), and a purpose-built retrieval skill is the librarian that walks to the right shelf on demand.
Two invariants drive everything below:
memory/concepts/ except the map pages, and nothing is ever appended to memory/buffer.md (bulk buffer appends trip the consolidation burst guard and per-run caps; the map bypasses the buffer entirely via assistant memory ingest).Identify the source and its size before committing:
du -sh /path/to/raw-corpus
find /path/to/raw-corpus -type f | wc -l
Tell the user what will happen: the raw files move into the workspace, a bounded number of summarization passes read them once to build the map, the map is ingested into memory, and a lookup skill is authored for drill-in. Skimming a large corpus is real LLM work that costs time and money; confirm before starting. For Fathom recording exports, read references/fathom.md first for format discovery and slicing guidance.
Land the raw files under an imports directory in the workspace, one directory per source:
Screen for credentials BEFORE copying: an arbitrary corpus can carry secret
material, and anything landed under imports/ becomes reachable by workspace
tools, backups, and retrieval flows.
cd "$VELLUM_WORKSPACE_DIR"
# 1a. Screen for secret-bearing FILE NAMES; review every hit with the user.
find /path/to/raw-corpus \( -name '.env*' -o -name '*.key' -o -name '*.pem' \
-o -name '*credential*' -o -name '*secret*' -o -name 'cookies*' \
-o -path '*tokens*' -o -path '*oauth*' \) -print
# 1b. Screen file CONTENTS for credential shapes. --hidden and --no-ignore
# matter: rg skips dotfiles and gitignored paths by default, which is
# exactly where credentials live. Capture the FULL list (no truncation):
test cases passed, vs 4 / 17 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.