---
name: willoscar/source-ingest
source: https://app.decimal.ai/s/willoscar-source-ingest@1/SKILL.md
source_sha256: 38a4c5f0ffc3
---

# Source Ingest

Goal: normalize mixed source inputs into local tutorial-ready text while preserving provenance.

## Inputs

- `sources/manifest.yml`

## Outputs

- `sources/index.jsonl`
- `sources/provenance.jsonl`

## Supported kinds

- `webpage`
- `pdf`
- `markdown`
- `repo`
- `docs_site`
- `video`

## Behavior

- Accept local file paths or remote URLs from `sources/manifest.yml`.
- For `video`, use transcript-first ingestion:
  - provided `transcript_locator`
  - Bilibili subtitles when available
  - otherwise fail clearly instead of pretending the watch page is usable text
- Plain YouTube/Bilibili watch pages should not be modeled as `kind: webpage`.
- Continue past optional-source failures and record warnings.
- Fail closed when any source marked `required: true` cannot be ingested.

## Script

### Quick Start

- `uv run python .codex/skills/source-ingest/scripts/run.py --workspace <workspace>`

### All Options

- `--workspace <dir>` (required)
- `--unit-id <U###>`
- `--inputs <semicolon-separated>`
- `--outputs <semicolon-separated>`
- `--checkpoint <C#>`

### Examples

- Ingest all listed sources:
  - `uv run python .codex/skills/source-ingest/scripts/run.py --workspace <workspace>`

## Troubleshooting

### Issue: repo or docs site ingests too much noise

**Fix**:
- Narrow the manifest to the most relevant docs root or replace the source with a specific docs page.