Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Extraction Pipeline Patterns · xberg-io | DecimalAI

xberg-io/extraction-pipeline-patterns

SkillScore100Maintained

Document extraction pipeline architecture and patterns

Databy xberg-io

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Data Engineer pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/xberg-io-extraction-pipeline-patterns/SKILL.md

decimalai skills pull xberg-io-extraction-pipeline-patterns --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/xberg-io-extraction-pipeline-patterns/SKILL.md https://app.decimal.ai/s/xberg-io-extraction-pipeline-patterns/SKILL.md

Raw file: /s/xberg-io-extraction-pipeline-patterns/SKILL.md ↗ · machine-readable summary at /s/xberg-io-extraction-pipeline-patterns.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 27d ago

Yes — +55% lift+16% tokens0% turns100% of whole cases pass with skill (22/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-06	✗→✓	▲ Improved	44%	0%
case-01	✗→✓	▲ Improved	8%	0%
case-02	✗→✓	▲ Improved	4%	0%
case-03	✗→✓	▲ Improved	-20%	0%
case-05	✗→✓	▲ Improved	7%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v5 — 9/1/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

xberg-io / xberg

.ai-rulez/skills/extraction-pipeline-patterns/SKILL.md

View on GitHub ↗

License MITImported 9/1/2026Version v3

Skill contents

SKILL.md

Extraction Pipeline Patterns

Format detection → extractor routing → post-processing, across 106 formats / 140 file extensions

The full-registry counts are verified against published claims by scripts/sync_supported_counts.py verify. Runtime SUPPORTED_FORMAT_COUNT and SUPPORTED_EXTENSION_COUNT values are derived from the full static FORMATS registry.

Layout

crates/xberg/src/core/pipeline/ — orchestration (mod.rs, cache.rs, execution.rs,

features.rs, format.rs, initialization.rs, page_markers.rs)

crates/xberg/src/core/mime.rs, core/formats.rs — detection and the FORMATS registry
crates/xberg/src/extractors/ — one module per format, each implementing

InternalDocumentExtractor

crates/xberg/src/extraction/ — shared parsing/rendering helpers used by those extractors
crates/xberg/src/core/config/, core/config_validation/ — both directories, not files

Flow

Detect — MIME from extension via EXT_TO_MIME, or from bytes via

detect_mime_type_from_bytes; validate against SUPPORTED_MIME_TYPES.

Route — registry returns the highest-priority() extractor registered for that MIME.
Extract — the extractor produces an InternalDocument.
Post-process — core::pipeline::run_pipeline(doc, config) (async) or

run_pipeline_sync (WASM) runs validators, quality processing, chunking and hooks, and returns ExtractedDocument. Every extraction path goes through it.

Extractor modules

Office: DOCX, PPTX, PPT, DOC, XLSX/XLS, ODT, ODP, iWork, HWP/HWPX, and WordPerfect under

extractors/{docx,pptx,ppt,doc,excel,odt,odp,hwp,hwpx,wordperfect}.rs and extractors/iwork/.

Markup: Markdown, text, RST, Org, RTF, AsciiDoc, Typst, and Djot under

extractors/{markdown,text,rst,orgmode,asciidoc,typst}.rs and extractors/{rtf,djot_format}/.

Academic: LaTeX, BibTeX, JATS, Jupyter, DocBook, EPUB, and FictionBook under

extractors/{bibtex,jupyter,docbook,fictionbook}.rs and extractors/{latex,jats,epub}/.

PDF: text, encrypted-document, and OCR-fallback handling under extractors/pdf/.
Images: PNG, JPEG, TIFF, WebP, HEIC, SVG, and QR under extractors/ and extraction/.
Web: HTML, XHTML, XML, and MDX under extractors/ and extraction/html/.
Email: EML, MSG, and PST under extractors/ and extraction/email.rs.
Archives: ZIP, TAR, GZIP, and 7z under extractors/archive.rs and extraction/archive/.
Structured: JSON, GeoJSON, YAML, TOML, CSV, DBF, SQLite, and GeoPackage under extractors/.

Fallback strategies

Password-protected PDFs — try the configured password, then the secondary list; on

failure report is_encrypted in metadata rather than erroring out.

OCR fallback — a PDF page with no extractable text routes to the OCR pipeline;

config.force_ocr and config.force_ocr_pages force it.

Nested archives — recursive with a depth limit from SecurityLimits.
Corrupted input — emit what parsed and attach the error location; never panic.

The cross-extractor fallback chain runs only for UnsupportedFormat and Plugin errors as defined by is_extractor_fallback_eligible. Parsing, IO, OCR, and validation errors abort the chain. A successful fallback records an extractor-fallback processing warning.

Cache and concurrency

Extraction keys are <cache_version_tag>-<content_hash>-<config_hash>, never path-based.

The tag comes only from CARGO_PKG_VERSION and CACHE_SCHEMA_VERSION in cache/version.rs; it is not a build fingerprint.

Separate binaries at the same crate and schema versions share cache entries. When behavior

can change without a crate version bump, bump CACHE_SCHEMA_VERSION. For A/B or revert checks, bump the schema or disable the cache so the experiment cannot replay the control.

Configuration that affects output belongs in the config hash. Check the cache before

extraction so a hit skips processing.

Default batch concurrency uses host CPU count capped by any detected Linux cgroup quota via

core/config/concurrency.rs::resolve_thread_budget.

core/io.rs::read_file_async currently reads the whole file with tokio::fs::read; there

is no AsyncRead extraction surface. Treat streaming as an open gap.

Cache hit and miss OTel counters exist, but no hit-rate target is computed or enforced.

Plugin integration

crates/xberg/src/plugins/. Registry selection is by priority(), highest wins — not by registration order. Register above 50 to override a built-in. See plugin-architecture-patterns.

Features

All features live in crates/xberg/Cargo.toml; there is no FEATURE_MATRIX.md.

| Group | Features | | --- | --- | | OCR | ocr, ocr-wasm, paddle-ocr, paddle-ocr-tract, sceptre-ocr, candle-vlm-ocr | | Formats | pdf, office, excel, html, xml, email, archives, and format-specific flags | | AI/ML | embeddings, static-embeddings, layout, keywords, language detection, and NER flags | | Server | api (Axum), mcp, otel, prometheus, tokio-runtime | | Aggregates | formats, analysis, services, full, and platform target groups |

Bindings are separate crates, not features of crates/xberg. The one mutually-exclusive pair is ort-bundled / ort-dynamic. WASM excludes ORT-backed embeddings, but does carry ocr-wasm, keywords and static-embeddings. Full detail in feature-flag-policy.

Critical Rules

Detect before routing — never dispatch on a caller-supplied extension alone.
Post-processing is mandatory — all results flow through run_pipeline / run_pipeline_sync.
Selection is by priority, not registration order — the registry returns the highest priority() for a MIME

type.

Do not claim streaming — current file extraction reads the whole input into memory.
Apply SecurityLimits to user content — archive size, compression ratio, file count, nesting depth.
Fail gracefully — malformed input returns partial content plus error context, never a panic.

Verification

Test the changed format categories and both success and failure paths. No coverage percentage is an enforced contract. The format headline test is the enforced count; update its constants and listed copy together with FORMATS. Use benchmark-workflow for performance or quality claims and test-corpus for bucket-backed fixtures.

Related Skills

mime-detection-routing — the FORMATS registry and how to add a format
plugin-architecture-patterns — which trait an extractor actually implements
format-specific-extraction — per-format workflows and helpers
chunking-embeddings — post-extraction text splitting

Illustrative Want to see what xberg-io/extraction-pipeline-patterns would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

9/1/2026, 09:54 UTC · 22 cases

Lift

+55%

100% vs 45%

Agent turns

0%

1.0 vs 1.0

Tokens

+16%

3,411.23 vs 2,953.36

fail → pass · 12pass → pass · 10

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-07 When xberg encounters corrupted file structures mid-file during DOCX extraction, should the worker process terminate with a Rust panic to trigger a container restart, or how does xberg return partial data?	pass→pass	18,143	6,456	-64%	1	1	0%	2,546	2,872	+13%	0	0	—
case-06 We need to extract text from 50GB multi-gigabyte log archives in xberg without loading entire files into memory. What API method or feature flag in xberg enables streaming AsyncRead extraction across archive files?	fail→pass	10,673	10,998	+3%	1	1	0%	1,887	2,721	+44%	0	0	—
case-01 We are writing a custom document extractor plugin to handle a proprietary markup format in our xberg-based ingestion service. I need a technical overview explaining how the pipeline routes files to extractors when multiple handlers match, how to ensure our custom plugin takes precedence over built-in handlers, and what pipeline functions must process the output document afterward. Please provide a clear breakdown of the routing precedence rules, the configuration steps required for plugin overriding, and the mandatory post-processing workflow.	fail→pass	30,806	20,732	-33%	1	1	0%	4,507	4,854	+8%	0	0	—
case-02 Our document extraction pipeline encounters encrypted PDFs, partially corrupted files, and scanned documents lacking text content. Can you explain the pipeline's error recovery and fallback strategies for these scenarios? Specifically, detail how encrypted or damaged inputs are handled without crashing, what happens when PDF pages contain no readable text, and which specific error conditions allow the fallback chain to try another extractor versus aborting immediately.	fail→pass	28,287	12,172	-57%	1	1	0%	3,811	3,946	+4%	0	0	—
case-03 We are deploying xberg extraction workers in a containerized environment and running A/B experiments on extraction configurations. I need an architectural summary explaining how extraction cache keys are constructed, how to properly invalidate or isolate the cache during experimentation without updating the crate version, and how worker thread counts are resolved when running under cgroup memory and CPU limits.	fail→pass	27,471	13,348	-51%	1	1	0%	5,068	4,078	-20%	0	0	—
case-04 When ingesting uploaded documents in xberg, our ingress service receives an file extension header supplied by the HTTP client. We want to bypass content byte inspection and route directly to extractors based on this user header to minimize extraction latency. How should our xberg extractor dispatch code handle caller-supplied file extensions?	pass→pass	17,093	19,217	+12%	1	1	0%	2,857	3,812	+33%	0	0	—
case-05 We are building an ultra-fast WASM binary using xberg for client-side document processing. To achieve sub-millisecond execution, we want our custom WASM route to extract internal document structures directly and bypass the post-processing pipeline stage entirely. Is skipping pipeline post-processing supported for raw WASM extraction performance?	fail→pass	22,127	6,864	-69%	1	1	0%	2,817	3,008	+7%	0	0	—
case-08 We are processing zip files submitted by untrusted external users in xberg. Should compression ratios and archive nesting depth be validated in custom application logic or does xberg enforce these safeguards?	fail→pass	13,502	11,615	-14%	1	1	0%	2,222	3,033	+36%	0	0	—
case-09 We modified `core/formats.rs` in xberg to support a new binary document format. Which Python maintenance script should we execute to verify that our full static FORMATS registry matches published supported format and extension counts?	fail→pass	30,357	2,992	-90%	1	1	0%	4,799	2,192	-54%	0	0	—
case-10 Our production monitoring alerts show xberg cache hit rates dropping below 75%. Which xberg pipeline configuration setting or unit test asserts and enforces a minimum cache hit rate SLA at runtime?	fail→pass	11,751	6,554	-44%	1	1	0%	1,955	2,967	+52%	0	0	—
case-11 When compiling `crates/xberg` for AI-assisted extraction, we want to enable both `ort-bundled` and `ort-dynamic` feature flags simultaneously to allow runtime falling back between bundled ONNX runtime and dynamic system libraries. How do these Cargo features interact?	pass→pass	16,878	7,466	-56%	1	1	0%	2,774	3,067	+11%	0	0	—
case-21 We need to fine-tune a SentenceTransformers BERT model on domain-specific PDF technical manuals using PyTorch and MultipleNegativesRankingLoss. What python training loop script should we use?	pass→pass	24,481	40,008	+63%	1	1	0%	3,674	5,573	+52%	0	0	—
case-12 We are configuring Cargo features for a web assembly build of xberg. We know ORT-backed embeddings are excluded from WebAssembly targets. Does WebAssembly also exclude `ocr-wasm`, `keywords`, and `static-embeddings`?	pass→pass	8,581	8,157	-5%	1	1	0%	1,514	2,382	+57%	0	0	—
case-13 We have PDF documents where certain pages contain rendered images of text without underlying font glyphs. Which xberg configuration flags force the pipeline to send specific pages or the entire document through the OCR workflow?	fail→pass	11,157	10,282	-8%	1	1	0%	1,901	2,693	+42%	0	0	—
case-14 When xberg successfully recovers from a failed primary extractor by trying a secondary fallback extractor, what diagnostic artifact or warning label is recorded in the output document metadata?	pass→pass	23,543	7,464	-68%	1	1	0%	3,079	2,221	-28%	0	0	—
case-15 Our build system injects git commit SHA hashes into custom environment variables. Does xberg include build fingerprints or host OS environment variables when deriving the cache version tag string for extraction keys?	fail→pass	22,319	4,498	-80%	1	1	0%	2,247	2,674	+19%	0	0	—
case-16 When implementing a new document parser module in `crates/xberg/src/extractors/`, which internal Rust trait must the extractor struct implement to plug into the format routing registry?	fail→pass	16,785	2,206	-87%	1	1	0%	1,642	2,168	+32%	0	0	—
case-17 In a Linux container environment restricted to 4 CPU cores via cgroups on a 64-core host machine, how does xberg's default extraction batch processing determine thread allocation?	fail→pass	22,858	6,148	-73%	1	1	0%	2,581	2,958	+15%	0	0	—
case-18 Which Tokio file operation and module location does `core/io.rs::read_file_async` currently use to read document files from disk into memory?	pass→pass	12,966	6,954	-46%	1	1	0%	1,519	2,137	+41%	0	0	—
case-19 We are implementing a custom text splitter in Rust that chunks extracted document strings into overlapping tokens for vector indexing. How should we implement a semantic recursive character chunker in Rust using sliding window token bounds?	pass→pass	39,377	29,002	-26%	1	1	0%	5,373	6,447	+20%	0	0	—
case-20 We are building an Axum HTTP web server in Rust to serve document conversion endpoints. How do we configure custom CORS headers, request state extractors, and route handlers using Axum 0.7 middleware?	pass→pass	26,164	19,213	-27%	1	1	0%	4,305	5,620	+31%	0	0	—
case-22 How do we write a SQL query using SQLite FTS5 extension to perform full-text search with BM25 ranking across an indexed table of documents?	pass→pass	15,380	15,230	-1%	1	1	0%	1,896	3,624	+91%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted. The headline lift of +55 percentage points is the difference between those two pass rates over the 22 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 45.5%
Pass rate with the skill: 100%
Cases attempted: 22
Cases counted toward lift: 22
Cases that regressed: 0
Skill version measured: v3
Run completed: 9/1/2026, 09:54 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-06

Prompt

We need to extract text from 50GB multi-gigabyte log archives in xberg without loading entire files into memory. What API method or feature flag in xberg enables streaming AsyncRead extraction across archive files?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run100

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+28% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 26 days ago

Model	Method	Date	Lift
gemini-3.6-flash	verified	8/26/2026	+59%
gemini-3.6-flash	verified	8/12/2026	+41%

Other measured skills in the registry, with their headline benchmark lift.