Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

数据工程域 · Data Engineering · telagod | DecimalAI

telagod/engineering-data-pipelines

SkillScore100Maintained

Data engineering knowledge reference covering Airflow, Dagster, Kafka Streams, Flink, dbt, and data quality patterns. Use when building data pipelines, ETL workflows, stream processing, or data quality checks.

Databy telagod

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Data Engineer pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/telagod-engineering-data-pipelines/SKILL.md

decimalai skills pull telagod-engineering-data-pipelines --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/telagod-engineering-data-pipelines/SKILL.md https://app.decimal.ai/s/telagod-engineering-data-pipelines/SKILL.md

Raw file: /s/telagod-engineering-data-pipelines/SKILL.md ↗ · machine-readable summary at /s/telagod-engineering-data-pipelines.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

23 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 5d ago

Yes — +9% lift+13% tokens0% turns100% of whole cases pass with skill (23/23)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-06	✗→✓	▲ Improved	62%	0%
case-20	✗→✓	▲ Improved	1%	0%
case-01	✓→✓	＝ Same ✓	-21%	0%
case-02	✓→✓	＝ Same ✓	14%	0%
case-03	✓→✓	＝ Same ✓	10%	0%

Showing 5 of 23 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

telagod / code-abyss

skills/engineering-data-pipelines/SKILL.md

View on GitHub ↗

License MITImported 8/3/2026Version v1Bundled 1 file

Skill contents

references/

SKILL.md

数据工程域 · Data Engineering

> 判断先于执行：决定「是否做 / 选什么 / 如何取舍」（栈、方案、架构、权衡）前，先读领域判断内核 skills/_kernel/backend/SKILL.md——它管 judgment，本秘典管 execution；冲突时以内核判断为准。

编排：Airflow(调度) | Dagster(资产) | Prefect(现代流)
流处理：Kafka Streams(嵌入式) | Flink(集群) | Spark Streaming
质量：Great Expectations | dbt tests | Soda Core

编排检查项

幂等(UPSERT/分区覆盖) | 增量(WHERE updated_at > last_run) | 事件驱动触发 | 跨 DAG 依赖 | 数据血缘(ref()/Asset deps)

流处理检查项

质量检查项

分层验证(源→转换→目标) | 完整性+准确性+一致性 | 及时性阈值 | 加权评分 | 告警(Slack/PagerDuty)

工具对比、API 用法、质量维度详见 references/details.md

Want to see if telagod/engineering-data-pipelines would activate on your phrasing? Try it before you install — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/24/2026, 19:19 UTC · 23 cases

Lift

+9%

100% vs 91%

Agent turns

1.0 vs 1.0

Tokens

+13%

2,274.13 vs 2,011.74

fail → pass · 2pass → pass · 21

Per-case results (23)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 We are designing a data pipeline where primary abstractions are persistent data assets with explicit materialization and lineage graphs, rather than task dependency graphs. Which orchestration framework natively models data assets rather than tasks?	pass→pass	12,925	17,532	+36%	1	1	0%	1,951	1,549	-21%	0	0	—
case-02 A daily batch ETL job failed midway and needs to be re-run for yesterday's partition without causing duplicate records or requiring manual table cleanup. What write pattern should be implemented for the target table write step?	pass→pass	10,579	10,158	-4%	1	1	0%	1,671	1,902	+14%	0	0	—
case-03 How should a scheduled batch query extract only newly updated records from an operational relational table during each run without re-scanning historical data?	pass→pass	15,445	14,598	-5%	1	1	0%	2,352	2,578	+10%	0	0	—
case-04 When building multi-stage SQL transformations, how should table dependencies be declared within models so that end-to-end data lineage is automatically tracked?	pass→pass	14,854	13,146	-11%	1	1	0%	2,395	2,243	-6%	0	0	—
case-05 A microservice written in Java requires light stream processing capabilities embedded directly within the service process without deploying an external distributed processing cluster. Which stream processing library fits this embedded model?	pass→pass	9,326	8,084	-13%	1	1	0%	1,412	1,473	+4%	0	0	—
case-06 A real-time telemetry streaming application receives events that arrive out of order due to variable mobile network latency. Which stream processing mechanism handles bounded out-of-order events based on event time?	fail→pass	6,802	8,064	+19%	1	1	0%	1,005	1,633	+62%	0	0	—
case-07 A stateful stream processing application tracks user session state over long durations. Over time, state size grows continuously causing memory exhaustion. What configuration prevents unbounded state growth for inactive key state?	pass→pass	9,902	11,818	+19%	1	1	0%	1,467	2,252	+54%	0	0	—
case-08 To recover stream processing state after worker failures without reprocessing stream history from offset zero, what mechanism must be configured to persist state periodically?	pass→pass	7,131	5,592	-22%	1	1	0%	1,147	1,147	0%	0	0	—
case-09 A financial transaction stream requires strictly zero duplicated writes from Kafka source to persistent storage across system restarts. What delivery guarantee semantic must be configured end-to-end?	pass→pass	5,611	6,426	+15%	1	1	0%	843	1,233	+46%	0	0	—
case-10 When downstream database sinks experience latency spikes, upstream streaming buffers risk overflowing. What metric and mechanism should be monitored to detect and regulate ingestion flow rate?	pass→pass	15,061	12,684	-16%	1	1	0%	2,166	2,184	+1%	0	0	—
case-11 At what stages of the pipeline lifecycle should data validation checks be executed to detect and isolate corrupted records early?	pass→pass	17,319	15,436	-11%	1	1	0%	2,505	2,623	+5%	0	0	—
case-12 A data warehouse team needs to define comprehensive quality checks for incoming transactional data beyond basic row count validation. What core quality dimensions must be evaluated?	pass→pass	15,602	16,385	+5%	1	1	0%	2,342	2,717	+16%	0	0	—
case-13 An operational reporting dashboard requires fresh data available by 06:00 AM daily. How should the pipeline evaluate whether data arrival times satisfy business SLAs?	pass→pass	19,196	18,468	-4%	1	1	0%	2,955	3,278	+11%	0	0	—
case-14 A data engineering team needs an aggregate health score across dozens of quality checks with varying business criticality. How should individual check results be combined into a overall health metric?	pass→pass	17,843	18,835	+6%	1	1	0%	2,840	3,363	+18%	0	0	—
case-15 When critical data quality assertions fail on core analytical models, how should the pipeline notify the operational data team immediately?	pass→pass	16,997	15,615	-8%	1	1	0%	2,452	2,772	+13%	0	0	—
case-16 Which open-source Python framework enables teams to define data assertions, run validation against DataFrames and SQL databases, and automatically produce HTML data documentation reports?	pass→pass	4,390	5,009	+14%	1	1	0%	624	1,043	+67%	0	0	—
case-17 When building analytical models using dbt, how should column-level nullability, uniqueness, and reference assertions be configured directly in project files?	pass→pass	13,122	12,277	-6%	1	1	0%	2,256	2,501	+11%	0	0	—
case-18 Which lightweight open-source CLI tool scans SQL tables using declarative YAML check files to validate data quality inside CI/CD automation?	pass→pass	6,890	6,585	-4%	1	1	0%	1,051	1,300	+24%	0	0	—
case-19 A team requires a modern Pythonic orchestrator that treats tasks as native Python functions and natively supports dynamic workflows and modern streaming triggers. Which framework fits this model?	pass→pass	13,642	6,605	-52%	1	1	0%	1,846	1,293	-30%	0	0	—
case-20 When facing a high-level strategic dilemma between building a custom data processing engine versus adopting an established commercial technology stack, which guidelines take precedence?	fail→pass	16,671	15,299	-8%	1	1	0%	2,394	2,408	+1%	0	0	—
case-21 Our enterprise is deciding whether to build an internal custom streaming engine or purchase a commercial managed streaming service platform. What strategy evaluation framework should guide this decision?	pass→pass	17,689	23,551	+33%	1	1	0%	2,587	3,856	+49%	0	0	—
case-22 Should our cloud data warehouse adopt Kimball dimensional modeling with star schemas or Data Vault 2.0 for enterprise historical data integration?	pass→pass	18,033	12,720	-29%	1	1	0%	2,661	2,335	-12%	0	0	—
case-23 How should role-based access control (RBAC) privileges and row-level security policies be structured across Snowflake databases for Finance and HR teams?	pass→pass	20,122	22,589	+12%	1	1	0%	3,348	4,622	+38%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 23 cases were attempted. The headline lift of +9 percentage points is the difference between those two pass rates over the 23 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 91.3%
Pass rate with the skill: 100%
Cases attempted: 23
Cases counted toward lift: 23
Cases that regressed: 0
Skill version measured: v1
Run completed: 8/24/2026, 19:19 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-06

Prompt

A real-time telemetry streaming application receives events that arrive out of order due to variable mobile network latency. Which stream processing mechanism handles bounded out-of-order events based on event time?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run100

Live evalNeeds ≥20 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+13% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 4 days ago

Other measured skills in the registry, with their headline benchmark lift.