---
name: dokhacgiakhoa-data-engineering-data-pipeline
description: "You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing."
license: "NOASSERTION"
metadata:
  decimalai-skill-id: "815655cd-2037-403d-8a96-377d14365cb7"
  decimalai-version-id: "ca5481e6-5bae-4f98-9494-86b4242a9ae8"
  decimalai-version: "1"
  decimalai-body-sha256: "22b669cb1cabae6b79543924aadfb6a5d31323fa0846418556e173c9d58dc419"
  decimalai-name: "dokhacgiakhoa/data-engineering-data-pipeline"
  decimalai-format: "1"
  decimalai-invocation: "model"
  decimalai-category: "data"
  decimalai-source: "https://app.decimal.ai/s/dokhacgiakhoa-data-engineering-data-pipeline@1/SKILL.md"
---
# Data Pipeline Architecture

You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.

## Use this skill when

- Working on data pipeline architecture tasks or workflows
- Needing guidance, best practices, or checklists for data pipeline architecture

## Do not use this skill when

- The task is unrelated to data pipeline architecture
- You need a different domain or tool outside this scope

## Requirements

$ARGUMENTS

## Core Capabilities

- Design ETL/ELT, Lambda, Kappa, and Lakehouse architectures
- Implement batch and streaming data ingestion
- Build workflow orchestration with Airflow/Prefect
- Transform data using dbt and Spark
- Manage Delta Lake/Iceberg storage with ACID transactions
- Implement data quality frameworks (Great Expectations, dbt tests)
- Monitor pipelines with CloudWatch/Prometheus/Grafana
- Optimize costs through partitioning, lifecycle policies, and compute optimization

## Instructions

## 🧠 Knowledge Modules (Fractal Skills)

### 1. [1. Architecture Design](./sub-skills/1-architecture-design.md)
### 2. [2. Ingestion Implementation](./sub-skills/2-ingestion-implementation.md)
### 3. [3. Orchestration](./sub-skills/3-orchestration.md)
### 4. [4. Transformation with dbt](./sub-skills/4-transformation-with-dbt.md)
### 5. [5. Data Quality Framework](./sub-skills/5-data-quality-framework.md)
### 6. [6. Storage Strategy](./sub-skills/6-storage-strategy.md)
### 7. [7. Monitoring & Cost Optimization](./sub-skills/7-monitoring-cost-optimization.md)
### 8. [1. Architecture Documentation](./sub-skills/1-architecture-documentation.md)
### 9. [2. Implementation Code](./sub-skills/2-implementation-code.md)
### 10. [3. Configuration Files](./sub-skills/3-configuration-files.md)
### 11. [4. Monitoring & Observability](./sub-skills/4-monitoring-observability.md)
### 12. [5. Operations Guide](./sub-skills/5-operations-guide.md)