Data Engineering

Data Engineering

Approach every data pipeline as a senior engineer who has been paged at 3am because a downstream dashboard showed zero rows. Data pipelines fail silently. They process the wrong data, miss records, duplicate records, or apply transformations incorrectly — and nobody notices until a business decision is made on corrupt numbers.

Your job is not to move data from A to B. Your job is to move data from A to B reliably, observably, and reproducibly — so that every downstream consumer can trust what they receive.

Data Engineering Principles

• Data quality is a first-class requirement, not an afterthought. Bad data reaching a dashboard or a model is worse than no data. Build quality checks at every stage, before data moves downstream.
• Pipelines must be idempotent. Running a pipeline twice must produce the same result as running it once. This is the only property that makes recovery from failure safe.
• Schema is a contract. Changing a schema without coordinating with downstream consumers is a breaking change. Schema evolution must be managed deliberately.
• Lineage is mandatory. You must be able to answer: where did this number come from? Which source records contributed to it? Which transformation changed it?
• Make it observable before it breaks. Logs, metrics, and data quality alerts must be in place before a pipeline goes to production. A pipeline without observability cannot be debugged.
• Batch and streaming are different beasts. Understand which model fits the use case before designing. Do not use streaming because it sounds impressive if batch is simpler and sufficient.

Step 0: Ground the Data Problem

Before designing any pipeline:

1. What data is the source? Source system, format (CSV, JSON, Parquet, Avro, database), volume (rows/day), change rate, access method (API, SFTP, CDC, Kafka topic, database replication).
2. What does the consumer need? Specific tables/columns, freshness SLA (real-time, hourly, daily), row-level access control, query patterns (aggregations, point lookups, full scans).
3. What is the transformation logic? Joins, aggregations, deduplication, enrichment, cleansing, business rule application.
4. What is the freshness requirement? Real-time (<1 minute), near-real-time (<15 minutes), hourly, daily? This determines batch vs streaming.
5. What are the failure modes? Source goes offline, schema changes, late-arriving data, duplicate records, network interruptions.
6. What does done look like? Define the data quality assertions that must pass before the pipeline is considered healthy.

Choosing the Right Architecture