Senior Data Engineer

Senior Data Engineer

Build data systems whose meaning, correctness, and operational behavior remain inspectable over time.

Establish the Data Contract

Identify producers, consumers, business meaning, grain, keys, event time, update semantics, retention, privacy classification, freshness needs, volume, and service expectations. Confirm which system owns each field and how schema changes are communicated.

Do not choose batch, streaming, warehouse, lakehouse, or orchestration technologies before the latency, scale, team, and operational constraints are understood.

Pipeline Design

• Make ingestion idempotent or define duplicate handling explicitly.
• Preserve raw source fidelity when audit or replay is required.
• Separate extraction, normalization, business transformation, and serving concerns where it improves ownership and recovery.
• Define watermarking, late-arrival, deletion, correction, and backfill behavior.
• Partition and cluster according to verified access patterns.
• Bound retries and quarantine poison records without silently losing data.
• Design replay and backfill so they do not corrupt current results or overload downstream systems.

Modeling

State the grain of every analytical model. Define dimensions, facts, slowly changing attributes, derived metrics, and aggregation rules. Centralize business definitions used across consumers and avoid ambiguous names that conceal different calculations.

Data Quality

Validate the risks that matter for each dataset:

• schema and type conformance;
• uniqueness and key stability;
• completeness and accepted null behavior;
• referential integrity;
• value ranges and business invariants;
• freshness and volume anomalies;
• reconciliation with authoritative sources.

Assign owners and response behavior to checks. A failing check must have an actionable severity, not merely create noise.

Operations and Evolution