---
name: wondelai/ddia-systems
source: https://app.decimal.ai/s/wondelai-ddia-systems@1/SKILL.md
source_sha256: c60cc44630b8
---

# Designing Data-Intensive Applications Framework

A principled approach to building reliable, scalable, and maintainable data systems. Apply these principles when choosing databases, designing schemas, architecting distributed systems, or reasoning about consistency and fault tolerance.

## Core Principle

**Data outlives code.** Applications are rewritten and frameworks come and go, but data persists for decades -- prioritize the long-term correctness, durability, and evolvability of the data layer. Most applications are data-intensive, not compute-intensive: the hard problems are data volume, complexity, and rate of change, and explicit consistency/availability/latency trade-offs separate robust systems from fragile ones.

## Scoring

**Goal: 10/10.** Score a data architecture by the seven Quick Diagnostic rows below: award ~1.4 points per row answered "yes" with evidence (deliberate, documented trade-off), 0 where the answer is "no" or unknown.

- **9-10:** every domain choice -- data model, storage engine, replication, partitioning, isolation, derived-data, fault handling -- is deliberate, documented, and matched to actual read/write/consistency requirements; failover tested.
- **5-6:** core choices made but two or three diagnostic rows fail -- e.g. default isolation level unknown, hot-key risk unhandled, or failover untested.
- **<=3:** choices driven by familiarity, not requirements; ignored failure modes (replication lag, write skew, hot partitions) and accidental complexity dominate.

Report the current score, which diagnostic rows failed, and the improvements needed to reach 10/10.

## The DDIA Framework

Seven domains for reasoning about data-intensive systems:

### 1. Data Models and Query Languages

**Core concept:** The data model shapes how you think about the problem. Relational, document, and graph models each impose different constraints and enable different query patterns.

**Why it works:** Choosing the wrong data model forces application code to compensate for representational mismatch, adding accidental complexity that compounds over time.

**Key insights:**
- Relational models excel at many-to-many relationships and ad-hoc queries; document models at one-to-many relationships and locality; graph models at recursive traversals over interconnected data
- Schema-on-write (relational) catches errors early; schema-on-read (document) offers flexibility
- Polyglot persistence -- different stores for different access patterns -- is often the right answer
- Object-relational impedance mismatch is a real cost; document models reduce it for self-contained aggregates

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **User profiles with nested data** | Document model for self-contained aggregates | Profile, addresses, and preferences in one MongoDB document |
| **Social network connections** | Graph model for relationship traversal | Neo4j Cypher: `MATCH (a)-[:FOLLOWS*2]->(b)` for friend-of-friend |
| **Financial ledger with joins** | Relational model for referential integrity | PostgreSQL foreign keys between accounts, transactions, entries |

See [references/data-models.md](references/data-models.md) when picking relational vs document vs graph or evaluating schema-on-read -- adds the full trade-off matrix and query-language comparisons.

### 2. Storage Engines

**Core concept:** Storage engines trade off read performance against write performance. Log-structured engines (LSM trees) optimize writes; page-oriented engines (B-trees) balance reads and writes.

**Key insights:**
- LSM trees: append-only writes, periodic compaction, excellent write throughput, higher read amplification
- B-trees: in-place updates, predictable read latency, write amplification from page splits
- Write amplification (one logical write causing multiple physical writes) matters for SSDs with limited write cycles
- Column-oriented storage dramatically improves analytical queries through compression and vectorized processing
- In-memory databases are fast because they avoid encoding overhead, not because they avoid disk

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **High write throughput** | LSM-tree engine | Cassandra or RocksDB for time-series ingestion at 100K+ writes/sec |
| **Mixed read/write OLTP** | B-tree engine | PostgreSQL B-tree indexes for transactional point lookups |
| **Analytical queries** | Column-oriented storage | ClickHouse or Parquet for scanning billions of rows, few columns |

See [references/storage-engines.md](references/storage-engines.md) when a workload is read/write-bound or you must choose indexes -- adds write/read-path diagrams, compaction strategies, column storage, and a benchmark-driven decision procedure.

### 3. Replication

**Core concept:** Replication keeps copies of data on multiple machines for fault tolerance, scalability, and latency reduction. The core challenge is handling changes consistently.

**Why it works:** Every replication strategy trades off consistency, availability, and latency. Making the trade-off explicit prevents subtle anomalies that surface only under load or failure.

**Key insights:**
- Single-leader: simple, strong consistency possible, but the leader is a bottleneck and single point of failure
- Multi-leader: better write availability across data centers, but complex conflict resolution
- Leaderless: highest availability via quorum reads/writes, but needs careful conflict handling
- Replication lag causes read-your-writes, monotonic-read, and causality violations
- Synchronous replication guarantees durability but adds latency; asynchronous risks data loss on failover
- CRDTs and last-writer-wins resolve conflicts with very different correctness guarantees

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Read-heavy web app** | Single-leader with read replicas | PostgreSQL primary + read replicas behind pgBouncer |
| **Multi-region writes** | Multi-leader replication | CockroachDB or Spanner with bounded staleness |
| **Shopping cart availability** | Leaderless with merge | DynamoDB with last-writer-wins or application-level cart merge |

See [references/replication.md](references/replication.md) when choosing single/multi/leaderless or debugging stale reads -- adds lag anomalies, quorum math, conflict resolution, and CRDTs.

### 4. Partitioning

**Core concept:** Partitioning (sharding) distributes data across nodes so each handles a subset, enabling horizontal scaling beyond a single machine.

**Key insights:**
- Key-range partitioning supports efficient range scans but risks hotspots on sequential keys
- Hash partitioning distributes load evenly but destroys sort order, making range queries expensive
- Local secondary indexes require scatter-gather queries; global secondary indexes require cross-partition updates
- Hotspots occur even with hashing when a single key is extremely popular (celebrity problem)
- Rebalancing strategies: fixed partition count, dynamic splitting, or proportional to nodes

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Time-series data** | Key-range partitioning by time + source | Partition by `(sensor_id, date)` to avoid current-day write hotspot |
| **User data at scale** | Hash partitioning on user ID | Cassandra consistent hashing on `user_id` for even distribution |
| **Celebrity/hot-key problem** | Key splitting with random suffix | Append random digit to hot key, fan out reads across 10 sub-partitions |

See [references/partitioning.md](references/partitioning.md) when sharding or fighting a hot key -- adds rebalancing strategies, request routing, and local-vs-global secondary index trade-offs.

### 5. Transactions and Consistency

**Core concept:** Transactions provide safety guarantees (ACID) that simplify application code by letting you pretend failures and concurrency don't exist -- within the transaction's scope.

**Why it works:** Without transactions, every piece of application code must handle partial failures, races, and concurrent modification. Transactions move that complexity into the database, handled correctly once.

**Key insights:**
- Isolation levels are a spectrum: read uncommitted, read committed, snapshot isolation, serializable
- Most databases default to read committed or snapshot isolation -- NOT serializable -- so you must understand the anomalies this permits
- Write skew: two transactions read the same data, decide, and write different records -- no row lock prevents it
- Serializable snapshot isolation (SSI) gives full serializability optimistically: no blocking, but aborts on conflict; two-phase locking blocks and deadlocks under contention
- Distributed transactions (two-phase commit) are expensive and fragile; design around single-partition operations instead

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Account balance transfer** | Serializable transaction | `BEGIN; UPDATE accounts ... -100 WHERE id=1; UPDATE accounts ... +100 WHERE id=2; COMMIT;` |
| **Inventory reservation** | SELECT FOR UPDATE to prevent write skew | `SELECT stock FROM items WHERE id = X FOR UPDATE` before decrementing |
| **Cross-service operations** | Saga instead of distributed transaction | Charge card, reserve inventory; on failure, run compensating refund |

See [references/transactions.md](references/transactions.md) when setting isolation levels or chasing a concurrency bug -- adds per-isolation anomaly tables, write-skew examples, 2PL vs SSI, and distributed-transaction pitfalls.

### 6. Batch and Stream Processing

**Core concept:** Batch processing transforms bounded datasets in bulk; stream processing transforms unbounded event streams continuously. Both compute derived data.

**Why it works:** Separating the system of record from derived data (caches, indexes, materialized views) lets each be optimized independently and rebuilt from source when requirements change.

**Key insights:**
- MapReduce is conceptually simple but operationally awkward; dataflow engines (Spark, Flink) generalize it with arbitrary DAGs
- Change data capture (CDC) turns database writes into a stream downstream systems can consume
- Stream-table duality: a stream is the changelog of a table; a table is the materialized state of a stream
- Exactly-once semantics require idempotent operations or transactional output
- Time windowing (tumbling, hopping, session) is essential for aggregating unbounded streams

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Daily analytics pipeline** | Batch processing with Spark | Read day's events from S3, aggregate, write to warehouse |
| **Real-time fraud detection** | Stream processing with Flink | Kafka payment events, rules over 5-second tumbling windows |
| **Syncing search index** | Change data capture | Debezium captures PostgreSQL WAL, Kafka feeds Elasticsearch |
| **Audit trail / event replay** | Event sourcing | Store `OrderPlaced`, `OrderShipped` events; rebuild state by replaying |

See [references/batch-stream.md](references/batch-stream.md) when designing a pipeline or deriving data from a system of record -- adds dataflow engines, CDC wiring, windowing, and exactly-once techniques.

### 7. Reliability and Fault Tolerance

**Core concept:** Faults are inevitable; failures are not. A reliable system continues operating correctly even when individual components fail. Design for faults, not against them.

**Key insights:**
- A fault is one component deviating from spec; a failure is the whole system stopping -- fault tolerance prevents the former becoming the latter
- Hardware faults are random and independent; software faults are correlated and systematic (more dangerous)
- Human error is the leading cause of outages -- minimize opportunity for mistakes, maximize ability to recover
- Timeouts are the fundamental fault detector, but tuning is hard: too short causes false positives, too long delays recovery
- Safety properties (nothing bad happens) must always hold; liveness (something good eventually happens) may be temporarily violated
- Byzantine fault tolerance is rarely needed outside blockchain; assume crash-stop or crash-recovery

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Service communication** | Timeouts + retries with backoff | `retry(max=3, backoff=exponential(base=1s, max=30s))` with jitter |
| **Leader election** | Consensus algorithm (Raft/Paxos) | etcd or ZooKeeper for distributed locks and leader election |
| **Graceful degradation** | Circuit breaker | Resilience4j: open circuit after 50% failures in 10-second window |

See [references/fault-tolerance.md](references/fault-tolerance.md) when tuning timeouts/retries or adding consensus -- adds fault classification, timeout-tuning math, Raft/Paxos mechanics, and safety/liveness guarantees.

## Common Mistakes

| Mistake | Why It Fails | Fix |
|---------|-------------|------|
| **Choosing a database by popularity** | Engines have fundamentally different trade-offs | Match storage engine to actual read/write patterns |
| **Ignoring replication lag** | Stale reads, phantom reads, lost updates | Implement read-your-writes and monotonic-read guarantees |
| **Distributed transactions everywhere** | 2PC is slow, fragile; coordinator is a SPOF | Design single-partition operations; use sagas across services |
| **Hash partitioning everything** | Destroys range query ability | Key-range partitioning for time-series; composite keys for locality |
| **Assuming serializable isolation** | Defaults are weaker; write skew appears in production | Check the actual default; use explicit locking where needed |
| **Conflating batch and stream** | Wrong tool adds latency or wasted complexity | Match processing model to data boundedness and latency needs |
| **Treating all faults as recoverable** | Corruption and Byzantine faults need different handling | Classify faults; design a recovery strategy per class |

## Quick Diagnostic

| Question | If No | Action |
|----------|-------|--------|
| Can you explain why you chose this database over alternatives? | Choice was familiarity, not requirements | Evaluate data model fit, read/write ratio, consistency needs, scaling path |
| Do you know your database's default isolation level? | Latent concurrency bugs | Check docs; test for write skew and phantom reads |
| Is your replication strategy explicitly chosen? | Implicit consistency/durability assumptions | Document sync vs async, failover behavior, lag tolerance |
| Can your system handle a hot partition key? | One popular entity can down the cluster | Add key-splitting or load shedding for hot keys |
| Do you separate system of record from derived data? | Every change requires migrating everything | Introduce CDC or event sourcing to decouple |
| Are timeouts and retries tuned, not defaulted? | Cascading failures or needless delays | Measure p99; set timeouts above p99, below cascade threshold |
| Have you tested failover in production conditions? | Recovery plan is theoretical | Run chaos experiments: kill leaders, partition networks, fill disks |

## Further Reading

For the complete treatment with detailed diagrams and research references:

- [*"Designing Data-Intensive Applications"*](https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321?tag=wondelai00-20) by Martin Kleppmann

## About the Author

**Martin Kleppmann** is a distributed-systems researcher at the University of Cambridge and a former engineer at LinkedIn and Rapportive, known for his work on CRDTs and local-first software. His book *Designing Data-Intensive Applications* (2017) is the definitive reference for engineers building data systems, praised for making distributed-systems concepts accessible and practical.