Mmcp.market

ddia-systems skill

by wondelai·wondelai/skills·2.3k stars·MIT

Design data systems by understanding storage engines, replication, partitioning, transactions, and consistency models. Use when the user mentions "database choice", "which database should I use", "SQL or NoSQL", "replication lag", "partitioning strategy", "consistency vs availability", "stream processing", "ACID transactions", "eventual consistency", "my queries are slow at scale", or "data is inconsistent across replicas". Also trigger when choosing a datastore, designing data pipelines, or debugging distributed-system consistency issues. Covers data models, batch/stream processing, and distributed consensus. For system design, see system-design. For resilience, see release-it.

A100/100content scan

Is the ddia-systems skill safe?

Clean: nothing in its files matched our rules. We read 8 files in the folder on 2026-09-28.

No findings.

Install the ddia-systems skill

A skill is a folder. Copy it into your agent's skills folder and the agent loads it when the task matches its description. 

git clone --depth 1 https://github.com/wondelai/skills.git /tmp/skills
mkdir -p ~/.claude/skills
cp -r /tmp/skills/plugins/wondelai-skills/skills/ddia-systems ~/.claude/skills/ddia-systems
available in every project

In the Claude apps, zip the folder and upload it from the Skills settings. The folder on GitHub

The instructions your agent would load

SKILL.md as published, without the frontmatter. Read it on GitHub

Designing Data-Intensive Applications Framework

A principled approach to building reliable, scalable, and maintainable data systems. Apply these principles when choosing databases, designing schemas, architecting distributed systems, or reasoning about consistency and fault tolerance.

Core Principle

Data outlives code. Applications are rewritten and frameworks come and go, but data persists for decades -- prioritize the long-term correctness, durability, and evolvability of the data layer. Most applications are data-intensive, not compute-intensive: the hard problems are data volume, complexity, and rate of change, and explicit consistency/availability/latency trade-offs separate robust systems from fragile ones.

Scoring

Goal: 10/10. Score a data architecture by the seven Quick Diagnostic rows below: award ~1.4 points per row answered "yes" with evidence (deliberate, documented trade-off), 0 where the answer is "no" or unknown.

  • 9-10: every domain choice -- data model, storage engine, replication, partitioning, isolation, derived-data, fault handling -- is deliberate, documented, and matched to actual read/write/consistency requirements; failover tested.
  • 5-6: core choices made but two or three diagnostic rows fail -- e.g. default isolation level unknown, hot-key risk unhandled, or failover untested.
  • <=3: choices driven by familiarity, not requirements; ignored failure modes (replication lag, write skew, hot partitions) and accidental complexity dominate.

Report the current score, which diagnostic rows failed, and the improvements needed to reach 10/10.

The DDIA Framework

Seven domains for reasoning about data-intensive systems:

1. Data Models and Query Languages

Core concept: The data model shapes how you think about the problem. Relational, document, and graph models each impose different constraints and enable different query patterns.

Why it works: Choosing the wrong data model forces application code to compensate for representational mismatch, adding accidental complexity that compounds over time.

Key insights:

  • Relational models excel at many-to-many relationships and ad-hoc queries; document models at one-to-many relationships and locality; graph models at recursive traversals over interconnected data
  • Schema-on-write (relational) catches errors early; schema-on-read (document) offers flexibility
  • Polyglot persistence -- different stores for different access patterns -- is often the right answer
  • Object-relational impedance mismatch is a real cost; document models reduce it for self-contained aggregates

Code applications:

See references/data-models.md when picking relational vs document vs graph or evaluating schema-on-read -- adds the full trade-off matrix and query-language comparisons.

2. Storage Engines

Core concept: Storage engines trade off read performance against write performance. Log-structured engines (LSM trees) optimize writes; page-oriented engines (B-trees) balance reads and writes.

Key insights:

  • LSM trees: append-only writes, periodic compaction, excellent write throughput, higher read amplification
  • B-trees: in-place updates, predictable read latency, write amplification from page splits
  • Write amplification (one logical write causing multiple physical writes) matters for SSDs with limited write cycles
  • Column-oriented storage dramatically improves analytical queries through compression and vectorized processing
  • In-memory databases are fast because they avoid encoding overhead, not because they avoid disk

Code applications:

See references/storage-engines.md when a workload is read/write-bound or you must choose indexes -- adds write/read-path diagrams, compaction strategies, column storage, and a benchmark-driven decision procedure.

3. Replication

Core concept: Replication keeps copies of data on multiple machines for fault tolerance, scalability, and latency reduction. The core challenge is handling changes consistently.

Why it works: Every replication strategy trades off consistency, availability, and latency. Making the trade-off explicit prevents subtle anomalies that surface only under load or failure.

Key insights:

  • Single-leader: simple, strong consistency possible, but the leader is a bottleneck and single point of failure
  • Multi-leader: better write availability across data centers, but complex conflict resolution
  • Leaderless: highest availability via quorum reads/writes, but needs careful conflict handling
  • Replication lag causes read-your-writes, monotonic-read, and causality violations
  • Synchronous replication guarantees durability but adds latency; asynchronous risks data loss on failover
  • CRDTs and last-writer-wins resolve conflicts with very different correctness guarantees

Code applications:

See references/replication.md when choosing single/multi/leaderless or debugging stale reads -- adds lag anomalies, quorum math, conflict resolution, and CRDTs.

4. Partitioning

Core concept: Partitioning (sharding) distributes data across nodes so each handles a subset, enabling horizontal scaling beyond a single machine.

Key insights:

  • Key-range partitioning supports efficient range scans but risks hotspots on sequential keys
  • Hash partitioning distributes load evenly but destroys sort order, making range queries expensive
  • Local secondary indexes require scatter-gather queries; global secondary indexes require cross-partition updates
  • Hotspots occur even with hashing when a single key is extremely popular (celebrity problem)
  • Rebalancing strategies: fixed partition count, dynamic splitting, or proportional to nodes

Code applications:

See references/partitioning.md when sharding or fighting a hot key -- adds rebalancing strategies, request routing, and local-vs-global secondary index trade-offs.

5. Transactions and Consistency

Core concept: Transactions provide safety guarantees (ACID) that simplify application code by letting you pretend failures and concurrency don't exist -- within the transaction's scope.

Why it works: Without transactions, every piece of application code must handle partial failures, races, and concurrent modification. Transactions move that complexity into the database, handled correctly once.

Key insights:

  • Isolation levels are a spectrum: read uncommitted, read committed, snapshot isolation, serializable
  • Most databases default to read committed or snapshot isolation -- NOT serializable -- so you must understand the anomalies this permits
  • Write skew: two transactions read the same data, decide, and write different records -- no row lock prevents it
  • Serializable snapshot isolation (SSI) gives full serializability optimistically: no blocking, but aborts on conflict; two-phase locking blocks and deadlocks under contention
  • Distributed transactions (two-phase commit) are expensive and fragile; design around single-partition operations instead

Code applications:

See references/transactions.md when setting isolation levels or chasing a concurrency bug -- adds per-isolation anomaly tables, write-skew examples, 2PL vs SSI, and distributed-transaction pitfalls.

6. Batch and Stream Processing

Core concept: Batch processing transforms bounded datasets in bulk; stream processing transforms unbounded event streams continuously. Both compute derived data.

Why it works: Separating the system of record from derived data (caches, indexes, materialized views) lets each be optimized independently and rebuilt from source when requirements change.

Key insights:

  • MapReduce is conceptually simple but operationally awkward; dataflow engines (Spark, Flink) generalize it with arbitrary DAGs
  • Change data capture (CDC) turns database writes into a stream downstream systems can consume
  • Stream-table duality: a stream is the changelog of a table; a table is the materialized state of a stream
  • Exactly-once semantics require idempotent operations or transactional output
  • Time windowing (tumbling, hopping, session) is essential for aggregating unbounded streams

Code applications:

See references/batch-stream.md when designing a pipeline or deriving data from a system of record -- adds dataflow engines, CDC wiring, windowing, and exactly-once techniques.

7. Reliability and Fault Tolerance

Core concept: Faults are inevitable; failures are not. A reliable system continues operating correctly even when individual components fail. Design for faults, not against them.

Key insights:

  • A fault is one component deviating from spec; a failure is the whole system stopping -- fault tolerance prevents the former becoming the latter
  • Hardware faults are random and independent; software faults are correlated and systematic (more dangerous)
  • Human error is the leading cause of outages -- minimize opportunity for mistakes, maximize ability to recover
  • Timeouts are the fundamental fault detector, but tuning is hard: too short causes false positives, too long delays recovery
  • Safety properties (nothing bad happens) must always hold; liveness (something good eventually happens) may be temporarily violated
  • Byzantine fault tolerance is rarely needed outside blockchain; assume crash-stop or crash-recovery

Code applications:

See references/fault-tolerance.md when tuning timeouts/retries or adding consensus -- adds fault classification, timeout-tuning math, Raft/Paxos mechanics, and safety/liveness guarantees.

Common Mistakes

Quick Diagnostic

Further Reading

For the complete treatment with detailed diagrams and research references:

More skills from wondelai/skills

  • A37signals-wayBuild lean, opinionated products using the 37signals philosophy from "Getting Real", "Rework", and "Shape Up". Use when the user mentions "Getting Real", "Rework", "Shape Up", "37signals", "Basecamp method", "six-week cycles", "fixed time variable scope", "appetite vs estimates", "betting table", "breadboarding", "fat marker sketch", "build less", "underdo the competition", "opinionated software", "we have too many meetings", "how do we ship faster", or "stop overbuilding". Also trigger when cutting scope to ship sooner, running a small team, or avoiding long-term roadmaps. Covers shaping, betting, building, and the art of saying no. For MVP validation, see lean-startup. For design sprints, see design-sprint.
  • A37signals-wayBuild lean, opinionated products using the 37signals philosophy from "Getting Real", "Rework", and "Shape Up". Use when the user mentions "Getting Real", "Rework", "Shape Up", "37signals", "Basecamp method", "six-week cycles", "fixed time variable scope", "appetite vs estimates", "betting table", "breadboarding", "fat marker sketch", "build less", "underdo the competition", "opinionated software", "we have too many meetings", "how do we ship faster", or "stop overbuilding". Also trigger when cutting scope to ship sooner, running a small team, or avoiding long-term roadmaps. Covers shaping, betting, building, and the art of saying no. For MVP validation, see lean-startup. For design sprints, see design-sprint.
  • A37signals-wayBuild lean, opinionated products using the 37signals philosophy from "Getting Real", "Rework", and "Shape Up". Use when the user mentions "Getting Real", "Rework", "Shape Up", "37signals", "Basecamp method", "six-week cycles", "fixed time variable scope", "appetite vs estimates", "betting table", "breadboarding", "fat marker sketch", "build less", "underdo the competition", "opinionated software", "we have too many meetings", "how do we ship faster", or "stop overbuilding". Also trigger when cutting scope to ship sooner, running a small team, or avoiding long-term roadmaps. Covers shaping, betting, building, and the art of saying no. For MVP validation, see lean-startup. For design sprints, see design-sprint.
  • Aarchitecture-optimizationGuided journey from a working codebase grown slow and tangled to one measurably fast, cleanly bounded, and readable. Orchestrates eight skills phase by phase - working-with-legacy-code, clean-architecture, software-design-philosophy, refactoring-patterns, system-design, ddia-systems, release-it, pragmatic-programmer - every phase carries its method inline so it runs standalone, asking the user questions at every decision point and recording results in the project docs/ folder (PERFORMANCE.md, ARCHITECTURE.md, ARCHITECTURE-OPTIMIZATION-PLAN.md) so the journey resumes across sessions. Use when the user wants to make an app faster, untangle drifted boundaries, fix slow endpoints and queries, or says ''it works but it is slow and getting worse''. For an untested prototype, improve-code-quality; for an aged codebase you fear to touch, remove-technical-debt; for greenfield structure, design-code-architecture; for marketing-site page speed, improve-website. For one framework in isolation, invoke that skill directly.
  • Aarchitecture-optimizationGuided journey from a working codebase grown slow and tangled to one measurably fast, cleanly bounded, and readable. Orchestrates eight skills phase by phase - working-with-legacy-code, clean-architecture, software-design-philosophy, refactoring-patterns, system-design, ddia-systems, release-it, pragmatic-programmer - every phase carries its method inline so it runs standalone, asking the user questions at every decision point and recording results in the project docs/ folder (PERFORMANCE.md, ARCHITECTURE.md, ARCHITECTURE-OPTIMIZATION-PLAN.md) so the journey resumes across sessions. Use when the user wants to make an app faster, untangle drifted boundaries, fix slow endpoints and queries, or says ''it works but it is slow and getting worse''. For an untested prototype, improve-code-quality; for an aged codebase you fear to touch, remove-technical-debt; for greenfield structure, design-code-architecture; for marketing-site page speed, improve-website. For one framework in isolation, invoke that skill directly.
  • Aarchitecture-optimizationGuided journey from a working codebase grown slow and tangled to one measurably fast, cleanly bounded, and readable. Orchestrates eight skills phase by phase - working-with-legacy-code, clean-architecture, software-design-philosophy, refactoring-patterns, system-design, ddia-systems, release-it, pragmatic-programmer - every phase carries its method inline so it runs standalone, asking the user questions at every decision point and recording results in the project docs/ folder (PERFORMANCE.md, ARCHITECTURE.md, ARCHITECTURE-OPTIMIZATION-PLAN.md) so the journey resumes across sessions. Use when the user wants to make an app faster, untangle drifted boundaries, fix slow endpoints and queries, or says ''it works but it is slow and getting worse''. For an untested prototype, improve-code-quality; for an aged codebase you fear to touch, remove-technical-debt; for greenfield structure, design-code-architecture; for marketing-site page speed, improve-website. For one framework in isolation, invoke that skill directly.
  • Ablue-ocean-strategyCreate uncontested market space using value innovation instead of competing head-to-head. Use when the user mentions "blue ocean", "red ocean", "strategy canvas", "ERRC framework", "value innovation", "non-customers", "buyer utility map", "the market is too crowded", "how do we stand out", or "escape the price war". Also trigger when exploring a new market category, or finding underserved or non-customers. Covers the Four Actions Framework, Six Paths, buyer utility map, and value-cost trade-offs. For real strategy formulation and bad-strategy detection, see good-strategy-bad-strategy. For tech adoption strategy, see crossing-the-chasm. For product positioning, see obviously-awesome.
  • Ablue-ocean-strategyCreate uncontested market space using value innovation instead of competing head-to-head. Use when the user mentions "blue ocean", "red ocean", "strategy canvas", "ERRC framework", "value innovation", "non-customers", "buyer utility map", "the market is too crowded", "how do we stand out", or "escape the price war". Also trigger when exploring a new market category, or finding underserved or non-customers. Covers the Four Actions Framework, Six Paths, buyer utility map, and value-cost trade-offs. For real strategy formulation and bad-strategy detection, see good-strategy-bad-strategy. For tech adoption strategy, see crossing-the-chasm. For product positioning, see obviously-awesome.
  • Ablue-ocean-strategyCreate uncontested market space using value innovation instead of competing head-to-head. Use when the user mentions "blue ocean", "red ocean", "strategy canvas", "ERRC framework", "value innovation", "non-customers", "buyer utility map", "the market is too crowded", "how do we stand out", or "escape the price war". Also trigger when exploring a new market category, or finding underserved or non-customers. Covers the Four Actions Framework, Six Paths, buyer utility map, and value-cost trade-offs. For real strategy formulation and bad-strategy detection, see good-strategy-bad-strategy. For tech adoption strategy, see crossing-the-chasm. For product positioning, see obviously-awesome.
  • Aclean-architectureStructure software around the Dependency Rule: source code dependencies point inward from frameworks to use cases to entities. Use when the user mentions "architecture layers", "dependency rule", "ports and adapters (hexagonal)", "onion architecture", "screaming architecture", "where should business logic go", "decouple from the database", "swap the framework without a rewrite", or "keep business rules independent". Also trigger when deciding which layer code belongs in, isolating core logic from infrastructure, defining module boundaries, or debating whether the framework should call your code or the reverse. Covers component principles, boundaries, and SOLID. For code-level quality, see clean-code. For domain modeling, see domain-driven-design.
  • Aclean-architectureStructure software around the Dependency Rule: source code dependencies point inward from frameworks to use cases to entities. Use when the user mentions "architecture layers", "dependency rule", "ports and adapters (hexagonal)", "onion architecture", "screaming architecture", "where should business logic go", "decouple from the database", "swap the framework without a rewrite", or "keep business rules independent". Also trigger when deciding which layer code belongs in, isolating core logic from infrastructure, defining module boundaries, or debating whether the framework should call your code or the reverse. Covers component principles, boundaries, and SOLID. For code-level quality, see clean-code. For domain modeling, see domain-driven-design.
  • Aclean-architectureStructure software around the Dependency Rule: source code dependencies point inward from frameworks to use cases to entities. Use when the user mentions "architecture layers", "dependency rule", "ports and adapters (hexagonal)", "onion architecture", "screaming architecture", "where should business logic go", "decouple from the database", "swap the framework without a rewrite", or "keep business rules independent". Also trigger when deciding which layer code belongs in, isolating core logic from infrastructure, defining module boundaries, or debating whether the framework should call your code or the reverse. Covers component principles, boundaries, and SOLID. For code-level quality, see clean-code. For domain modeling, see domain-driven-design.

All agent skills → · MCP servers