williamzujkowski/Chaos Engineering Experiment Designer

Purpose & When-To-Use

Trigger conditions:

• Resilience testing needed for distributed system or microservices architecture
• Disaster recovery validation beyond traditional testing
• SRE practice adoption requiring systematic failure experimentation
• Production confidence gaps in system behavior under failure conditions
• Pre-deployment validation of fault tolerance mechanisms
• Post-incident chaos engineering to prevent recurrence

Use this skill to:

• Design hypothesis-driven chaos experiments with measurable outcomes
• Define steady-state baselines and deviation thresholds
• Configure controlled failure injection with progressive escalation
• Generate tool-specific experiment configurations (Chaos Mesh, LitmusChaos, Chaos Monkey)
• Establish blast radius controls and abort conditions
• Create reproducible experiment workflows integrated with CI/CD

Do NOT use for:

• Traditional load or performance testing (use testing-strategy-composer)
• Security penetration testing (use security-assessment-framework)
• Functional correctness testing (use testing-strategy-composer)
• One-off manual fault injection without hypothesis or measurement

Pre-Checks

Time normalization:

NOW_ET = 2025-10-25T21:30:36-04:00

Required inputs validation:

• ] system_architecture includes component diagram with dependencies
• ] resilience_goals specify quantitative targets (e.g., 99.9% availability)
• ] experiment_scope defines clear boundaries (services, environments, regions)
• ] existing_monitoring lists available metrics, dashboards, and alerting

Source freshness checks:

• Principles of Chaos Engineering (accessed 2025-10-25T21:30:36-04:00): https://principlesofchaos.org/
• Chaos Mesh v2.x documentation (accessed 2025-10-25T21:30:36-04:00): https://chaos-mesh.org/
• LitmusChaos 3.x framework (accessed 2025-10-25T21:30:36-04:00): https://litmuschaos.io/
• Netflix Chaos Monkey practices (accessed 2025-10-25T21:30:36-04:00): https://netflix.github.io/chaosmonkey/

Abort conditions:

• If system_architecture lacks dependency information → request clarification
• If no monitoring baseline exists → emit TODO: establish steady-state metrics first
• If production environment lacks rollback capabilities → restrict to non-prod only

Procedure

Tier 1: Quick Experiment Design (≤2k tokens)

Fast path for common scenarios:

1. Validate experiment readiness

• Check monitoring baseline exists
• Verify rollback capabilities
• Confirm blast radius boundaries

1. Define steady-state hypothesis

• Identify key user-facing metrics (latency, error rate, throughput)
• Establish normal operating ranges from historical data
• Example: "P95 latency < 200ms AND error rate < 0.1% during business hours"

1. Select failure scenario (common patterns)

• Pod/instance termination (Chaos Monkey pattern)
• Network latency/partition injection
• Resource exhaustion (CPU, memory, I/O)
• Cloud region/availability zone failure

1. Configure minimal experiment

• Start with 1-5% traffic/instances
• 5-minute duration maximum
• Auto-abort if steady state violated by >20%
• Single service scope

1. Output T1 experiment spec

yaml experiment_name: "<service>-<failure-type>-v1" hypothesis: "<steady-state-assertion>" scope: "<service-name> in <environment>" blast_radius: "<percentage> of instances" duration: "5m" abort_conditions: "<steady-state-threshold>"

T1 deliverable: Minimal experiment specification ready for review.

Tier 2: Production-Ready Experiment (≤6k tokens)

Extended validation with tool-specific configuration:

1. Enhanced steady-state definition

• Define multiple observability signals (Golden Signals: latency, traffic, errors, saturation)
• Specify SLO-aligned thresholds
• Include downstream dependency health checks
• Configure Prometheus/Datadog queries for real-time validation

1. Advanced failure scenario design

• Select from chaos engineering taxonomy (accessed 2025-10-25T21:30:36-04:00): https://principlesofchaos.org/
• Infrastructure failures: instance termination, disk failure, network partition
• Application failures: process crash, memory leak simulation, database connection pool exhaustion
• Dependency failures: upstream service degradation, third-party API timeout
• Define progressive escalation path: 1% → 5% → 25% → 50%

1. Blast radius and safety controls

• Geographic boundaries: single AZ, multi-AZ, or multi-region
• Service boundaries: leaf services before core platform services
• Time boundaries: off-peak hours, maintenance windows
• Automated abort triggers:
• Steady state deviation > configured threshold (e.g., 15%)
• Customer-facing SLO breach
• Manual kill switch activation
• Rollback procedures: immediate fault injection termination, traffic rerouting, instance replacement

1. Generate tool-specific configuration

For Kubernetes + Chaos Mesh:

• PodChaos for instance termination
• NetworkChaos for latency/partition injection
• StressChaos for resource exhaustion
• IOChaos for disk failure simulation

For Kubernetes + LitmusChaos:

• ChaosExperiment CRD definition
• ChaosEngine linking workload to fault
• Probes for steady-state validation
• ChaosResult for metrics export

For AWS + Chaos Monkey:

• ASG-scoped termination policies
• Conformity Monkey for architectural validation
• Simian Army integration

1. Monitoring and reporting setup

• Pre-experiment baseline capture (15-30 minutes)
• During-experiment real-time dashboards
• Post-experiment comparison analysis
• Prometheus metrics export for:
• chaos_experiment_duration_seconds
• chaos_steady_state_deviation_percent
• chaos_blast_radius_instances_affected

1. Experiment execution workflow

1. Baseline collection (pre-experiment)
2. Fault injection start
3. Continuous steady-state monitoring
4. Auto-abort on threshold breach OR manual intervention
5. Fault injection termination
6. Recovery validation (post-experiment)
7. Results analysis and report generation

T2 deliverable: Production-ready experiment with tool configs, safety controls, and monitoring integration.

T2 sources:

• Chaos Mesh fault types (accessed 2025-10-25T21:30:36-04:00): https://chaos-mesh.org/ - supports PodChaos, NetworkChaos, IOChaos, TimeChaos, StressChaos
• LitmusChaos CRD architecture (accessed 2025-10-25T21:30:36-04:00): https://litmuschaos.io/ - uses ChaosExperiment, ChaosEngine, ChaosResult custom resources
• Netflix best practices (accessed 2025-10-25T21:30:36-04:00): Start small (single node), enable monitoring first, gradual escalation, automate over time
• Google Cloud chaos engineering (accessed 2025-10-25T21:30:36-04:00): Build hypothesis around steady state, replicate real-world conditions, minimize blast radius

Tier 3: Advanced Experiment Suite (≤12k tokens)

Comprehensive resilience validation (use only when explicitly requested):

1. Multi-dimensional experiment matrix

• Combine failure modes: network partition + instance termination
• Cascade scenarios: upstream dependency failure → downstream impact
• Time-based variations: gradual degradation vs sudden failure
• Geographic distribution: multi-region failover validation

1. Automated experiment pipelines

• CI/CD integration for continuous chaos testing
• GameDay automation with scheduled experiment runs
• Regression testing for resilience (post-deployment validation)

1. Advanced metrics and analysis

• MTTR (Mean Time To Recovery) calculation
• Blast radius expansion rate
• Failure propagation graph
• Resilience score calculation

1. Org-wide chaos engineering program

• Skill development and training plans
• Runbook generation from experiments
• Blameless postmortem templates
• Chaos engineering maturity assessment

T3 deliverable: Enterprise-scale chaos engineering program with automation, metrics, and cultural integration.

Decision Rules

Experiment scope selection:

• If system is new (<6 months in production) → T1 minimal experiment, non-production only
• If system has established monitoring + SLOs → T2 production experiment with 1-5% blast radius
• If mature resilience practice exists → T2 with progressive escalation to 25-50%
• If multi-team coordination needed → T3 with GameDay orchestration

Tool selection:

• If Kubernetes-native deployment → prefer Chaos Mesh or LitmusChaos
• If AWS EC2/ASG workloads → consider Chaos Monkey or AWS Fault Injection Simulator
• If multi-cloud or hybrid → Chaos Mesh (cloud-agnostic) or Gremlin (SaaS)
• If budget constraints → open-source LitmusChaos or Chaos Mesh over commercial Gremlin

Safety thresholds:

• Abort if: Steady-state deviation >15% OR customer SLO breach OR manual intervention
• Start conservatively: 1-5% blast radius, 5-minute duration
• Escalate gradually: 2x blast radius per iteration if previous experiment passed
• Production readiness gate: 3+ successful non-production experiments before production testing

Ambiguity handling:

• If steady-state metrics unclear → work with SRE/ops to define; emit TODO list
• If blast radius boundaries ambiguous → default to most conservative (1%, single AZ, leaf services)
• If rollback procedures undefined → restrict to non-production until procedures documented

Output Contract

Primary output: experiment_plan (JSON)

json
{
  "experiment_id": "string (unique identifier)",
  "hypothesis": {
    "steady_state": "string (measurable assertion)",
    "metrics": [
      {
        "name": "string (e.g., p95_latency_ms)",
        "baseline": "number (historical average)",
        "threshold": "number (max acceptable deviation)"
      }
    ]
  },
  "failure_injection": {
    "type": "string (pod-kill|network-delay|cpu-stress|region-failure)",
    "target": "string (service/component name)",
    "parameters": "object (tool-specific config)"
  },
  "blast_radius": {
    "scope": "string (service|AZ|region)",
    "percentage": "number (1-100)",
    "max_instances": "number"
  },
  "duration": "string (ISO 8601 duration, e.g., PT5M)",
  "abort_conditions": [
    "string (condition triggering experiment termination)"
  ],
  "rollback_procedure": "string (steps to restore normal state)"
}

Secondary output: implementation_config (tool-specific YAML/JSON)

• Chaos Mesh: PodChaos, NetworkChaos, or StressChaos YAML manifest
• LitmusChaos: ChaosExperiment, ChaosEngine CRDs
• Chaos Monkey: Configuration properties or API payloads

Tertiary output: safety_controls (checklist)

• ] Monitoring dashboards configured
• ] Alerting thresholds set
• ] Rollback runbook accessible
• ] Stakeholder notification plan
• ] Manual abort procedure documented
• ] Post-experiment cleanup steps defined

Required fields: All JSON schema fields above are mandatory. Missing fields → skill emits TODO and stops.

Examples

Example: Pod termination experiment for payment service

yaml
# Input
system_architecture: "Payment service (3 replicas) → Database (RDS)"
resilience_goals: "99.9% availability, P95 latency <200ms"
experiment_scope: "Payment service pods in staging, 1 pod max"
existing_monitoring: "Prometheus + Grafana, payment_request_duration_ms"

# Output (Chaos Mesh PodChaos)
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: payment-pod-kill-exp
  namespace: staging
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - staging
    labelSelectors:
      app: payment-service
  duration: 5m
  scheduler:
    cron: "@every 1h"  # Automated GameDay

Quality Gates

Token budgets (enforced):

• T1: ≤2000 tokens (minimal experiment spec)
• T2: ≤6000 tokens (production-ready with tool config)
• T3: ≤12000 tokens (advanced suite + program design)

Safety requirements:

• Every experiment MUST define abort conditions
• Blast radius MUST be explicitly bounded
• Production experiments REQUIRE successful non-prod validation first

Auditability:

• All experiments logged with timestamp, executor, and results
• Changes to experiment parameters tracked in version control
• Results exported to observability platform (Prometheus/Datadog)

Determinism:

• Same experiment specification → reproducible results (within statistical variance)
• Randomized failure injection uses seeded RNG for replay capability

Quality checklist:

• ] Steady-state hypothesis is measurable and falsifiable
• ] Failure injection reflects real-world scenarios
• ] Blast radius minimizes customer impact
• ] Monitoring captures experiment success/failure
• ] Rollback procedure tested and documented

Resources

Official documentation:

• Principles of Chaos Engineering (accessed 2025-10-25T21:30:36-04:00): https://principlesofchaos.org/
• Chaos Mesh documentation (accessed 2025-10-25T21:30:36-04:00): https://chaos-mesh.org/
• LitmusChaos framework (accessed 2025-10-25T21:30:36-04:00): https://litmuschaos.io/
• Netflix Chaos Monkey (accessed 2025-10-25T21:30:36-04:00): https://netflix.github.io/chaosmonkey/

Templates and examples:

• See resources/experiment-template.yaml for full experiment specification
• See resources/blast-radius-config.json for safety boundary examples

Related skills:

• cloud-native-deployment-orchestrator - for understanding Kubernetes deployment topology
• devops-pipeline-architect - for CI/CD integration of chaos experiments
• observability-slo-calculator - for defining steady-state thresholds aligned with SLOs