Kubernetes Operations

Purpose

Operating Kubernetes clusters in production requires mastery of resource management, scheduling patterns, networking architecture, storage strategies, security hardening, and autoscaling. This skill provides operations-first frameworks for right-sizing workloads, implementing high-availability patterns, securing clusters with RBAC and Pod Security Standards, and systematically troubleshooting common failures.

Use this skill when deploying applications to Kubernetes, configuring cluster resources, implementing NetworkPolicies for zero-trust security, setting up autoscaling (HPA, VPA, KEDA), managing persistent storage, or diagnosing operational issues like CrashLoopBackOff or resource exhaustion.

When to Use This Skill

Common Triggers:

• "Deploy my application to Kubernetes"
• "Configure resource requests and limits"
• "Set up autoscaling for my pods"
• "Implement NetworkPolicies for security"
• "My pod is stuck in Pending/CrashLoopBackOff"
• "Configure RBAC with least privilege"
• "Set up persistent storage for my database"
• "Spread pods across availability zones"

Operations Covered:

• Resource management (CPU/memory, QoS classes, quotas)
• Advanced scheduling (affinity, taints, topology spread)
• Networking (NetworkPolicies, Ingress, Gateway API)
• Storage operations (StorageClasses, PVCs, CSI)
• Security hardening (RBAC, Pod Security Standards, policies)
• Autoscaling (HPA, VPA, KEDA, cluster autoscaler)
• Troubleshooting (systematic debugging playbooks)

Resource Management

Quality of Service (QoS) Classes

Kubernetes assigns QoS classes based on resource requests and limits:

Guaranteed (Highest Priority):

• Requests equal limits for CPU and memory
• Never evicted unless exceeding limits
• Use for critical production services

yaml
resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "512Mi"  # Same as request
    cpu: "500m"

Burstable (Medium Priority):

• Requests less than limits (or only requests set)
• Can burst above requests
• Evicted under node pressure
• Use for web servers, most applications

yaml
resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"  # 2x request
    cpu: "500m"

BestEffort (Lowest Priority):

• No requests or limits set
• First to be evicted under pressure
• Use only for development/testing

Decision Framework: Which QoS Class?

| Workload Type | QoS Class | Configuration | |---------------|-----------|---------------| | Critical API/Database | Guaranteed | requests == limits | | Web servers, services | Burstable | limits 1.5-2x requests | | Batch jobs | Burstable | Low requests, high limits | | Dev/test environments | BestEffort | No limits |

Resource Quotas and LimitRanges

Enforce multi-tenancy with ResourceQuotas (namespace limits) and LimitRanges (per-container defaults):

yaml
# ResourceQuota: Namespace-level limits
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    pods: "50"

For detailed resource management patterns including Vertical Pod Autoscaler (VPA), see references/resource-management.md.

Advanced Scheduling

Node Affinity

Control which nodes pods schedule on with required (hard) or preferred (soft) constraints:

yaml
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
          - g4dn.xlarge  # GPU instance

Taints and Tolerations

Reserve nodes for specific workloads (inverse of affinity):

bash
# Taint GPU nodes to prevent non-GPU workloads
kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule

yaml
# Pod tolerates GPU taint
tolerations:
- key: "workload"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"

Topology Spread Constraints

Distribute pods evenly across failure domains (zones, nodes):

yaml
topologySpreadConstraints:
- maxSkew: 1  # Max difference in pod count
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: critical-app

For advanced scheduling patterns including pod priority and preemption, see references/scheduling-patterns.md.

Networking

NetworkPolicies (Zero-Trust Security)

Implement default-deny security with NetworkPolicies:

yaml
# Default deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

yaml
# Allow specific ingress (frontend → backend)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-allow-frontend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Ingress vs. Gateway API

Ingress (Legacy):

• Widely supported, mature ecosystem
• Limited expressiveness
• Use for existing applications

Gateway API (Modern):

• Role-oriented design (cluster ops vs. app devs)
• More expressive (HTTPRoute, TCPRoute, TLSRoute)
• Recommended for new applications (GA in Kubernetes 1.29+)

yaml
# Gateway API example
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-routes
spec:
  parentRefs:
  - name: production-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: backend
      port: 8080

For detailed networking patterns including service mesh integration, see references/networking.md.

Storage

StorageClasses (Define Performance Tiers)

StorageClasses define storage tiers for different workload needs:

yaml
# AWS EBS SSD (high performance)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iopsPerGB: "50"
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete

Storage Decision Matrix

| Workload | Performance | Access Mode | Storage Class | |----------|-------------|-------------|---------------| | Database | High | ReadWriteOnce | SSD (gp3/io2) | | Shared files | Medium | ReadWriteMany | NFS/EFS | | Logs (temp) | Low | ReadWriteOnce | Standard HDD | | ML models | High | ReadOnlyMany | Object storage (S3) |

Access Modes:

• ReadWriteOnce (RWO): Single node read-write (most common)
• ReadOnlyMany (ROX): Multiple nodes read-only
• ReadWriteMany (RWX): Multiple nodes read-write (requires network storage)

For detailed storage operations including volume snapshots and CSI drivers, see references/storage.md.

Security

RBAC (Role-Based Access Control)

Implement least-privilege access with RBAC:

yaml
# Role (namespace-scoped)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
---
# RoleBinding (assign role to user)
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: production
subjects:
- kind: User
  name: jane@example.com
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Pod Security Standards

Enforce secure pod configurations at the namespace level:

yaml
# Namespace with Restricted PSS (most secure)
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Pod Security Levels:

• Restricted: Most secure, removes all privilege escalations (use for applications)
• Baseline: Minimally restrictive, prevents known escalations
• Privileged: Unrestricted (only for system workloads)

For detailed security patterns including policy enforcement (Kyverno/OPA) and secrets management, see references/security.md.

Autoscaling

Horizontal Pod Autoscaler (HPA)

Scale pod replicas based on CPU, memory, or custom metrics:

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5min before scaling down

KEDA (Event-Driven Autoscaling)

Scale based on events beyond CPU/memory (queues, cron schedules, Prometheus metrics):

yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: rabbitmq-scaler
spec:
  scaleTargetRef:
    name: message-processor
  minReplicaCount: 0   # Scale to zero when queue empty
  maxReplicaCount: 30
  triggers:
  - type: rabbitmq
    metadata:
      queueName: tasks
      queueLength: "10"  # Scale up when >10 messages

Autoscaling Decision Matrix

| Scenario | Use HPA | Use VPA | Use KEDA | Use Cluster Autoscaler | |----------|---------|---------|----------|------------------------| | Stateless web app with traffic spikes | ✅ | ❌ | ❌ | Maybe | | Single-instance database | ❌ | ✅ | ❌ | Maybe | | Queue processor (event-driven) | ❌ | ❌ | ✅ | Maybe | | Pods pending (insufficient nodes) | ❌ | ❌ | ❌ | ✅ |

For detailed autoscaling patterns including VPA and cluster autoscaler configuration, see references/autoscaling.md.

Troubleshooting

Common Pod Issues

Pod Stuck in Pending:

bash
kubectl describe pod <pod-name>

# Common causes:
# - Insufficient CPU/memory: Reduce requests or add nodes
# - Node selector mismatch: Fix nodeSelector or add labels
# - PVC not bound: Create PVC or fix name
# - Taint intolerance: Add toleration or remove taint

CrashLoopBackOff:

bash
kubectl logs <pod-name>
kubectl logs <pod-name> --previous  # Check previous crash

# Common causes:
# - Application crash: Fix code or configuration
# - Missing environment variables: Add to deployment
# - Liveness probe failing: Increase initialDelaySeconds
# - OOMKilled: Increase memory limit or fix leak

ImagePullBackOff:

bash
kubectl describe pod <pod-name>

# Common causes:
# - Image doesn't exist: Fix image name/tag
# - Authentication required: Create imagePullSecrets
# - Network issues: Check NetworkPolicies, firewall rules

Service Not Accessible:

bash
kubectl get endpoints <service-name>  # Should list pod IPs

# If endpoints empty:
# - Service selector doesn't match pod labels
# - Pods aren't ready (readiness probe failing)
# - Check NetworkPolicies blocking traffic

For systematic troubleshooting playbooks including networking and storage issues, see references/troubleshooting.md.

Reference Documentation

Deep Dives

• references/resource-management.md - Resource requests/limits, QoS classes, ResourceQuotas, VPA
• references/scheduling-patterns.md - Node affinity, taints/tolerations, topology spread, priority
• references/networking.md - NetworkPolicies, Ingress, Gateway API, service mesh integration
• references/storage.md - StorageClasses, PVCs, CSI drivers, volume snapshots
• references/security.md - RBAC, Pod Security Standards, policy enforcement, secrets
• references/autoscaling.md - HPA, VPA, KEDA, cluster autoscaler configuration
• references/troubleshooting.md - Systematic debugging playbooks for common failures

Examples

• examples/manifests/ - Copy-paste ready YAML manifests
• examples/python/ - Automation scripts (audit, cost analysis, validation)
• examples/go/ - Operator development examples

Tools

• scripts/validate-resources.sh - Audit pods without resource limits
• scripts/audit-networkpolicies.sh - Find namespaces without NetworkPolicies
• scripts/cost-analysis.sh - Resource cost breakdown by namespace

Related Skills

• building-ci-pipelines - Deploy to Kubernetes from CI/CD (kubectl apply, Helm, GitOps)
• observability - Monitor clusters and workloads (Prometheus, Grafana, tracing)
• secret-management - Secure secrets in Kubernetes (External Secrets, Sealed Secrets)
• testing-strategies - Test manifests and deployments (Kubeval, Conftest, Kind)
• infrastructure-as-code - Provision Kubernetes clusters (Terraform, Cluster API)
• gitops-workflows - Declarative cluster management (Flux, ArgoCD)

Best Practices Summary

Resource Management:

• Always set CPU/memory requests and limits
• Use VPA for automated rightsizing
• Implement resource quotas per namespace
• Monitor actual usage vs. requests

Scheduling:

• Use topology spread constraints for high availability
• Apply taints for workload isolation (GPU, spot instances)
• Set pod priority for critical workloads

Networking:

• Implement NetworkPolicies with default-deny
• Use Gateway API for new applications
• Apply rate limiting at ingress layer

Storage:

• Use CSI drivers (not legacy provisioners)
• Define StorageClasses per performance tier
• Enable volume snapshots for stateful apps

Security:

• Enforce Pod Security Standards (Restricted for apps)
• Implement RBAC with least privilege
• Use policy engines for guardrails (Kyverno/OPA)
• Scan images for vulnerabilities

Autoscaling:

• Use HPA for stateless workloads
• Use KEDA for event-driven workloads
• Enable cluster autoscaler with limits
• Set PodDisruptionBudgets to prevent over-disruption