---
name: lloydchang/troubleshoot-kubernetes
source: https://app.decimal.ai/s/lloydchang-troubleshoot-kubernetes@1/SKILL.md
source_sha256: a5c64870dc47
---

# Troubleshoot Kubernetes — Multi-Cloud Enterprise Automation

## Purpose
Enterprise-grade automation solution for k8s troubleshoot operations across AWS, Azure, GCP, and on-premise environments to maximize operational efficiency while maintaining security and compliance standards.

## When to Use
- **Pod Failures**: When pods are stuck in CrashLoopBackOff, pending, or failing to start
- **Network Issues**: When services can't communicate or have connectivity problems
- **Resource Problems**: When nodes are resource constrained or applications are throttled
- **Cluster Health**: When cluster components are unhealthy or degraded
- **Performance Issues**: When applications are slow or have high latency
- **Storage Problems**: When persistent volumes aren't mounting or have issues
- **DNS Resolution**: When name resolution isn't working within the cluster

## Gotchas

### Common Pitfalls
- **Symptom vs Cause**: Kubernetes symptoms often don't directly indicate root causes
- **Resource Requests**: Missing or incorrect resource requests can cause scheduling issues
- **Image Pull Issues**: Registry access problems can cause pod startup failures
- **Namespace Isolation**: Issues in one namespace may affect cluster-wide resources

### Edge Cases
- **Cluster Autoscaling**: Autoscaling events can cause temporary resource unavailability
- **Node Drain**: Node maintenance can cause pod rescheduling and temporary issues
- **Certificate Rotation**: Certificate rotation can cause temporary connectivity issues
- **Multi-Tenant Clusters**: Resource contention between tenants can cause complex issues

### Performance Issues
- **Cluster Size**: Large clusters (>1000 nodes) can have API server latency
- **Watch Connections**: Too many watch connections can overwhelm the API server
- **Log Volume**: High log volume can affect cluster performance and troubleshooting
- **Resource Metrics**: Collecting detailed metrics can impact cluster performance

### Security Considerations
- **Privileged Escalation**: Troubleshooting may require elevated permissions
- **Sensitive Data**: Logs and diagnostics may contain sensitive information
- **Cluster Access**: Troubleshooting tools need appropriate RBAC permissions
- **Audit Logs**: All troubleshooting actions should be logged for security compliance

### Troubleshooting
- **API Server Access**: Network issues may prevent access to the Kubernetes API
- **Tool Compatibility**: Different kubectl versions may have compatibility issues
- **Context Confusion**: Multiple kubeconfig contexts can cause commands to run on wrong cluster
- **Permission Errors**: Insufficient permissions can prevent diagnostic commands

## Inputs
- **operation**: Operation type (required)
- **targetResource**: Target resource identifier (required)
- **cloudProvider**: Cloud provider - `aws|azure|gcp|onprem|all` (optional, default: `all`)
- **parameters**: Operation-specific parameters (optional)
- **environment**: Target environment (optional, default: `production`)
- **dryRun**: Dry run mode (optional, default: `true`)

## Process
1. **Cloud Provider Detection**: Identify target cloud providers and environments
2. **Input Validation**: Comprehensive parameter validation and security checks
3. **Multi-Cloud Context Analysis**: Analyze current state across all providers
4. **Operation Planning**: Generate optimized execution plan
5. **Safety Assessment**: Risk analysis and impact evaluation across providers
6. **Execution**: Perform operations with monitoring and validation
7. **Results Analysis**: Process results and generate reports

## Outputs
- **Operation Results**: Detailed execution results and status per provider
- **Compliance Reports**: Validation and compliance status across environments
- **Performance Metrics**: Operation performance and efficiency metrics by provider
- **Recommendations**: Optimization suggestions and next steps
- **Audit Trail**: Complete operation history for compliance across all providers

## Environment
- **AWS**: EKS, EC2, Lambda, CloudWatch, IAM, S3
- **Azure**: AKS, VMs, Functions, Monitor, Azure AD
- **GCP**: GKE, Compute Engine, Cloud Functions, Cloud Monitoring
- **On-Premise**: Kubernetes clusters, VMware, OpenStack, Prometheus
- **Multi-Cloud Tools**: Terraform, Ansible, Crossplane, Cluster API

## Dependencies
- **Python 3.8+**: Core execution environment
- **Cloud SDKs**: boto3, azure-sdk, google-cloud
- **Kubernetes**: kubernetes client for cluster operations
- **Multi-Cloud Libraries**: terraform-python, ansible-python

## Scripts
- `core/scripts/automation/k8s-troubleshoot.py`: Main automation implementation
- `core/scripts/automation/k8s-troubleshoot_handler.py`: Cloud-specific operations
- `core/scripts/automation/multi_cloud_orchestrator.py`: Cross-provider coordination

## Trigger Keywords
k8s, troubleshoot, automation, enterprise, multi-cloud, aws, azure, gcp, onprem

## Human Gate Requirements
- **Production changes**: Production environment operations require approval
- **High-impact operations**: Critical operations require review
- **Security changes**: Security modifications need validation

## Enterprise Features
- **Multi-tenant Support**: Isolated operations per tenant
- **Role-based Access Control**: Enterprise IAM integration
- **Audit Logging**: Complete audit trail for compliance
- **Performance Monitoring**: SLA tracking and metrics
- **Security Hardening**: Encryption and compliance standards
- **Dynamic Code Generation**: Agents can modify logic dynamically
- **Cross-Cloud Orchestration**: Coordinated operations across providers

## References

Load these files when needed:
- `scripts/k8s-diagnostic.py` - Core Kubernetes diagnostic and analysis tools
- `scripts/resource-analyzer.py` - Resource usage and constraint analysis
- `references/k8s-troubleshooting-guide.md` - Step-by-step troubleshooting procedures
- `assets/diagnostic-templates.yaml` - Standard diagnostic templates for common issues
- `examples/troubleshooting-scenarios/` - Real-world troubleshooting examples and solutions