Prometheus Configuration Specialist

Purpose & When-To-Use

Trigger conditions:

• You need to configure Prometheus for metrics collection from Kubernetes, Consul, EC2, or static targets
• You need to create alerting rules with burn rate calculations or multi-window patterns
• You need to optimize PromQL queries or reduce cardinality for high-volume metrics
• You need to set up federation for multi-datacenter or cross-service monitoring
• You need to configure Alertmanager routing with grouping, inhibition, or multiple receivers
• You need to pre-compute expensive queries using recording rules

Complements:

• observability-stack-configurator: For overall observability stack design
• observability-unified-dashboard: For Grafana dashboard design with Prometheus datasources
• observability-slo-calculator: For SLO/error budget definitions that drive alerting rules

Out of scope:

• Application instrumentation (use language-specific Prometheus client libraries)
• Long-term metrics storage (use Thanos, Cortex, or Mimir)
• Log aggregation (use Loki or ELK)
• Distributed tracing (use Tempo, Jaeger, or Zipkin)

Pre-Checks

Time normalization:

• Compute NOW_ET using NIST/time.gov semantics (America/New_York, ISO-8601)
• Use NOW_ET for all access dates in citations

Verify inputs:

• ✅ Required: At least one scrape target specification (service discovery config or static targets)
• ✅ Required: Prometheus version specified (recommend 3.0+ for UTF-8, OTLP, Remote Write 2.0)
• ⚠️ Optional: Alert definitions (if alerting is needed)
• ⚠️ Optional: Recording rule definitions (if query optimization is needed)
• ⚠️ Optional: Alertmanager receivers (PagerDuty, Slack, email, webhook)
• ⚠️ Optional: Federation topology (if multi-DC or cross-service monitoring is required)

Validate service discovery:

• If kubernetes_sd_config: Verify Kubernetes API access and RBAC permissions
• If consul_sd_config: Verify Consul agent accessibility and service catalog
• If ec2_sd_config: Verify AWS credentials and EC2 instance tags
• If file_sd_config: Verify JSON/YAML file path and refresh interval

Check cardinality constraints:

• Every unique combination of label key-value pairs creates a new time series
• High-cardinality labels (user IDs, request IDs, timestamps) cause memory/storage issues
• Recommended: <10M active time series per Prometheus instance
• Use metric_relabel_configs to drop high-cardinality labels

Source freshness:

• Prometheus 3.0 released November 14, 2024 (accessed NOW_ET)
• Prometheus 3.5 (upcoming LTS release, 2025)
• Alerting best practices and recording rule conventions stable across versions

Abort if:

• No scrape targets specified → EMIT TODO: "Specify at least one scrape target (kubernetes_sd, consul_sd, ec2_sd, static_configs)"
• Service discovery config incomplete → EMIT TODO: "Provide complete service discovery configuration (API endpoints, credentials, filters)"
• Alert definitions lack severity or description → EMIT TODO: "Add severity label and description annotation to all alerts"

Procedure

T1: Basic Prometheus Setup (≤2k tokens, 80% use case)

Scenario: Single service with static targets or file-based service discovery, basic alerting, no recording rules.

Steps:

1. Global Configuration:

• Set scrape_interval: 15s (balance between data freshness and storage)
• Set evaluation_interval: 15s (how often to evaluate alerting/recording rules)
• Set external_labels for federation or remote write (e.g., datacenter: us-east-1)

1. Scrape Configuration:

• Define job_name (logical grouping, e.g., api-service, postgres-exporter)
• Choose service discovery:
• Static: static_configs with targets: ['localhost:9090']
• File-based: file_sd_configs with files: ['/etc/prometheus/targets/*.json']
• Set scrape_interval override if different from global

1. Basic Alerting Rules:

• Create alerts.yml with groups
• Alert on symptoms (high latency, error rate) not causes (CPU, disk)
• Include severity label (severity: critical|warning|info)
• Add description and summary annotations

1. Alertmanager Integration:

• Configure alertmanager_config with static_configs pointing to Alertmanager instance
• Set send_resolved: true to notify when alert resolves

Output:

• prometheus.yml with global config, single scrape job, alerting rules file reference
• alerts.yml with 2-5 basic alerts
• No recording rules (not needed for T1 simplicity)

Token budget: ≤2000 tokens

T2: Multi-Service Discovery + Recording Rules (≤6k tokens)

Scenario: Multiple services with Kubernetes/Consul/EC2 service discovery, recording rules for expensive queries, Alertmanager routing with grouping.

Steps:

1. Service Discovery Configuration:

Kubernetes Service Discovery:

• Use kubernetes_sd_configs with role: pod (discover all pods with prometheus.io/scrape: "true" annotation)
• Relabeling pattern:

yaml relabel_configs:

• source_labels: __meta_kubernetes_pod_annotation_prometheus_io_scrape]

action: keep regex: true

• source_labels: __meta_kubernetes_pod_annotation_prometheus_io_path]

action: replace target_label: __metrics_path__ regex: (.+)

• source_labels: __address__, __meta_kubernetes_pod_annotation_prometheus_io_port]

action: replace target_label: __address__ regex: (^:]+)(?::\d+)?;(\d+) replacement: $1:$2

• source_labels: __meta_kubernetes_namespace]

action: replace target_label: kubernetes_namespace

• source_labels: __meta_kubernetes_pod_name]

action: replace target_label: kubernetes_pod_name

• Supported roles: node, pod, service, endpoints, ingress (accessed NOW_ET: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config)

Consul Service Discovery:

• Use consul_sd_configs with server: 'consul.service.consul:8500'
• Filter by service tags: tags: ['production', 'monitoring-enabled']

EC2 Service Discovery:

• Use ec2_sd_configs with AWS region and filters
• Relabel based on EC2 tags: __meta_ec2_tag_<tagkey>

1. Recording Rules:

• Naming convention: level:metric:operations (accessed NOW_ET: https://prometheus.io/docs/practices/naming/)
• Level: Aggregation level (job, instance, cluster)
• Metric: Base metric name
• Operations: Aggregation operations (sum, avg, rate)
• Example:

yaml groups:

• name: api_recording_rules

interval: 30s rules:

• record: job:http_requests_total:rate5m

expr: sum(rate(http_requests_total5m])) by (job)

• record: job:http_request_duration_seconds:p95

expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket5m])) by (job, le))

• Use cases: Pre-compute dashboard queries, optimize slow PromQL queries, aggregate high-cardinality metrics

1. Relabeling Strategies:

Metric Relabeling (metric_relabel_configs):

• Drop high-cardinality labels:

yaml metric_relabel_configs:

• source_labels: user_id]

action: labeldrop regex: .

• source_labels: __name__]

action: drop regex: 'expensive_metric_.'

Target Relabeling (relabel_configs):

• Modify labels before scraping (transform service discovery metadata)

1. Alerting Rules (Advanced):

Multi-Window Burn Rate Alerts:

• Detect fast SLO burn (error budget exhausted in days instead of weeks)
• Example: 14.4× burn rate (exhaust 30-day budget in 2 days) for critical, 6× for warning
• Pattern:

yaml groups:

• name: slo_alerts

rules:

• alert: ErrorBudgetBurn_Critical

expr: | ( sum(rate(http_requests_total{status=~"5.."}1h])) / sum(rate(http_requests_total1h])) ) > (14.4 0.001) for: 2m labels: severity: critical annotations: summary: "Error budget burning 14.4× faster than allowed" description: "{{ $labels.job }} has {{ $value | humanizePercentage }} error rate (SLO: 99.9%, budget exhausted in 2 days)"

Symptom-Based Alerts:

• Alert on latency, error rate, saturation (not CPU/memory directly)
• Golden Signals: Latency, Traffic, Errors, Saturation
• Example:

yaml

• alert: HighLatency

expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket5m])) by (job, le)) > 0.5 for: 5m labels: severity: warning annotations: summary: "High latency on {{ $labels.job }}" description: "p95 latency is {{ $value }}s (threshold: 0.5s)"

1. Alertmanager Routing:

• Routing tree: Group alerts by cluster + alertname, wait 30s for batch
• Receivers: PagerDuty (critical), Slack (warning/info), email (all)
• Inhibition: Suppress lower-severity alerts when higher-severity alerts are firing
• Example:

yaml route: receiver: 'default-email' group_by: 'cluster', 'alertname'] group_wait: 30s group_interval: 5m repeat_interval: 4h routes:

• match:

severity: critical receiver: 'pagerduty'

• match:

severity: warning receiver: 'slack'

receivers:

• name: 'pagerduty'

pagerduty_configs:

• service_key: '<PD_SERVICE_KEY>'
• name: 'slack'

slack_configs:

• api_url: '<SLACK_WEBHOOK_URL>'

channel: '#alerts' title: '{{ .GroupLabels.alertname }}' text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

• name: 'default-email'

email_configs:

• to: 'ops@example.com'

inhibit_rules:

• source_match:

severity: critical target_match: severity: warning equal: 'cluster', 'alertname']

Output:

• prometheus.yml with Kubernetes/Consul/EC2 service discovery, relabeling configs
• recording_rules.yml with 5-10 recording rules (level:metric:operations naming)
• alerts.yml with multi-window burn rate alerts and symptom-based alerts
• alertmanager.yml with routing tree, receivers, inhibition rules

Token budget: ≤6000 tokens

T3: Enterprise Federation + PromQL Optimization (≤12k tokens)

Scenario: Multi-datacenter federation, cardinality management, PromQL query optimization, Prometheus 3.0+ features (UTF-8, OTLP, Remote Write 2.0).

Steps:

1. Federation Configuration:

Hierarchical Federation (Multi-DC):

• Pattern: Per-datacenter Prometheus servers scrape local services, global Prometheus server federates aggregated metrics
• Benefits: Scales to tens of datacenters and millions of nodes (accessed NOW_ET: https://prometheus.io/docs/prometheus/latest/federation/)
• Global server config:

yaml scrape_configs:

• job_name: 'federate-us-east-1'

scrape_interval: 30s honor_labels: true metrics_path: '/federate' params: 'match]':

• '{job="prometheus"}'
• '{__name__=~"job:."}' # Only federate aggregated recording rules

static_configs:

• targets:
• 'prometheus-us-east-1:9090'
• 'prometheus-us-west-2:9090'

Cross-Service Federation:

• Pattern: Service A Prometheus federates metrics from Service B Prometheus to correlate cross-service metrics
• Use case: Cluster scheduler federating resource usage from multiple service Prometheus servers

1. PromQL Optimization:

Query Performance Best Practices:

• Filter early: Use label matchers to narrow time series before aggregation
• ❌ Slow: sum(http_requests_total) (aggregates 10k+ time series)
• ✅ Fast: sum(http_requests_total{job="api-service", status=~"5.."}) (aggregates 10-50 time series)
• Avoid broad selectors: Never use bare metric names (api_http_requests_total) without labels
• Use recording rules: Pre-compute expensive queries (accessed NOW_ET: https://prometheus.io/docs/prometheus/latest/querying/basics/)
• Limit time ranges: Avoid queries over >24h without recording rules
• Example optimized query:

promql # Compute error rate using pre-recorded job-level metrics (fast) job:http_requests_total:rate5m{job="api-service", status=~"5.."} / job:http_requests_total:rate5m{job="api-service"}

Cardinality Management:

• Problem: High-cardinality labels (user IDs, request IDs) create millions of time series → memory/disk explosion
• Detection: Query topk(10, count by (__name__)({__name__=~".+"})) to find high-cardinality metrics
• Solutions:

1. Drop labels: Use metric_relabel_configs to remove high-cardinality labels
2. Aggregate: Use recording rules to pre-aggregate high-cardinality metrics
3. Sample: Use metric_relabel_configs with action: drop to sample metrics

• Example cardinality reduction:

yaml metric_relabel_configs: # Drop user_id label (high cardinality)

• source_labels: user_id]

action: labeldrop regex: . # Keep only 5xx errors (reduce cardinality of status label)

• source_labels: status]

action: keep regex: '5..'

1. Prometheus 3.0+ Features:

UTF-8 Support (Prometheus 3.0+):

• Feature: Allows all valid UTF-8 characters in metric and label names (accessed NOW_ET: https://prometheus.io/blog/2024/11/14/prometheus-3-0/)
• Example: http_requests_total{endpoint="用户登录"} (Chinese characters now valid)
• Migration: UTF-8 mode enabled by default in Prometheus 3.0

OpenTelemetry OTLP Receiver (Prometheus 3.0+):

• Feature: Prometheus can receive OTLP metrics natively
• Endpoint: /api/v1/otlp/v1/metrics
• Configuration:

yaml otlp: protocols: http: endpoint: 0.0.0.0:9090

• Use case: Consolidate Prometheus and OpenTelemetry pipelines

Remote Write 2.0 (Prometheus 3.0+):

• Feature: Native support for metadata, exemplars, created timestamps, native histograms
• Benefits: Better interoperability with long-term storage (Thanos, Cortex, Mimir)

1. Advanced Relabeling Patterns:

Extract Kubernetes Annotations into Labels: yaml relabel_configs:

• source_labels: __meta_kubernetes_pod_annotation_app_version]

action: replace target_label: version

• source_labels: __meta_kubernetes_pod_annotation_team]

action: replace target_label: team

Drop Expensive Metrics Based on Name Pattern: yaml metric_relabel_configs:

• source_labels: __name__]

action: drop regex: 'go_.|process_.' # Drop Go runtime metrics to save storage

1. Recording Rules for Aggregation:

Multi-Level Aggregation: yaml groups:

• name: instance_aggregation

interval: 30s rules: # Level 1: Instance-level

• record: instance:http_requests_total:rate5m

expr: sum(rate(http_requests_total5m])) by (instance, job, status)

# Level 2: Job-level (aggregates Level 1)

• record: job:http_requests_total:rate5m

expr: sum(instance:http_requests_total:rate5m) by (job, status)

# Level 3: Cluster-level (aggregates Level 2)

• record: cluster:http_requests_total:rate5m

expr: sum(job:http_requests_total:rate5m) by (status)

1. Alertmanager Advanced Features:

Time-Based Routing (Mute Alerts During Maintenance): yaml route: routes:

• match:

severity: warning mute_time_intervals:

• weekends
• maintenance_window

mute_time_intervals:

• name: weekends

time_intervals:

• weekdays: 'saturday', 'sunday']
• name: maintenance_window

time_intervals:

• times:
• start_time: '23:00'

end_time: '01:00'

Grouping by Multiple Labels: yaml route: group_by: ['cluster', 'namespace', 'alertname'] group_wait: 30s group_interval: 5m repeat_interval: 12h

Output:

• prometheus.yml with federation endpoints, OTLP receiver, Remote Write 2.0
• Multi-level recording rules (instance → job → cluster aggregation)
• Cardinality management relabeling configs
• PromQL optimization recommendations with query examples
• Alertmanager advanced routing (time-based muting, multi-label grouping)

Token budget: ≤12000 tokens

Decision Rules

When to use federation vs remote write:

• Federation: Multi-DC with global aggregation, <10 Prometheus servers
• Remote Write: Long-term storage, >10 Prometheus servers, different retention policies

When to create recording rules:

• Query execution time >5s on Grafana dashboard
• Query used in multiple dashboards or alerts
• High-cardinality metric needs pre-aggregation (e.g., >100k time series)

Alert severity assignment:

• Critical: User-impacting outage, page on-call engineer immediately (e.g., API error rate >5%)
• Warning: Potential issue, notify Slack, no page (e.g., API latency p95 >500ms)
• Info: FYI notification, email only (e.g., deployment completed)

Service discovery selection:

• Kubernetes: Use kubernetes_sd_configs with role: pod for dynamic pod discovery
• Consul: Use consul_sd_configs for VM-based infrastructure with Consul service catalog
• EC2: Use ec2_sd_configs for AWS instances with consistent tagging
• File-based: Use file_sd_configs for static infrastructure or external service discovery

Cardinality limits:

• Target: <10M active time series per Prometheus instance
• Alert: If prometheus_tsdb_symbol_table_size_bytes >1GB or prometheus_tsdb_head_series >10M
• Action: Drop high-cardinality labels or aggregate with recording rules

Abort conditions:

• Prometheus memory usage >80% of available → reduce cardinality or add recording rules
• Scrape duration >scrape interval → increase interval or optimize exporters
• Alert fatigue (>50 alerts firing) → review alert thresholds and use inhibition rules

Output Contract

prometheus.yml schema:

yaml
global:
  scrape_interval: <duration>
  evaluation_interval: <duration>
  external_labels:
    <label_name>: <label_value>

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['<alertmanager_host>:<port>']

rule_files:
  - 'alerts.yml'
  - 'recording_rules.yml'

scrape_configs:
  - job_name: '<job_name>'
    kubernetes_sd_configs: [...]  # OR consul_sd_configs, ec2_sd_configs, static_configs
    relabel_configs: [...]
    metric_relabel_configs: [...]

alerts.yml schema:

yaml
groups:
  - name: <group_name>
    rules:
      - alert: <alert_name>
        expr: <promql_expression>
        for: <duration>
        labels:
          severity: critical|warning|info
        annotations:
          summary: <short_description>
          description: <detailed_description_with_templating>

recording_rules.yml schema:

yaml
groups:
  - name: <group_name>
    interval: <duration>
    rules:
      - record: <level>:<metric>:<operations>
        expr: <promql_expression>
        labels:
          <label_name>: <label_value>

alertmanager.yml schema:

yaml
route:
  receiver: <default_receiver>
  group_by: [<label_name>, ...]
  group_wait: <duration>
  group_interval: <duration>
  repeat_interval: <duration>
  routes:
    - match:
        <label_name>: <label_value>
      receiver: <receiver_name>

receivers:
  - name: <receiver_name>
    pagerduty_configs: [...]
    slack_configs: [...]
    email_configs: [...]

inhibit_rules:
  - source_match:
      <label_name>: <label_value>
    target_match:
      <label_name>: <label_value>
    equal: [<label_name>, ...]

Required fields:

• prometheus.yml: global.scrape_interval, scrape_configs[].job_name
• alerts.yml: alert, expr, labels.severity, annotations.summary
• recording_rules.yml: record, expr
• alertmanager.yml: route.receiver, receivers[].name

Validation:

• All PromQL expressions syntactically valid: promtool check rules <file.yml>
• Prometheus config valid: promtool check config prometheus.yml
• Alertmanager config valid: amtool check-config alertmanager.yml

Examples

Example 1: Kubernetes Service Discovery with Recording Rules

Scenario: Scrape all pods with prometheus.io/scrape: "true" annotation, create recording rules for API latency.

prometheus.yml:

yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

recording_rules.yml:

yaml
groups:
  - name: api_latency
    interval: 30s
    rules:
      - record: job:http_request_duration_seconds:p95
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))
      - record: job:http_request_duration_seconds:p99
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))

Quality Gates

Token budgets:

• T1: ≤2000 tokens (basic scrape + alerting)
• T2: ≤6000 tokens (service discovery + recording rules + Alertmanager routing)
• T3: ≤12000 tokens (federation + PromQL optimization + cardinality management)

Safety:

• ❌ Never: Include secrets in metric labels (passwords, API keys, tokens)
• ❌ Never: Use high-cardinality labels (user IDs, request IDs, UUIDs) without aggregation
• ✅ Always: Validate PromQL expressions with promtool check rules
• ✅ Always: Use metric_relabel_configs to drop secrets if accidentally exposed

Auditability:

• All Prometheus configs in version control (Git)
• Recording rule naming follows level:metric:operations convention
• Alert annotations include summary and description with templating
• Alertmanager routing documented with receiver purposes

Determinism:

• Same scrape targets + same relabeling = same time series
• Recording rules evaluated at fixed intervals (deterministic)
• Alert grouping by cluster + alertname produces predictable batches

Performance:

• Scrape duration <80% of scrape interval (avoid missed scrapes)
• PromQL query execution time <5s (use recording rules if slower)
• Cardinality <10M active time series per Prometheus instance
• Alert evaluation time <1s (use recording rules to pre-aggregate)

Resources

Official Documentation:

• Prometheus 3.0 announcement (UTF-8, OTLP, Remote Write 2.0): https://prometheus.io/blog/2024/11/14/prometheus-3-0/ (accessed NOW_ET)
• Configuration reference: https://prometheus.io/docs/prometheus/latest/configuration/configuration/ (accessed NOW_ET)
• Alerting best practices: https://prometheus.io/docs/practices/alerting/ (accessed NOW_ET)
• Recording rules: https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/ (accessed NOW_ET)
• Naming conventions: https://prometheus.io/docs/practices/naming/ (accessed NOW_ET)
• Federation: https://prometheus.io/docs/prometheus/latest/federation/ (accessed NOW_ET)

Tooling:

• promtool: Validate Prometheus configs and PromQL queries
• amtool: Validate Alertmanager configs and manage silences
• Prometheus exporters: Node Exporter, Blackbox Exporter, PostgreSQL Exporter, etc.

Related Skills:

• observability-stack-configurator: Overall observability stack design
• observability-unified-dashboard: Grafana dashboard design with Prometheus datasources
• observability-slo-calculator: SLO/error budget definitions for alerting rules
• kubernetes-manifest-generator: Kubernetes deployment manifests for Prometheus + Alertmanager