Langfuse Infra Scaling

Langfuse Infra Scaling

Use this skill for Langfuse web-ingestion, web, and web-iso autoscaling work in langfuse/infrastructure.

Goal

Optimize the cost/performance tradeoff: run the smallest reliable container footprint that preserves acceptable latency, error rate, CPU/memory headroom, event-loop health, and burst capacity. Treat both idle overprovisioning and insufficient headroom as problems. Recommendations should say which side of the tradeoff they optimize and why.

Core Rules

• Read the current branch and files first. Do not rely on remembered values.
• Do not optimize only for cost or only for latency. Prefer changes that reduce waste when health is green, and preserve or add headroom when snappy routes, errors, CPU/memory, event-loop delay, or autoscaler load show risk.
• Keep web_ingestion_cpu = 2048 unless the user explicitly overrides it. These Node.js containers reserve one CPU for the event loop and one for background work.
• Treat web_ingestion_instances as the autoscaler minimum. In terraform/modules/ecs_service/resources.tf, the autoscaler target uses min_capacity = var.instances.
• For prod web-ingestion, keep the minimum container count at 3.
• Preserve burst capacity when lowering the minimum: max_capacity = web_ingestion_instances * web_ingestion_autoscaler_max_capacity_multiplier.
• Distinguish cost levers: if a service usually sits at its minimum task count, raising RPM targets will not save money; lower the minimum after validating HA/compliance floors, and raise the max-capacity multiplier if needed to preserve burst ceiling.
• Update dashboard markers whenever RPM target or boundary changes.
• Use exact OpenTofu math for markers. Do not mental-math ceil(target * boundary) because Terraform float behavior can produce off-by-one results.
• For web and web-iso, do not reduce RPM per container solely because service latency is high. First inspect full APM traces, including ClickHouse spans, to distinguish container saturation from database-bound requests.
• If slow web-iso traces are dominated by ClickHouse child spans, do not lower RPM targets as the primary fix. Lowering RPM changes web-container scaling and request distribution, but it does not reduce ClickHouse query cost, total query workload, or an individual long-running query duration.
• If scaling should be adjusted, create or update a GitHub PR. For the Linear follow-up, comment the evidence onto a ticket that already exists and label it; a new ticket has no parent, so show it and file it once that is approved, and never assign it. Step 8 has the detail.

Files

Service scaling values live in:

terraform/environments/services/env-prod-us.tfvars
terraform/environments/services/env-prod-eu.tfvars
terraform/environments/services/env-prod-hipaa.tfvars
terraform/environments/services/env-prod-jp.tfvars

Dashboard marker values live in:

terraform/environments/monitoring/env-prod-us.tfvars
terraform/environments/monitoring/env-prod-eu.tfvars
terraform/environments/monitoring/env-prod-hipaa.tfvars