Ideogram Incident Response

Ideogram Incident Response

Overview

Stabilize an Ideogram integration before chasing image quality or replaying paid requests. Preserve evidence, reduce blast radius, reconcile accepted work, protect sensitive media, and choose recovery actions by incident class.

Prerequisites

• Incident commander, environment, start time, impact statement, and communication channel.
• Read access to content-free application, queue, vendor-status, webhook, polling, storage, and billing evidence.
• Known controls for admission, concurrency, key revocation, traffic rollback, queue drain, and publication stop.

Current Contract

Incidents commonly cross separate boundaries: server-side Api-Key, prepaid shared team credit, default in-flight capacity, async generation_id, signed but non-guaranteed webhook delivery, item safety, expiring URLs, and application storage. Recovery must not collapse these into a generic retry.

Authentication

Never paste the API key into incident chat or commands. Verify secret provenance through metadata; if exposure is credible, stop new traffic, revoke under owner approval, rotate consumers, and audit historical leakage.

Instructions

1. Declare severity, impact, affected environment and tenants, known time window, and incident commander.
2. Freeze risky deployments and broad retries; preserve sanitized statuses, identifiers, queue state, and storage receipts.
3. Classify credential, depleted credit, validation, 429 pressure, vendor capacity, unsafe publication, webhook loss, expired URL, or storage failure.
4. Contain with the smallest control: close admission, reduce concurrency, stop publishing, switch to polling, isolate storage, or roll back traffic.
5. Reconcile every accepted async identifier before resubmission and prevent duplicate asset publication.
6. Recover with synthetic canaries, then restore bounded traffic while monitoring safety, durable completion, errors, latency, and spend.
7. Record timeline, decisions, evidence, customer impact, cleanup, follow-ups, and final state.

Tool Discipline

Use Read, Glob, and Grep for evidence and known runbooks. Use Write and Edit for approved incident records or fixes. Do not revoke keys, add credit, delete assets, replay jobs, or deploy without incident authority.

Approval Boundaries

The incident commander approves containment and restoration; security owns credential response, billing owns credit, moderation owns unsafe publication, and data owners approve asset access or deletion. Separate reversible mitigation from destructive action.

Error Handling