▸case-01 I need to migrate our web application's legacy SQL session store to a dedicated Redis cluster across our production environment without dropping active user logins. Can you write a detailed implementation plan for this? Include a clear definition of what constitutes completion, an ordered list of task steps with specific verification methods for each, a table of potential risks alongside early warning indicators, and explicit conditions that would require pausing or aborting the rollout. | fail→pass | 36,271 | 33,702 | -7% | 1 | 1 | 0% | 5,634 | 5,432 | -4% | 0 | 0 | — |
▸case-02 We are planning a zero-downtime migration of our core multi-tenant database from AWS Aurora PostgreSQL to a self-hosted PostgreSQL cluster in GCP. Please create a structured execution roadmap for my engineering team. I want the plan to specify exact criteria for when the migration is finished, step-by-step procedures where every step has a way to verify success before moving forward, a section outlining risk signals and planned responses, and hard boundaries on what must not be attempted during this pass. | fail→fail | 54,707 | 32,932 | -40% | 1 | 1 | 0% | 7,763 | 4,803 | -38% | 0 | 0 | — |
▸case-03 Our team is replacing our legacy monolith payment processing logic with a modern async Stripe integration, and we need a detailed deployment plan before starting work. Please build a plan document that defines the measurable endpoint test for success, lists all actionable steps along with their check procedures and outputs, captures risks and tripwires, and explicitly lists stop conditions so an automated subagent or teammate can run it autonomously. | pass→pass | 41,204 | 29,480 | -28% | 1 | 1 | 0% | 6,571 | 4,774 | -27% | 0 | 0 | — |
▸case-04 We are currently executing step 4 of our database migration plan, but running the index creation script resulted in a lock timeout on the active tenant table. Here is our original plan document and error log. Should we improvise a workaround on the live server, or how should we handle this execution blocker? | pass→fail | 18,081 | 16,692 | -8% | 1 | 1 | 0% | 2,140 | 2,710 | +27% | 0 | 0 | — |
▸case-05 Our payment service migration finished yesterday. We experienced a 12-minute outage during cutover due to an unexpected connection pool exhaustion. Can you help us write a post-incident retrospective report detailing root cause, timeline, and lessons learned for our post-mortem review? | pass→pass | 28,254 | 25,253 | -11% | 1 | 1 | 0% | 3,006 | 4,024 | +34% | 0 | 0 | — |
▸case-06 Our executive team wants to build an AI-powered customer support chatbot, but the business goals, user personas, and target response metrics are completely undefined. Can you interview me or generate a set of clarifying questions to draft a Product Requirements Document (PRD) and brief first? | pass→pass | 17,332 | 22,333 | +29% | 1 | 1 | 0% | 1,833 | 2,682 | +46% | 0 | 0 | — |
▸case-07 We need to integrate a third-party KYC vendor API into our user registration pipeline. The vendor documentation is unclear on whether their webhook payload supports our custom tenant header, which is critical for our architecture. The rest of the task involves updating UI components, database models, and internal event emitters. Please write an execution plan for this integration. Note that we are tempted to build the DB models and UI screens first since they are straightforward. | pass→pass | 22,791 | 19,730 | -13% | 1 | 1 | 0% | 2,882 | 2,770 | -4% | 0 | 0 | — |
▸case-08 I need to update a single environment variable MAX_WORKERS from 4 to 8 in our staging Kubernetes deployment, restart the pod, and verify CPU usage stays below 50% for 10 minutes. I was planning to write a multi-page project plan document with risk matrix tables and multi-phase checkpoints. Please provide the appropriate plan for this 15-minute routine task. | fail→fail | 16,678 | 13,031 | -22% | 1 | 1 | 0% | 1,627 | 2,095 | +29% | 0 | 0 | — |
▸case-09 We are refactoring our billing service to support multi-currency invoices. The team wants a plan that ensures no schema migrations are executed during this initial code refactor pass, as database changes are restricted until next quarter. Construct a plan document where this boundary is strictly enforced. | fail→pass | 29,636 | 18,559 | -37% | 1 | 1 | 0% | 3,887 | 3,702 | -5% | 0 | 0 | — |
▸case-10 Our team wants to clean up our monolithic notification service code base. Engineers suggested defining the goal as 'refactor the notification service code and clean up helper functions'. Write a plan document for this effort, ensuring the goal is expressed strictly as a testable outcome rather than an activity. | fail→pass | 24,355 | 22,925 | -6% | 1 | 1 | 0% | 2,921 | 3,755 | +29% | 0 | 0 | — |
▸case-11 We are adding OAuth2 login support to our API gateway. A team member proposed a step: 'Then integrate and test the OAuth library with all downstream microservices.' Please write a proper execution plan where this vague step is decomposed into independently verifiable steps with concrete check methods. | pass→pass | 20,799 | 16,228 | -22% | 1 | 1 | 0% | 2,504 | 3,767 | +50% | 0 | 0 | — |
▸case-12 We are switching our search engine cluster from Elasticsearch to Typesense under high traffic. We want to identify potential failure points early before total traffic cutover. Create a plan document that pre-names risks with concrete early warning signals and planned reactions. | pass→pass | 33,093 | 24,062 | -27% | 1 | 1 | 0% | 3,682 | 3,833 | +4% | 0 | 0 | — |
▸case-13 We are migrating our background worker queue from Celery to SQS. If SQS FIFO throughput limitations exceed our peak rate of 3,000 messages/sec during benchmarking, the entire SQS architecture choice becomes invalid and we must stop. Generate an execution plan that captures this stop condition. | pass→pass | 29,794 | 20,047 | -33% | 1 | 1 | 0% | 3,299 | 3,271 | -1% | 0 | 0 | — |
▸case-14 We want to update our frontend build toolchain from Webpack 4 to Vite 5. We have 50 internal npm packages, 3 of which use legacy CommonJS syntax that might break under Vite's ESM bundler. Tempting option: upgrade Webpack configs, convert all React components, and then check package compatibility at the end. Write an execution plan that avoids this happy-path-first trap. | pass→pass | 24,658 | 19,306 | -22% | 1 | 1 | 0% | 3,229 | 3,946 | +22% | 0 | 0 | — |
▸case-15 We are upgrading our ETL pipeline from Python 3.8 to Python 3.12. Write a plan document where the goal is framed as an outcome test rather than 'upgrade Python packages'. | fail→fail | 20,028 | 28,610 | +43% | 1 | 1 | 0% | 3,154 | 4,631 | +47% | 0 | 0 | — |
▸case-16 We are planning a multi-week refactor of our monolith's domain model across 4 separate engineering sprints. We need to know when to pause and reassess progress between sessions. Construct an execution plan for this multi-session project. | fail→pass | 26,873 | 21,941 | -18% | 1 | 1 | 0% | 3,321 | 4,090 | +23% | 0 | 0 | — |
▸case-17 We are updating the TLS certificates across 50 internal microservice proxies. A bad certificate deployment could break internal RPC calls across all services. Write a plan document that defines tripwires and reactions for proxy connection errors during rolling updates. | pass→pass | 27,031 | 25,403 | -6% | 1 | 1 | 0% | 3,575 | 4,046 | +13% | 0 | 0 | — |
▸case-18 We have an intermittent latency spike in our WebSocket gateway occurring every few hours, but we don't know the root cause or how to reproduce it deterministically yet. Write a plan document for this situation, focusing on investigating and establishing observability rather than jumping into refactoring. | pass→pass | 28,006 | 22,575 | -19% | 1 | 1 | 0% | 3,693 | 3,621 | -2% | 0 | 0 | — |
▸case-19 We are splitting the user notification module out of our core monolith into a standalone gRPC service. Create a plan document that follows strict plan document formatting with goal as an outcome test, step verification, risk tripwires, and stop conditions. | fail→pass | 35,528 | 23,823 | -33% | 1 | 1 | 0% | 5,048 | 3,911 | -23% | 0 | 0 | — |
▸case-20 We are switching our public DNS provider from Route53 to Cloudflare with zero edge downtime. Engineers want to update all NS records at once without checking TTL propagation or dark ping testing. Create a plan document that orders steps by information value to catch delegation failures early. | fail→fail | 30,578 | 21,074 | -31% | 1 | 1 | 0% | 3,991 | 4,119 | +3% | 0 | 0 | — |
▸case-21 We are adding a non-null column tenant_id to a 500-million-row audit_logs table in PostgreSQL. We must ensure table locks do not exceed 500ms during schema application. Draft an execution plan for this database migration. | fail→fail | 18,141 | 39,220 | +116% | 1 | 1 | 0% | 3,372 | 4,506 | +34% | 0 | 0 | — |
▸case-22 We are rolling out a new recommendation algorithm via feature flags to 100% of users over 3 days. Write an execution plan detailing canary percentage increments, metric tripwires, and stop conditions. | fail→fail | 25,567 | 27,426 | +7% | 1 | 1 | 0% | 3,199 | 3,883 | +21% | 0 | 0 | — |