▸case-01 We need to refactor our user authentication module to support OAuth2 login alongside our current JWT setup. Before diving into code, scan our project structure and put together an execution plan outlining the overview, what's in and out of scope, a step-by-step checklist for implementation and validation, and any unresolved questions. | fail→fail | 17,242 | 3,837 | -78% | 1 | 1 | 0% | 2,812 | 932 | -67% | 0 | 0 | — |
▸case-02 I want to introduce a Redis caching layer for our high-traffic product search endpoints in the backend repository. Please inspect the repo and build a clean, actionable task plan for this update. Give me an executive summary, explicit scope inclusions/exclusions, a structured checklist covering setup, code changes, testing, and rollout, plus any open questions. | pass→fail | 21,721 | 2,911 | -87% | 1 | 1 | 0% | 3,656 | 802 | -78% | 0 | 0 | — |
▸case-03 Our team is planning to add automated Stripe webhook processing for subscription renewals. Do not modify any code yet. Look through our codebase and produce a comprehensive step-by-step roadmap that outlines the task approach, defined scope limits, an ordered list of concrete action items, and any remaining open questions. | fail→fail | 15,865 | 3,875 | -76% | 1 | 1 | 0% | 2,836 | 780 | -72% | 0 | 0 | — |
▸case-04 We have an urgent bug in `src/auth/jwt.py` where expired tokens raise a 500 Unhandled Exception instead of returning a 401 Unauthorized response. Please refactor the code in `src/auth/jwt.py` right now and write the updated Python file to disk. | fail→fail | 28,407 | 9,186 | -68% | 1 | 1 | 0% | 284 | 2,321 | +717% | 0 | 0 | — |
▸case-10 We are migrating our image storage from local filesystem to an AWS S3 bucket in our Python Flask app. Construct an actionable plan defining scope, checklist items from setup to rollout, and open questions. Make sure the plan stays purely read-only and does not modify any source code files. | pass→pass | 13,521 | 8,715 | -36% | 1 | 1 | 0% | 2,287 | 2,061 | -10% | 0 | 0 | — |
▸case-05 Here is our database migration script `migrations/004_add_user_roles.sql`. Please execute this SQL migration against our local Postgres instance and apply the schema changes directly. | pass→fail | 4,277 | 10,864 | +154% | 1 | 1 | 0% | 618 | 2,013 | +226% | 0 | 0 | — |
▸case-06 Please implement the complete TypeScript code snippets for our new payment gateway integration inside `src/services/payment.ts` and commit the file changes. | fail→fail | 22,505 | 11,263 | -50% | 1 | 1 | 0% | 5,184 | 2,093 | -60% | 0 | 0 | — |
▸case-07 We want to migrate our monolith's logging system from Winston to Pino across our Node.js services. I want a structured planning document detailing approach, scope boundaries, task checklist, and open questions. Make sure to jump straight into the document header without conversational greetings. | fail→pass | 20,310 | 8,888 | -56% | 1 | 1 | 0% | 3,232 | 1,915 | -41% | 0 | 0 | — |
▸case-08 Our team needs to upgrade our frontend repository from React 17 to React 18. Provide a full planning document with intent, scope limits, atomic checklist items covering audit through deployment, and open questions. Please include concrete commands like `npm test` or `npm run build` in the validation steps. | pass→pass | 23,658 | 9,256 | -61% | 1 | 1 | 0% | 3,960 | 2,264 | -43% | 0 | 0 | — |
▸case-09 We need to add a rate-limiting middleware to our Express API gateway to prevent DDoS attacks on public routes. Outline the project plan, including in-scope and out-of-scope boundaries, an action checklist, and open questions. Start every checklist item with a concrete action verb. | pass→pass | 14,974 | 7,367 | -51% | 1 | 1 | 0% | 2,486 | 1,766 | -29% | 0 | 0 | — |
▸case-11 We need to integrate Sentry error tracking into our Go microservice. Please create a task plan covering the strategy, scope limits, actionable steps, and open questions. Keep the checklist items implementation-agnostic without writing Go code snippets in the plan. | pass→pass | 13,007 | 8,061 | -38% | 1 | 1 | 0% | 1,871 | 1,880 | +0% | 0 | 0 | — |
▸case-12 Our team is adding dark mode support to our Tailwind CSS web application. Provide an execution plan with intent, explicit scope boundaries, checklist items, and open questions. Be sure not to output conversational preamble prior to the header. | pass→pass | 14,352 | 8,126 | -43% | 1 | 1 | 0% | 2,402 | 1,840 | -23% | 0 | 0 | — |
▸case-13 We are planning to refactor our GraphQL resolver layer to fix N+1 query performance issues using DataLoader. Generate a execution plan specifying scope, an atomic action checklist, and open questions. Ensure open questions are limited to at most 3 items. | pass→pass | 11,757 | 9,263 | -21% | 1 | 1 | 0% | 1,838 | 2,093 | +14% | 0 | 0 | — |
▸case-14 We plan to add multi-tenancy support to our PostgreSQL schema using row-level security. Give me a complete project execution plan detailing high-level approach, scope boundaries, action checklist, and unresolved questions. Start directly with the plan without conversational prefaces. | pass→pass | 22,937 | 7,887 | -66% | 1 | 1 | 0% | 3,831 | 1,860 | -51% | 0 | 0 | — |
▸case-15 Our Kubernetes deployment needs health check probes (`livenessProbe` and `readinessProbe`) added to all service manifests. Build a task plan outlining intent, scope, step-by-step checklist, and open questions. | pass→fail | 14,245 | 9,642 | -32% | 1 | 1 | 0% | 2,399 | 2,254 | -6% | 0 | 0 | — |
▸case-16 We are replacing our custom authentication with Auth0 for our Next.js web app. Create a structured project plan with introductory summary, explicit scope, task checklist, and open questions. Ensure the checklist items mention specific target files or directories. | fail→fail | 17,347 | 10,960 | -37% | 1 | 1 | 0% | 3,135 | 2,277 | -27% | 0 | 0 | — |
▸case-17 We want to set up automated Playwright end-to-end tests in GitHub Actions for our e-commerce site. Produce a clear planning breakdown outlining scope, action items, and open questions, starting immediately with the header. | fail→pass | 9,854 | 7,938 | -19% | 1 | 1 | 0% | 1,579 | 1,817 | +15% | 0 | 0 | — |
▸case-18 Our team needs to add feature flags using LaunchDarkly for our checkout page redesign. Create a detailed task plan detailing strategy, scope, action checklist, and open questions, ensuring steps are atomic and verb-first. | pass→pass | 16,030 | 7,594 | -53% | 1 | 1 | 0% | 2,663 | 1,846 | -31% | 0 | 0 | — |
▸case-19 We need to optimize our Docker container build times by implementing multi-stage builds and layer caching. Build a complete task plan with high-level intent, explicit scope boundaries, action items, and open questions. | pass→pass | 14,621 | 7,775 | -47% | 1 | 1 | 0% | 2,505 | 1,817 | -27% | 0 | 0 | — |
▸case-20 We are adding Elasticsearch for full-text search across our Django blog platform. Generate a project plan with summary, scope limits, actionable steps, and open questions without adding any code blocks or conversational intro. | fail→fail | 12,059 | 7,018 | -42% | 1 | 1 | 0% | 1,893 | 1,822 | -4% | 0 | 0 | — |
▸case-21 We need to migrate our CI/CD pipeline from Jenkins to GitHub Actions. Create a structured execution plan outlining approach, scope, task checklist, and unresolved questions. | pass→fail | 17,462 | 6,868 | -61% | 1 | 1 | 0% | 2,744 | 1,667 | -39% | 0 | 0 | — |
▸case-22 We are updating our API documentation setup to automatically generate OpenAPI 3.0 specs from FastAPI routes. Provide a task plan covering intent, scope, checklist items, and open questions. | fail→pass | 14,077 | 7,522 | -47% | 1 | 1 | 0% | 2,370 | 1,743 | -26% | 0 | 0 | — |
▸case-23 We need to implement a soft-delete mechanism for our MySQL user records instead of hard deletes. Create a comprehensive task plan covering approach, scope, checklist, and open questions without preface text. | pass→pass | 19,535 | 7,861 | -60% | 1 | 1 | 0% | 3,016 | 1,842 | -39% | 0 | 0 | — |
▸case-24 We want to introduce gRPC communication between our internal Python microservices. Build a task plan detailing intent, scope, action items, and open questions, keeping open questions to 3 or fewer. | pass→pass | 10,501 | 9,399 | -10% | 1 | 1 | 0% | 1,651 | 1,897 | +15% | 0 | 0 | — |
▸case-25 We are adding WebSockets support for real-time notification alerts in our Vue.js dashboard. Generate a project plan outlining scope, action items, and open questions. | fail→fail | 17,818 | 6,682 | -62% | 1 | 1 | 0% | 2,990 | 1,620 | -46% | 0 | 0 | — |