▸case-01 We are migrating our backend service from Express to Fastify across 20+ modules. I want to leverage AI agents for implementation while keeping risk low. Could you decompose this feature into agent-sized work items and provide a structured breakdown listing each task unit, its verification steps, primary risk focus, and recommended model tier? | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-03 We need to update our database migration scripts to add a new column across 15 tables in a Postgres database. A junior engineer suggested creating one prompt for the LLM to process all 15 tables at once to save overhead. How should this workload be structured into tasks for AI agents? | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-12 We have a task to change the default timeout constant from 30s to 45s across 10 configuration files, and another task to design the zero-downtime deployment strategy across microservices. Which model tiers should handle these? | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-05 An agent is mid-way through fixing a subtle memory leak in a C++ service, and it has failed 3 trial attempts while outputting large log snippets. Should we trigger a context compaction right now to free up context tokens before the next retry? | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-11 We have a single task prompt that combines changing the database schema, updating the public API route, and updating the frontend React components. How should this prompt be refactored for agent safety? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-15 Before letting an AI agent attempt to fix a flaky test in our CI system, what artifact or data must be recorded first? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-14 When auditing an AI-generated payment reconciliation script, what specific software aspects should human reviewers inspect rather than relying on standard unit test passes? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-19 We are building a basic single-call LLM endpoint in Python to extract structured user contact info (name, email, phone) from inbound support tickets. We are not using an agentic framework or multi-step execution. What prompt engineering technique ensures the LLM outputs strictly valid JSON matching specified keys? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-07 During code review of an AI-generated PR, two reviewers are spending significant time debating variable naming conventions, formatting tabs vs spaces, and comment styles. Is this an effective review focus for AI-generated code? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-16 A developer wants to kick off an autonomous implementation agent with a prompt: 'Improve the performance of our API.' What step must be taken prior to executing the agent? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-02 I am building an automated dev pipeline where LLMs resolve bug tickets independently. Can you outline an operational execution process for this setup, explaining how to sequence evaluation loops, handle context windows, and manage model escalation when lower-tier models hit execution limits? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-10 When initiating an AI agent task to refactor a critical payment gateway service, what is the exact sequence of steps for evaluation relative to code execution? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-22 When writing task specifications for AI agents working on microservice updates, what specific element must be included in each task unit definition regarding task completion? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-13 An agent finished writing a helper utility function in TypeScript and now needs to write the immediate unit tests designed specifically for that helper. Should we open a fresh session or continue the existing session? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-20 We are setting up a GitHub Actions CI workflow file to run unit tests on every commit pushed to our repository. What key under the `on:` trigger directive in the workflow YAML file specifies running on code pushes? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-17 A developer wants to set Opus as the default model tier for all engineering tasks (including generating boilerplate DTOs) because it has the highest benchmark scores. What is the recommended model tier routing practice? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-04 Our engineering team has access to Claude Haiku, Sonnet, and Opus. We need to: 1) Classify incoming error log types, 2) Rewrite a multi-file state management subsystem, and 3) Investigate a complex race condition involving distributed transactions. Which model should be assigned to which task to minimize cost without risking quality? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-06 An AI agent just completed drafting the entire architecture specification and data schema for a new billing microservice. We are ready to transition into writing the implementation code. Should we continue in this current session to preserve context or reset? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-08 An automated agent using Haiku hit a retry threshold of 2 attempts due to an unexpected API timeout from an external service failure. Should the execution framework automatically escalate the task to Opus? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-09 We want to set up telemetry for our AI agent engineering pipeline in Prometheus. List the critical operational metrics that must be tracked per task to maintain cost discipline. | pass→pass | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-18 An AI agent generated a new REST API endpoint for downloading user invoice PDFs. What should human reviewers check beyond whether the endpoint returns HTTP 200? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-21 A developer is writing a Jest unit test for a standalone pure function `calculateDiscount(price, percent)`. What Jest assertion method should be used to verify that `calculateDiscount(100, 20)` returns `80`? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |