▸case-17 Our enterprise backend needs to migrate a 500GB MySQL database to PostgreSQL with zero downtime. Create a step-by-step implementation plan detailing technical dependencies, cutover milestones, and rollback risks. | pass→pass | 41,975 | 16,203 | -61% | 1 | 1 | 0% | 6,792 | 2,856 | -58% | 0 | 0 | — |
▸case-06 We need to validate user email addresses and password lengths in our Node.js microservice. Should we pull in an external npm package like validator or write standard utility functions? Evaluate the security, health, and dependency risk considerations. | fail→fail | 18,562 | 9,486 | -49% | 1 | 1 | 0% | 2,898 | 1,860 | -36% | 0 | 0 | — |
▸case-15 We need an AGENTS.md file for our repository to define system instructions, build steps, and commit conventions for automated AI coders working in our repo. Produce the complete markdown document. | fail→pass | 15,306 | 10,457 | -32% | 1 | 1 | 0% | 2,482 | 2,053 | -17% | 0 | 0 | — |
▸case-18 We are evaluating Go logging libraries for a microservice high-throughput environment: zap vs logrus. Evaluate both packages regarding memory efficiency, active maintenance status, and lock contention risk. | pass→pass | 14,711 | 10,118 | -31% | 1 | 1 | 0% | 2,377 | 1,923 | -19% | 0 | 0 | — |
▸case-01 We just received a client specification for rebuilding our customer management portal using React and Node.js. Can you analyze these requirements and outline the project scope, estimate the implementation effort for each module, and highlight any technical risks we should factor into our client proposal? | fail→pass | 8,783 | 17,200 | +96% | 1 | 1 | 0% | 1,345 | 2,032 | +51% | 0 | 0 | — |
▸case-02 Our background data processor is running extremely slowly and bottlenecking memory during peak hours. Here is the implementation: [code snippet]. Can you analyze where the resource bottleneck is occurring and provide a clean, high-performance refactoring along with low-level explanations for why the changes improve efficiency? | fail→fail | 14,950 | 4,912 | -67% | 1 | 1 | 0% | 2,280 | 1,066 | -53% | 0 | 0 | — |
▸case-03 Our payment processing module currently lacks automated testing. Here is the module source code: [code snippet]. Could you set up a comprehensive test suite using PyTest that covers the primary payment paths and key failure scenarios so we can safely deploy? | fail→pass | 13,806 | 16,253 | +18% | 1 | 1 | 0% | 2,784 | 3,810 | +37% | 0 | 0 | — |
▸case-04 We are planning to migrate our legacy Python Flask e-commerce monolith to a microservices architecture on AWS EKS. Provide a structured execution roadmap outlining task dependencies, key milestones, and major technical risks. Do not ask me clarifying questions, just produce the plan. | pass→pass | 35,200 | 13,440 | -62% | 1 | 1 | 0% | 5,782 | 2,506 | -57% | 0 | 0 | — |
▸case-05 Our Python asyncio web scraper crashes with an OutOfMemory error after running for 2 hours. The code appends unawaited tasks to a global set: `pending_tasks.add(asyncio.create_task(fetch(url)))` without removing completed tasks. Isolate the exact root cause and provide the minimal fix. | pass→pass | 7,169 | 5,213 | -27% | 1 | 1 | 0% | 1,177 | 1,285 | +9% | 0 | 0 | — |
▸case-07 Simplify this JavaScript validation function without changing its behavior: `function check(u) { if (u !== null) { if (u !== undefined) { if (u.age >= 18) { if (u.isActive === true) { return true; } else { return false; } } else { return false; } } else { return false; } } else { return false; } }`. | fail→fail | 16,807 | 6,238 | -63% | 1 | 1 | 0% | 3,491 | 1,678 | -52% | 0 | 0 | — |
▸case-08 Our repository `acme/auth-service` has zero documentation. Create a minimal documentation structure including README.md, AGENTS.md, and outline files for the `/docs/` directory. | pass→pass | 15,418 | 15,127 | -2% | 1 | 1 | 0% | 2,628 | 3,089 | +18% | 0 | 0 | — |
▸case-09 We need a helper function in Python to turn user titles into URL-safe slugs (e.g. 'Hello World!' to 'hello-world'). Should we add python-slugify as a project dependency or use standard library modules like re and unicodedata? | fail→pass | 13,624 | 7,804 | -43% | 1 | 1 | 0% | 2,267 | 1,752 | -23% | 0 | 0 | — |
▸case-10 Refactor this JavaScript array mapping code to use modern ES6 syntax: `var titles = []; for (var i = 0; i < books.length; i++) { if (books[i].published) { titles.push(books[i].title); } }`. | fail→pass | 5,009 | 2,881 | -42% | 1 | 1 | 0% | 901 | 904 | +0% | 0 | 0 | — |
▸case-16 Simplify this Node.js async function: `async function fetchData(url) { return new Promise((resolve, reject) => { fetch(url).then(res => res.json()).then(data => resolve(data)).catch(err => reject(err)); }); }`. | fail→pass | 5,733 | 4,448 | -22% | 1 | 1 | 0% | 1,062 | 1,161 | +9% | 0 | 0 | — |
▸case-11 Write a Python function to compute the Nth Fibonacci number using dynamic programming. Add fun emojis and friendly encouraging words to the response! | fail→pass | 8,301 | 2,921 | -65% | 1 | 1 | 0% | 1,458 | 817 | -44% | 0 | 0 | — |
▸case-12 Our PostgreSQL database reports `ERROR: deadlock detected` when two concurrent transactions update rows in `accounts` table in reverse order: Tx1 updates ID 1 then ID 2; Tx2 updates ID 2 then ID 1. Isolate the cause and provide the minimal SQL fix. | pass→pass | 9,434 | 6,528 | -31% | 1 | 1 | 0% | 1,634 | 1,515 | -7% | 0 | 0 | — |
▸case-13 A client wants a custom Stripe integration and catalog system built on Next.js with 5 core modules: User Auth, Product Catalog, Shopping Cart, Stripe Checkout, Order History. Analyze this proposal request to produce a scope breakdown, effort estimate in developer days, and technical risk assessment. | pass→pass | 21,617 | 10,514 | -51% | 1 | 1 | 0% | 3,505 | 2,101 | -40% | 0 | 0 | — |
▸case-14 In Go, our log collector appends 100,000 strings to a slice in a tight loop: `var logs []string; for _, msg := range raw { logs = append(logs, msg) }`. It causes high GC pressure and memory allocations. Provide optimized code with system-level explanation. | pass→pass | 18,656 | 13,365 | -28% | 1 | 1 | 0% | 3,108 | 2,618 | -16% | 0 | 0 | — |
▸case-19 Our enterprise sales team is closing a contract for our SaaS software platform. Can you write standard legal language for an intellectual property indemnification clause regarding third-party patent litigation? | pass→fail | 13,134 | 7,127 | -46% | 1 | 1 | 0% | 2,137 | 1,594 | -25% | 0 | 0 | — |
▸case-20 We are racking new top-of-rack switches and 2U enterprise servers in our physical server room. What cabling layout and patch panel standards should our IT technician follow for 10GbE network racks? | pass→pass | 19,208 | 12,663 | -34% | 1 | 1 | 0% | 3,103 | 2,475 | -20% | 0 | 0 | — |
▸case-21 Our engineering manager is onboarding a newly hired software developer. What standard administrative steps should HR complete for tax withholding forms and direct deposit payroll setup? | pass→pass | 14,466 | 8,640 | -40% | 1 | 1 | 0% | 2,222 | 1,658 | -25% | 0 | 0 | — |
▸case-22 Our Rust string parsing crate needs a test module. Here is the parser function signature: `pub fn parse_config(raw: &str) -> Result<Config, ParseError>`. Write unit test cases using Rust's `#[cfg(test)]` module testing valid configs, empty input, and malformed syntax. | fail→pass | 12,166 | 7,004 | -42% | 1 | 1 | 0% | 2,484 | 1,712 | -31% | 0 | 0 | — |