▸case-02 We are planning to upgrade our payment gateway integration. Please analyze how the current checkout subsystem processes orders from end to end. I want a full codebase analysis covering entry points into the stack, the call sequence, structural insights, a table of key files, internal and external dependencies, and guidelines on what to reuse or avoid. | fail→pass | 19,370 | 14,945 | -23% | 1 | 1 | 0% | 4,167 | 3,256 | -22% | 0 | 0 | — |
▸case-13 We need to optimize our GraphQL query resolvers for nested user queries. Analyze how query execution flows from GraphQL HTTP endpoint down to ORM queries. I would like the analysis headings to be numbered 1 through 6 at H1 level. | fail→fail | 16,980 | 14,101 | -17% | 1 | 1 | 0% | 3,449 | 3,433 | -0% | 0 | 0 | — |
▸case-14 Analyze our WebSocket connection gateway before we implement horizontal scaling. Trace connection lifecycle from handshake to message broadcasting. Provide key files, dependencies, and recommendations. | fail→fail | 19,174 | 12,531 | -35% | 1 | 1 | 0% | 3,721 | 2,875 | -23% | 0 | 0 | — |
▸case-01 I need to refactor our authentication module before starting single sign-on support. Can you explore the existing auth codebase and provide a structured breakdown detailing the main entry triggers, step-by-step execution path, architectural patterns, key source files, dependencies, and practical recommendations for our upcoming dev work? | fail→pass | 17,347 | 10,585 | -39% | 1 | 1 | 0% | 3,745 | 2,778 | -26% | 0 | 0 | — |
▸case-03 Before we build the new real-time notifications service, can you do a deep dive into the existing messaging backend? Please document where user actions enter the system, trace the execution flow, explain current architectural abstractions, summarize core files, list dependencies, and provide recommendations for new development. | fail→pass | 22,973 | 10,966 | -52% | 1 | 1 | 0% | 4,675 | 2,617 | -44% | 0 | 0 | — |
▸case-04 We need to update our web analytics telemetry ingestion pipeline. Analyze the existing telemetry codebase before we write new code. Give me entry triggers, step-by-step execution sequence, architecture patterns, a list of main files formatted as a plain bulleted text list, dependencies, and dev recommendations. | fail→fail | 16,176 | 10,551 | -35% | 1 | 1 | 0% | 3,111 | 2,730 | -12% | 0 | 0 | — |
▸case-05 We are refactoring our PostgreSQL schema migration runner. Analyze how migrations are currently loaded, executed, and tracked. Present your analysis, but please use an ASCII directory tree for key files and write recommendations as a single summary paragraph. | fail→fail | 8,198 | 8,401 | +2% | 1 | 1 | 0% | 1,853 | 2,336 | +26% | 0 | 0 | — |
▸case-06 Our document generation module needs an overhaul. Before writing any code, analyze the PDF rendering pipeline. Provide entry points, execution path, architecture, key files table, and ensure external and internal dependencies are combined into a single flat bullet list. | pass→fail | 14,857 | 8,443 | -43% | 1 | 1 | 0% | 2,971 | 2,068 | -30% | 0 | 0 | — |
▸case-07 We are planning to modify our role-based access control (RBAC) middleware. Analyze how permissions are evaluated from HTTP requests down to database checks. Output your breakdown as a JSON object with keys like 'entryPoints' and 'keyFiles' so our parser can read it. | fail→fail | 9,576 | 10,843 | +13% | 1 | 1 | 0% | 2,104 | 2,729 | +30% | 0 | 0 | — |
▸case-08 Our API gateway rate limiting mechanism needs analysis. Trace how incoming requests hit the token bucket check and Redis storage. Summarize the codebase structure, but instead of step-by-step numbers, provide a general narrative paragraph for the execution path. | fail→fail | 7,521 | 8,619 | +15% | 1 | 1 | 0% | 1,411 | 2,310 | +64% | 0 | 0 | — |
▸case-09 Analyze our Elasticsearch search indexing worker before we add vector search capabilities. Trace how document changes trigger background re-indexing. Provide entry triggers, step sequence, patterns, key files table, dependencies, and recommendations. | pass→pass | 21,711 | 10,952 | -50% | 1 | 1 | 0% | 4,186 | 2,676 | -36% | 0 | 0 | — |
▸case-10 We are adding chunked uploads to our file storage service. Analyze the current file upload pipeline from multipart request handling through cloud storage persistence. Present key files as a bulleted key-value list with file path and description. | fail→fail | 9,495 | 11,330 | +19% | 1 | 1 | 0% | 1,855 | 2,654 | +43% | 0 | 0 | — |
▸case-11 Analyze our distributed cache invalidation mechanism in the user profile service. Show how cache miss triggers affect database reads and invalidation events. Format your findings cleanly covering entry points, call path, architecture, files, dependencies, and tips. | fail→pass | 16,220 | 9,635 | -41% | 1 | 1 | 0% | 3,185 | 2,529 | -21% | 0 | 0 | — |
▸case-12 Before adding retry backoff logic to our outbound webhook dispatcher, explore how webhooks are queued and sent. Detail the call path and structure, but render key files using HTML table tags (<table>) for custom styling. | fail→fail | 19,358 | 7,954 | -59% | 1 | 1 | 0% | 5,348 | 2,040 | -62% | 0 | 0 | — |
▸case-15 Analyze the background email notification queue processing code. Show how events trigger email templating and SMTP dispatch. Please present your recommendations as a single bulleted list of tips without specific sub-categories. | fail→fail | 11,049 | 9,444 | -15% | 1 | 1 | 0% | 2,057 | 2,447 | +19% | 0 | 0 | — |
▸case-16 Analyze the OAuth2 token refresh and validation mechanism in our auth service. Trace execution from endpoint hit to JWT verification. Detail entry points, call path, architecture patterns, key files table, dependencies, and dev recommendations. | fail→pass | 25,184 | 10,686 | -58% | 1 | 1 | 0% | 5,163 | 2,631 | -49% | 0 | 0 | — |
▸case-17 Explore our compliance audit logging module before we extend event tracking to financial transactions. Document how application events are intercepted, formatted, and persisted to audit storage. | fail→pass | 14,421 | 10,671 | -26% | 1 | 1 | 0% | 2,775 | 2,465 | -11% | 0 | 0 | — |
▸case-18 Analyze our background job scheduling system. Detail how scheduled triggers spawn worker threads and handle failures. Output a complete codebase analysis covering entry points, call path, architecture, files table, dependencies, and recommendations. | fail→pass | 27,062 | 13,330 | -51% | 1 | 1 | 0% | 5,716 | 3,153 | -45% | 0 | 0 | — |
▸case-19 Explore our feature flag evaluation system before we integrate remote config updates. Show how flag evaluations resolve targeting rules and fallback defaults. | fail→pass | 16,052 | 13,544 | -16% | 1 | 1 | 0% | 3,724 | 3,136 | -16% | 0 | 0 | — |
▸case-20 We found a null pointer bug in auth_service.py where user.profile is accessed when user is anonymous. Please write a Python code patch to fix this bug with appropriate null checks and unit test updates. | pass→fail | 11,135 | 6,576 | -41% | 1 | 1 | 0% | 2,384 | 1,996 | -16% | 0 | 0 | — |
▸case-21 Please write a suite of Pytest unit tests for our payment processing module in checkout/payment.py to cover success, card declined, and timeout scenarios. | pass→pass | 11,384 | 7,281 | -36% | 1 | 1 | 0% | 2,964 | 2,180 | -26% | 0 | 0 | — |
▸case-22 I just finished implementing the new OAuth token refresh endpoint across 5 files. Please draft a Pull Request description for GitHub explaining the changes, testing performed, and deployment notes for reviewers. | pass→pass | 9,610 | 6,023 | -37% | 1 | 1 | 0% | 1,940 | 1,570 | -19% | 0 | 0 | — |