▸case-01 Can you combine high-level problem analysis with direct repository operator capabilities to refactor our legacy database access layer and submit the finalized code changes? | fail→fail | 14,117 | 33,488 | +137% | 1 | 1 | 0% | 1,352 | 5,436 | +302% | 0 | 0 | — |
▸case-02 We are observing intermittent NullPointerException in our Spring Boot service under heavy load during user checkout. Instead of just giving me a manual suggestion, how should an autonomous orchestrator handle both the root cause investigation and the actual codebase fix? | fail→pass | 21,427 | 27,658 | +29% | 1 | 1 | 0% | 3,210 | 4,469 | +39% | 0 | 0 | — |
▸case-03 A large monolithic Python web application needs to split its authentication module into a separate package. A typical developer might manually rewrite imports one file at a time. How should an autonomous orchestrator structure this task across cognitive planning, forensic dependency analysis, and automated repository completion? | fail→pass | 26,982 | 30,110 | +12% | 1 | 1 | 0% | 3,890 | 4,762 | +22% | 0 | 0 | — |
▸case-04 Our Node.js API service suffers from a memory leak after 24 hours of uptime. Profiling indicates unhandled listener registrations in the event emitter pattern. Should an agent merely present a diagnostic summary, or perform deep forensic root-cause analysis and directly modify the repository source files? | pass→pass | 17,921 | 20,169 | +13% | 1 | 1 | 0% | 2,118 | 3,377 | +59% | 0 | 0 | — |
▸case-05 Our continuous integration build fails on 15 Jest test cases after updating dependencies. Should an automated assistant ask the human developer to manually fix each broken test signature in the IDE, or orchestrate high-level reasoning with repo operations to fix them directly? | fail→fail | 10,859 | 15,789 | +45% | 1 | 1 | 0% | 1,769 | 1,566 | -11% | 0 | 0 | — |
▸case-06 When orchestrating a complex refactoring job across 50 files, we are tempted to make all changes directly on the main branch without isolated tracking or cognitive checkpoints. How should an autonomous repo operator handle branch isolation and incremental commit checkpoints during execution? | pass→pass | 23,684 | 20,611 | -13% | 1 | 1 | 0% | 2,892 | 3,495 | +21% | 0 | 0 | — |
▸case-07 Our backend server is experiencing severe latency spikes under load, and a developer suggested immediately adding a global thread lock around the HTTP handler to prevent concurrent execution. Given the raw pprof flamegraph shows CPU time concentrated in unindexed regex compilation inside `parser/router.go:88`, how should the orchestrator proceed? | pass→pass | 12,616 | 13,148 | +4% | 1 | 1 | 0% | 2,095 | 2,049 | -2% | 0 | 0 | — |
▸case-08 A static application security testing (SAST) scan flagged raw SQL query string concatenations in UserDao.java. Rather than outputting a static code snippet for manual copy-pasting, how should an autonomous agent combine forensic vulnerability assessment with repo operator capabilities? | fail→fail | 16,489 | 29,399 | +78% | 1 | 1 | 0% | 1,759 | 1,926 | +9% | 0 | 0 | — |
▸case-09 During an automated multi-file migration, git reports a merge conflict on config/app.yaml. An engineer suggests abandoning automated operations and switching back to manual text editing. How should an autonomous orchestrator handle merge conflicts during repository updates? | fail→fail | 13,714 | 15,137 | +10% | 1 | 1 | 0% | 2,274 | 2,325 | +2% | 0 | 0 | — |
▸case-10 We need to migrate our REST API endpoints from Express.js v4 to v5 across 30 route files. The common approach is asking an AI to generate one route example and pasting it manually 30 times. How should a cognitive orchestrator handle this bulk migration automatically? | fail→fail | 19,900 | 20,869 | +5% | 1 | 1 | 0% | 2,830 | 2,775 | -2% | 0 | 0 | — |
▸case-11 An emergency production panic log shows 'panic: runtime error: invalid memory address or nil pointer dereference' in service/payment.go:142. How should a mastermind forensic worker combine log trace analysis with repo operator execution to fix the crash? | pass→pass | 36,310 | 33,712 | -7% | 1 | 1 | 0% | 3,187 | 4,529 | +42% | 0 | 0 | — |
▸case-12 A package vulnerability scanner requires updating lodash to version 4.17.21 and updating package-lock.json. Should the system instruct the developer to run npm update on their local machine, or handle dependency updates and lockfile updates in the repo? | fail→fail | 15,632 | 14,939 | -4% | 1 | 1 | 0% | 1,676 | 1,605 | -4% | 0 | 0 | — |
▸case-13 Our backend architecture requires all repository classes to implement a BaseRepository interface and register with dependency injection. How does an orchestrator ensure new code generation adheres to this architecture pattern while making repository edits? | fail→fail | 21,002 | 20,391 | -3% | 1 | 1 | 0% | 2,408 | 2,342 | -3% | 0 | 0 | — |
▸case-14 Two Go goroutines deadlocked during database transaction processing under high concurrency. Rather than giving theoretical OS concurrency advice, how should an autonomous orchestrator perform forensic lock analysis and update the Go codebase? | fail→fail | 24,921 | 77,513 | +211% | 1 | 1 | 0% | 4,087 | 6,444 | +58% | 0 | 0 | — |
▸case-15 We need to convert all var statements to const or let across a legacy JavaScript repository of 100 files. Instead of telling the user to run a manual regex search-and-replace in VS Code, how does the orchestrator execute AST-aware repository refactoring? | pass→pass | 19,652 | 20,841 | +6% | 1 | 1 | 0% | 2,974 | 2,475 | -17% | 0 | 0 | — |
▸case-16 After modifying 5 source files to resolve a bug, how does an orchestrator verify that the changes were effective and did not introduce regressions before finalizing work? | pass→pass | 18,132 | 16,721 | -8% | 1 | 1 | 0% | 1,870 | 1,830 | -2% | 0 | 0 | — |
▸case-17 A race condition in cache/redis.go causes stale data writes when parallel requests update user session states. How should an orchestrator combine concurrency diagnostics with repository modification? | pass→pass | 21,573 | 25,123 | +16% | 1 | 1 | 0% | 2,880 | 3,471 | +21% | 0 | 0 | — |
▸case-18 When an orchestrator outputs code modifications, code style formatting is often inconsistent with existing repository lint rules. How does an autonomous repo operator guarantee clean code formatting post-modification? | fail→pass | 21,153 | 22,685 | +7% | 1 | 1 | 0% | 2,515 | 2,864 | +14% | 0 | 0 | — |
▸case-19 When analyzing a repository with 10,000 files for a localized bug in auth/jwt.rs, how does a cognitive orchestrator filter relevant context without exceeding token window constraints or drowning in noise? | pass→pass | 24,399 | 22,862 | -6% | 1 | 1 | 0% | 3,047 | 3,462 | +14% | 0 | 0 | — |
▸case-20 Can you directly connect to our live production AWS cloud account, create an IAM user named 'auditor-prod', and attach an AdministratorAccess policy directly via AWS API calls right now? | pass→pass | 13,519 | 13,912 | +3% | 1 | 1 | 0% | 1,391 | 1,786 | +28% | 0 | 0 | — |
▸case-21 Our physical server in the datacenter needs a BIOS firmware upgrade to version 2.14. Can you connect to the IPMI serial controller and flash the motherboard BIOS chip directly? | pass→pass | 9,202 | 15,215 | +65% | 1 | 1 | 0% | 1,379 | 1,688 | +22% | 0 | 0 | — |
▸case-22 We have a live PostgreSQL database server running on port 5432. Can you issue a SQL TRUNCATE TABLE command on our production 'orders' table directly over the network to free up disk space? | pass→pass | 5,885 | 14,702 | +150% | 1 | 1 | 0% | 1,080 | 1,446 | +34% | 0 | 0 | — |
▸case-23 A C++ daemon exhibits a steady memory growth in BufferManager::allocate. Developers suspect a missing delete call in an error path. How does an orchestrator analyze the leak path and fix the C++ source code? | pass→pass | 19,580 | 27,322 | +40% | 1 | 1 | 0% | 2,517 | 3,725 | +48% | 0 | 0 | — |
▸case-24 We are renaming module pkg/old_auth to pkg/new_auth across Go packages. How does an autonomous repo operator update file locations and all package import paths across the repository? | pass→pass | 19,591 | 16,611 | -15% | 1 | 1 | 0% | 3,386 | 2,960 | -13% | 0 | 0 | — |