▸case-04 Explain what the regex pattern ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ matches and point out any edge cases. | pass→pass | 17,324 | 20,825 | +20% | 1 | 1 | 0% | 2,169 | 3,245 | +50% | 0 | 0 | — |
▸case-03 I just modified main.py in my project repository. Please provide a concise, standard 1-line git commit message following Conventional Commits format for a bug fix in user authentication. | pass→pass | 8,463 | 2,432 | -71% | 1 | 1 | 0% | 448 | 454 | +1% | 0 | 0 | — |
▸case-01 I need to draft a comprehensive three-part technical architecture document in my workspace directory at /workspace/docs. Please write each section sequentially, starting with the system overview, followed by the database schema details, and ending with the API endpoints specification. Output a status summary of the created files when complete. | pass→fail | 43,402 | 57,776 | +33% | 1 | 1 | 0% | 8,249 | 7,512 | -9% | 0 | 0 | — |
▸case-02 I want to write an end-to-end user tutorial across four separate markdown files in /workspace/guides. Draft Chapter 1 through Chapter 4 in order, ensuring each subsequent chapter references the setup from the previous one, and provide a final index listing all generated guide files. | fail→fail | 38,366 | 12,461 | -68% | 1 | 1 | 0% | 7,332 | 504 | -93% | 0 | 0 | — |
▸case-05 Review this 5-line Python function snippet for potential division-by-zero errors: def avg(lst): return sum(lst) / len(lst). | pass→pass | 6,623 | 16,783 | +153% | 1 | 1 | 0% | 1,169 | 1,306 | +12% | 0 | 0 | — |
▸case-06 Draft microservice technical specification markdown files in /workspace/services for auth-service, payment-service, and notification-service. You might be inclined to write payment specs assuming sync REST calls to auth, but payment needs asynchronous event publishing. Write each service specification sequentially into its separate file. | fail→fail | 10,983 | 14,166 | +29% | 1 | 1 | 0% | 284 | 539 | +90% | 0 | 0 | — |
▸case-07 Draft OpenAPI 3.0 YAML specifications in /workspace/api_specs split across auth.yaml, products.yaml, and orders.yaml. A common mistake is using Swagger 2.0 syntax like `swagger: '2.0'`. Ensure all specs strictly use OpenAPI 3.0 schema syntax and write each file to disk sequentially. | fail→fail | 33,879 | 39,292 | +16% | 1 | 1 | 0% | 7,650 | 8,385 | +10% | 0 | 0 | — |
▸case-08 Create a sequence of PostgreSQL migration SQL files in /workspace/migrations: 001_create_users.sql, 002_create_posts.sql, and 003_create_comments.sql. Beginners often put foreign key constraints in 001_create_users.sql that reference posts, causing execution failure. Ensure foreign keys strictly reference existing tables created in earlier migrations. | fail→pass | 6,045 | 22,194 | +267% | 1 | 1 | 0% | 295 | 3,994 | +1254% | 0 | 0 | — |
▸case-09 Draft a Rust library crate structure in /workspace/crate with src/lib.rs, src/parser.rs, and src/evaluator.rs. Developers often forget to declare submodules in lib.rs, making them inaccessible. Generate each source file sequentially, ensuring proper module declarations. | fail→fail | 4,938 | 19,013 | +285% | 1 | 1 | 0% | 719 | 1,761 | +145% | 0 | 0 | — |
▸case-10 Produce release notes for versions v1.0.0, v1.1.0, and v2.0.0 in /workspace/releases/ saved as v1.0.0.md, v1.1.0.md, and v2.0.0.md. Do not omit semantic versioning convention categories. Write each version file sequentially to the workspace. | fail→fail | 10,621 | 19,645 | +85% | 1 | 1 | 0% | 327 | 1,952 | +497% | 0 | 0 | — |
▸case-11 Write Terraform configuration files in /workspace/infra: main.tf, variables.tf, and outputs.tf. A common anti-pattern is hardcoding variable values directly in main.tf instead of referencing declared input variables. Write each configuration file sequentially into /workspace/infra using input variable interpolation. | fail→fail | 22,027 | 10,834 | -51% | 1 | 1 | 0% | 297 | 397 | +34% | 0 | 0 | — |
▸case-12 Build a Vue 3 frontend component set in /workspace/frontend/src/components with Header.vue, Sidebar.vue, and Footer.vue. Do not use legacy Vue 2 Options API `export default { data() {} }`. Write each component file sequentially using Vue 3 `<script setup>` syntax. | fail→fail | 10,761 | 12,603 | +17% | 1 | 1 | 0% | 313 | 532 | +70% | 0 | 0 | — |
▸case-13 Set up a Python package in /workspace/mypackage/ containing __init__.py, core.py, and utils.py. Avoid circular imports by making sure core.py imports utility functions from utils.py using explicit relative imports, and expose package symbols in __init__.py. | pass→fail | 9,628 | 12,091 | +26% | 1 | 1 | 0% | 1,860 | 511 | -73% | 0 | 0 | — |
▸case-14 Prepare container deployment files in /workspace/deploy: Dockerfile.web, Dockerfile.api, and docker-compose.yml. Avoid using deprecated Compose file syntax like `version: '2'`. Generate each file sequentially, pointing docker-compose.yml to the respective Dockerfiles. | pass→fail | 11,709 | 11,908 | +2% | 1 | 1 | 0% | 1,539 | 551 | -64% | 0 | 0 | — |
▸case-15 Draft a 3-part textbook on machine learning in /workspace/book with section1_intro.md, section2_models.md, and section3_ethics.md. Ensure mathematical equations in section2_models.md are formatted in valid LaTeX block syntax ($$...$$) and write each section file sequentially to the workspace. | fail→fail | 11,462 | 6,929 | -40% | 1 | 1 | 0% | 360 | 566 | +57% | 0 | 0 | — |
▸case-16 Create Kubernetes deployment manifests in /workspace/k8s/ including deployment.yaml, service.yaml, and ingress.yaml. A common misconfiguration is mismatched label selectors between Deployment and Service. Save all three files sequentially, ensuring Service selectors match Deployment template labels. | fail→fail | 4,310 | 17,718 | +311% | 1 | 1 | 0% | 227 | 538 | +137% | 0 | 0 | — |
▸case-17 Write design system token files in /workspace/design-tokens: colors.json, typography.json, and spacing.json. Ensure colors.json uses standard 6-character hex color format (`#RRGGBB`). Save each design token JSON file sequentially to disk. | fail→fail | 27,035 | 42,929 | +59% | 1 | 1 | 0% | 1,925 | 8,370 | +335% | 0 | 0 | — |
▸case-18 Generate GraphQL schema definition files in /workspace/graphql: user.graphql, product.graphql, and order.graphql. Avoid using non-standard type syntax. Write each schema file sequentially, ensuring proper GraphQL ID type and relationship directives. | pass→fail | 17,950 | 15,672 | -13% | 1 | 1 | 0% | 2,610 | 1,086 | -58% | 0 | 0 | — |
▸case-19 Draft server setup Bash scripts in /workspace/scripts/: 01-common.sh, 02-nginx.sh, and 03-db.sh. Beginners often forget strict error handling, causing scripts to run past failed commands. Save these scripts to disk sequentially using proper shebang lines and strict mode settings. | fail→fail | 14,555 | 13,032 | -10% | 1 | 1 | 0% | 216 | 598 | +177% | 0 | 0 | — |
▸case-20 Set up CI/CD pipeline definition files in /workspace/.github/workflows: build.yml, test.yml, and deploy.yml. A common syntax error in GitHub Actions is omitting the `on:` trigger mapping. Write each workflow file sequentially, configuring appropriate workflow trigger events. | fail→pass | 9,579 | 26,488 | +177% | 1 | 1 | 0% | 251 | 4,063 | +1519% | 0 | 0 | — |
▸case-21 Build Express.js routing files in /workspace/routes: authRoutes.js, userRoutes.js, and productRoutes.js. A common mistake is using `app.get()` instead of `express.Router()`. Write each route file sequentially using standard Router instances and export them with CommonJS `module.exports`. | pass→fail | 11,286 | 16,140 | +43% | 1 | 1 | 0% | 2,611 | 584 | -78% | 0 | 0 | — |
▸case-22 Create localization JSON files in /workspace/locales: en.json, es.json, and fr.json. Developers often create mismatched key hierarchies across locale files. Save each locale file sequentially, keeping identical nested JSON key paths across all three languages. | fail→fail | 38,145 | 16,475 | -57% | 1 | 1 | 0% | 5,158 | 606 | -88% | 0 | 0 | — |