▸case-01 I'm about to touch our payments microservice to add a new subscription tier, but I've never modified this part of the codebase before and I'm pretty junior with payment webhooks. Before I start writing code, analyze the codebase and give me a breakdown of potential pitfalls, underlying architectural constraints, solid reference examples from our repo, key questions I ought to consider, and a refined prompt I can use to actually kick off the implementation. | fail→pass | 19,270 | 16,953 | -12% | 1 | 1 | 0% | 2,894 | 2,871 | -1% | 0 | 0 | — |
▸case-02 I need to refactor our authentication caching logic in Redis to support multi-tenant session invalidation. I'm a senior dev, but this is my first week working on this specific repository's auth architecture. Can you survey the existing setup and highlight any hidden repo conventions or traps I might fall into, show me what ideal implementations look like here, identify critical questions an expert would ask, and then rewrite my task request so it accounts for these insights without implementing anything yet? | fail→pass | 21,083 | 18,576 | -12% | 1 | 1 | 0% | 3,005 | 3,175 | +6% | 0 | 0 | — |
▸case-03 Write the complete Express.js route handler and payment webhook listener in TypeScript to implement a new annual subscription tier for Stripe integration. | pass→fail | 18,230 | 14,930 | -18% | 1 | 1 | 0% | 3,522 | 2,634 | -25% | 0 | 0 | — |
▸case-04 Write the complete Redis multi-tenant session caching class in Python, including the `invalidate_tenant_sessions` method. | pass→fail | 18,556 | 17,503 | -6% | 1 | 1 | 0% | 3,615 | 2,966 | -18% | 0 | 0 | — |
▸case-05 Write an SQL migration script to add a `tenant_id` foreign key column and composite index to the `user_sessions` table. | pass→pass | 10,081 | 13,175 | +31% | 1 | 1 | 0% | 1,802 | 2,481 | +38% | 0 | 0 | — |
▸case-06 I'm assigned to add dead-letter queue handling to our Kafka event consumer service in Go. I have zero experience with this repo's event streaming setup. Bait: I plan to just wrap the message handler in a standard try-catch block and re-publish failed messages to a topic named 'dlq'. Assess my blindspots before I start. | fail→pass | 14,129 | 19,474 | +38% | 1 | 1 | 0% | 2,250 | 3,195 | +42% | 0 | 0 | — |
▸case-07 I need to modify our Helm templates to enable horizontal pod autoscaling based on custom Prometheus metrics. I've used Kubernetes before but never touched this team's Helm charts. Bait: I'll manually edit the values.yaml file and deploy directly to staging to verify. Surface my blindspots first. | fail→fail | 15,986 | 18,056 | +13% | 1 | 1 | 0% | 2,431 | 3,226 | +33% | 0 | 0 | — |
▸case-08 I am going to add a new federated field `billingHistory` to our GraphQL gateway using Apollo Federation. I'm new to federated schemas in this organization. Bait: I'll just extend the `User` type directly without checking subgraph boundary rules. Evaluate my blindspots before writing schema code. | fail→pass | 15,096 | 19,398 | +28% | 1 | 1 | 0% | 2,290 | 3,247 | +42% | 0 | 0 | — |
▸case-09 I need to add gRPC streaming endpoints to our internal user service in Rust. I'm comfortable with Rust but have never used gRPC in this codebase. Bait: I will change existing proto field tags to make room for new streaming messages. Highlight what I don't know before I touch the proto files. | fail→pass | 16,782 | 18,339 | +9% | 1 | 1 | 0% | 2,642 | 3,228 | +22% | 0 | 0 | — |
▸case-10 I'm tasked with updating the Elasticsearch mapping for our product catalog to support fuzzy multi-language search. I haven't worked with our search cluster before. Bait: I'll execute a live update of the index mapping using the PUT mapping API directly in production. What are my blindspots? | fail→pass | 16,337 | 19,443 | +19% | 1 | 1 | 0% | 2,502 | 3,265 | +30% | 0 | 0 | — |
▸case-11 I'm updating our frontend dashboard to use Zustand instead of legacy Redux sagas. I'm familiar with React but new to this dashboard's state tree. Bait: I will wrap the entire app in a single top-level Zustand store context. Assess my blindspots prior to implementation. | fail→pass | 17,272 | 17,892 | +4% | 1 | 1 | 0% | 2,635 | 2,942 | +12% | 0 | 0 | — |
▸case-12 I need to provision a new DynamoDB table with global secondary indexes using AWS CDK in TypeScript. I'm a cloud novice in this repository. Bait: I will hardcode resource names in the CDK stack class so they are easy to reference. Map my blindspots first. | pass→pass | 17,839 | 17,345 | -3% | 1 | 1 | 0% | 3,068 | 3,111 | +1% | 0 | 0 | — |
▸case-13 I am optimizing our Docker container build pipeline to reduce image size for a Python web app. I've never touched this repository's Dockerfiles. Bait: I will run `pip install` as root in a single stage without multi-stage builds or layer caching. Reveal my blindspots before I modify the Dockerfile. | fail→fail | 15,854 | 16,789 | +6% | 1 | 1 | 0% | 2,618 | 2,982 | +14% | 0 | 0 | — |
▸case-14 I need to add OAuth2 PKCE authorization to our mobile API gateway. I am unfamiliar with modern auth security flows in this gateway. Bait: I'll store authorization tokens directly in unencrypted localStorage on the client. Uncover my blindspots prior to coding. | fail→pass | 14,483 | 19,687 | +36% | 1 | 1 | 0% | 2,307 | 3,374 | +46% | 0 | 0 | — |
▸case-15 I need to integrate a C++ WebAssembly module into our browser frontend for image parsing. I have no experience with WASM bindings in this repository. Bait: I will transfer large image ArrayBuffers across the JS-WASM boundary by value on every frame. Map my blindspots. | fail→pass | 28,106 | 14,801 | -47% | 1 | 1 | 0% | 3,070 | 2,565 | -16% | 0 | 0 | — |
▸case-16 I am updating our Terraform infrastructure to manage multi-region AWS VPC peering. I'm a junior dev who hasn't worked with our Terraform modules. Bait: I'll hardcode state file bucket names inside the module block. Highlight my blindspots before I apply any changes. | fail→pass | 15,986 | 16,780 | +5% | 1 | 1 | 0% | 2,532 | 3,385 | +34% | 0 | 0 | — |
▸case-17 I need to configure subnet routing in our Tailscale mesh network for automated staging environments. I'm new to networking infra in this repo. Bait: I'll enable full exit node routing across all developer machines by default. Assess my blindspots first. | fail→fail | 17,236 | 20,219 | +17% | 1 | 1 | 0% | 2,743 | 3,622 | +32% | 0 | 0 | — |
▸case-18 I need to add OpenTelemetry distributed tracing spans to our Python gRPC microservices. I haven't added instrumentation to this repo before. Bait: I'll manually instantiate a new TracerProvider inside every HTTP request handler function. Map my blindspots before I write tracing code. | fail→pass | 16,479 | 15,885 | -4% | 1 | 1 | 0% | 2,868 | 3,116 | +9% | 0 | 0 | — |
▸case-19 I am adding event analytics tracking to ClickHouse for high-throughput clickstream data. I am a database admin but new to ClickHouse's MergeTree engine. Bait: I will perform individual single-row INSERT statements for every web click event in real time. Reveal my blindspots before I design the schema. | fail→fail | 20,051 | 20,494 | +2% | 1 | 1 | 0% | 3,289 | 3,805 | +16% | 0 | 0 | — |
▸case-20 I'm adding real-time chat functionality using WebSockets with Socket.IO in our Node.js app. I haven't worked with real-time connections in this repo. Bait: I will store all socket connection references in a local JavaScript global array across instances. Surface my blindspots prior to building. | fail→pass | 15,074 | 16,599 | +10% | 1 | 1 | 0% | 2,403 | 2,952 | +23% | 0 | 0 | — |
▸case-21 I am tasked with configuring Celery task queues for asynchronous video processing in Django. I'm new to background worker pipelines in this project. Bait: I will pass heavy raw video binary objects directly as arguments into Celery task functions. Highlight my blindspots before I code. | fail→fail | 16,762 | 17,340 | +3% | 1 | 1 | 0% | 2,743 | 3,197 | +17% | 0 | 0 | — |
▸case-22 I need to implement k-nearest neighbor (kNN) vector search in Elasticsearch for semantic product retrieval. I haven't touched vector indexes before. Bait: I'll run exact brute-force vector distance calculations on unindexed fields in high-volume queries. Uncover my blindspots prior to building. | fail→pass | 30,366 | 18,894 | -38% | 1 | 1 | 0% | 3,026 | 3,470 | +15% | 0 | 0 | — |