▸case-01 We are migrating our legacy payment service to AWS, but there are huge gaps in the requirements around regulatory compliance, user checkout flows, and database isolation. Can you identify the critical execution blockers, ask me the key individual questions needed to resolve them alongside the business rationale and safe enterprise defaults for each, and keep track of what remains open? | fail→pass | 18,734 | 11,813 | -37% | 1 | 1 | 0% | 3,094 | 2,120 | -31% | 0 | 0 | — |
▸case-02 Our team wants to build a new real-time health data sync feature for our mobile app, but the technical specification is vague and full of high-risk unknowns. Please review the missing details and give me a few concise, isolated questions to unblock planning. For each question, explain the underlying impact and suggest a conservative default choice. | fail→pass | 10,522 | 11,495 | +9% | 1 | 1 | 0% | 1,721 | 1,961 | +14% | 0 | 0 | — |
▸case-03 We are starting a massive refactoring of our multi-tenant authorization system, but we're stuck because several core requirements are totally undefined. Highlight the major blockers by asking one focused question per decision point, complete with why each decision is important and a recommended safe baseline option, so we can systematically resolve them. | fail→pass | 12,317 | 8,311 | -33% | 1 | 1 | 0% | 2,106 | 1,579 | -25% | 0 | 0 | — |
▸case-04 We have a fully refined user story for adding CSV export to the user audit dashboard, complete with accepted wireframes, explicit database query limits, and signed-off security specs. Please break this fully specified feature down into standard Jira implementation tasks with estimated story points. | pass→fail | 13,641 | 9,027 | -34% | 1 | 1 | 0% | 2,407 | 1,607 | -33% | 0 | 0 | — |
▸case-05 Our primary PostgreSQL cluster suffered a 45-minute outage yesterday due to disk space exhaustion from unrotated connection logs. The root cause and timeline are fully documented. Draft a standard incident post-mortem report outlining timeline, root cause, immediate remediation, and preventative action items. | pass→fail | 18,487 | 16,932 | -8% | 1 | 1 | 0% | 3,117 | 2,928 | -6% | 0 | 0 | — |
▸case-06 Our architecture board already decided last week to adopt Redis as our distributed caching layer over Memcached due to pub/sub requirements. Write a standard Architecture Decision Record (ADR) documenting this context, decision, and consequences. | pass→pass | 12,792 | 11,506 | -10% | 1 | 1 | 0% | 2,011 | 2,031 | +1% | 0 | 0 | — |
▸case-07 We are designing a flash-sale inventory reservation engine. We have doubts about Redis cluster sizing, locking timeouts, fraud detection limits, and checkout UX messaging under high load. Ask the necessary clarification questions with enterprise defaults and rationale, tracking open items. | fail→fail | 20,719 | 12,392 | -40% | 1 | 1 | 0% | 3,107 | 2,063 | -34% | 0 | 0 | — |
▸case-08 We are building an outbound SMS notification system for patient appointment reminders in our healthcare application. We need decisions on SMS gateway vendor encryption, retry backoff algorithms, patient consent opt-in scope, and message template customization. Give me the key decision questions with conservative defaults. | fail→fail | 19,412 | 12,393 | -36% | 1 | 1 | 0% | 3,026 | 2,174 | -28% | 0 | 0 | — |
▸case-09 Our fintech platform is introducing wire transfer automation. We do not know whether to require multi-factor approval above $10,000, what fallback mechanism to use if the core banking API times out, or how to handle ledger reconciliation errors. Provide the blocking questions, safe defaults, and track them. | pass→pass | 17,874 | 8,889 | -50% | 1 | 1 | 0% | 2,746 | 1,591 | -42% | 0 | 0 | — |
▸case-10 We are migrating EU customer analytics workloads from on-premise servers to AWS S3 and Snowflake. Key uncertainties exist around cross-region replication, GDPR data sovereignty boundaries, network peering latency, and dashboard refresh rates. Present the clarification questions with business impact rationale. | pass→pass | 13,666 | 10,894 | -20% | 1 | 1 | 0% | 2,196 | 1,977 | -10% | 0 | 0 | — |
▸case-11 We are adding FaceID/TouchID authentication to our mobile banking app. Unresolved areas include fallback password prompts, offline token expiration duration, biometric prompt failure UX messaging, and keychain hardware security module settings. Ask the critical questions with safe enterprise baselines. | fail→fail | 19,049 | 11,330 | -41% | 1 | 1 | 0% | 2,925 | 1,837 | -37% | 0 | 0 | — |
▸case-12 We are building a multi-tenant SaaS CRM. We haven't decided between database-per-tenant vs row-level security, tenant offboarding data deletion windows, or custom domain SSL termination. Ask the blocking questions required to proceed safely. | fail→pass | 15,724 | 8,996 | -43% | 1 | 1 | 0% | 2,371 | 1,618 | -32% | 0 | 0 | — |
▸case-13 We are deploying 50,000 smart factory sensors emitting telemetry every second. Critical unknowns include device payload encryption standards, data retention policy duration, edge drop policies when connection fails, and MQTT broker cluster size. Ask targeted questions with business rationale and safe defaults. | pass→pass | 15,783 | 12,341 | -22% | 1 | 1 | 0% | 2,665 | 2,076 | -22% | 0 | 0 | — |
▸case-14 Our global payroll engine needs a automated failover mechanism for monthly distribution runs. Gaps exist in RPO/RTO compliance targets, manual override authority permissions, backup banking API choices, and notification templates during failure. Identify critical execution blockers with defaults. | fail→fail | 22,297 | 10,277 | -54% | 1 | 1 | 0% | 3,174 | 1,714 | -46% | 0 | 0 | — |
▸case-15 We are launching an AI customer support bot that can issue refund credits up to $50. Unknowns remain regarding human supervisor escalation triggers, maximum daily refund caps per user, LLM model provider fallback, and prompt logging privacy standards. Formulate clarification questions with safe defaults. | fail→pass | 14,530 | 10,387 | -29% | 1 | 1 | 0% | 2,254 | 1,968 | -13% | 0 | 0 | — |
▸case-16 We are updating our checkout pipeline to support saved credit cards via third-party tokenization. Unknowns include PCI-DSS audit boundary definition, user consent UX for card saving, token invalidation policy on password reset, and database column encryption algorithms. Ask key questions with safe defaults. | fail→pass | 15,922 | 11,272 | -29% | 1 | 1 | 0% | 2,498 | 1,985 | -21% | 0 | 0 | — |
▸case-17 We are implementing mTLS across 200 Kubernetes microservices using Istio. High risks include CA certificate rotation frequency, fallback behavior during CA outage, service account permission scopes, and sidecar CPU limits. Provide blocking questions with rationale and defaults. | fail→pass | 19,625 | 12,236 | -38% | 1 | 1 | 0% | 2,810 | 1,970 | -30% | 0 | 0 | — |
▸case-18 We are developing an automated self-service returns portal for online retail. Key ambiguities include automated return approval thresholds, return shipping fee deductibility rules, instant store credit vs bank refund options, and warehouse inventory sync polling frequency. Ask questions with rationale. | fail→fail | 16,257 | 10,504 | -35% | 1 | 1 | 0% | 2,335 | 1,772 | -24% | 0 | 0 | — |
▸case-19 We are adding Enterprise SAML 2.0 single sign-on to our web app. Key unknowns surround Just-In-Time (JIT) user provisioning role mapping, session timeout duration, default tenant permission assignment, and IdP certificate rollover handling. Ask the critical questions with safe baselines. | fail→pass | 17,404 | 10,848 | -38% | 1 | 1 | 0% | 2,702 | 1,949 | -28% | 0 | 0 | — |
▸case-20 We are implementing withdrawal processing for a cryptocurrency exchange app. Core unknowns include mandatory delayed withdrawal windows for large amounts, cold-storage authorization workflow, email verification UX flow, and gas fee dynamic calculation strategies. Ask clarification questions with safe defaults. | fail→fail | 14,246 | 13,130 | -8% | 1 | 1 | 0% | 2,048 | 2,171 | +6% | 0 | 0 | — |
▸case-21 We are integrating Segment analytics across our web and mobile applications. Key gaps exist in anonymous user ID persistence duration, cookie consent banner compliance blocking, event payload PII scrubbing rules, and batch flush intervals. Ask key questions with rationale. | fail→pass | 20,146 | 12,934 | -36% | 1 | 1 | 0% | 2,943 | 2,073 | -30% | 0 | 0 | — |
▸case-22 We are launching a public developer API portal. Unresolved choices include free-tier rate limits, API key revocation triggers, developer account verification steps, and Redis rate-limiter bucket refill algorithms. Ask clarification questions with impact rationale. | fail→pass | 13,440 | 10,157 | -24% | 1 | 1 | 0% | 2,023 | 1,742 | -14% | 0 | 0 | — |