▸case-03 We are adding a new SMS notifications feature for user password resets. The brief states that our auth service triggers the notifications, but also mentions our audit logger service, customer support portal, and cellular gateway provider (Twilio). We know Security Ops owns auth service and Customer Success owns the support portal, but data flows and gateway ownership are omitted. Provide a context mapping breakdown for this change. | fail→pass | 17,991 | 8,725 | -52% | 1 | 1 | 0% | 2,710 | 1,996 | -26% | 0 | 0 | — |
▸case-04 Our mobile app needs to display real-time stock quotes. The backend connects to an internal caching service (owned by Data Infrastructure) and a third-party market data provider (Refinitiv) via WebSocket streams. The mobile client team owns the mobile app. Build a context map outlining the systems, flows, owners, and external dependencies for this stock quote feature. | pass→pass | 17,693 | 9,402 | -47% | 1 | 1 | 0% | 2,906 | 2,046 | -30% | 0 | 0 | — |
▸case-01 Our engineering team is preparing to upgrade our subscription billing pipeline, involving our checkout microservice, customer database, enterprise CRM, and payment processor. Can you help us establish a complete context map for this initiative? I need a breakdown listing the active software systems in play, their data integrations, the respective team or role maintainers, and any outside vendor contracts or external dependencies. Please call out any places where details are unclear so we know what to clarify. | fail→pass | 20,435 | 10,362 | -49% | 1 | 1 | 0% | 2,987 | 2,279 | -24% | 0 | 0 | — |
▸case-02 I am drafting the architecture specification for an automated inventory sync system connecting our Shopify storefront, internal warehouse database, and logistics partner APIs. Please generate a context mapping outline for this domain. It should cover all current platforms in the problem scope, the communication channels between them, ownership details for each system, and any external party dependencies. Highlight any unknown details or open questions that require further clarification. | fail→pass | 23,830 | 16,906 | -29% | 1 | 1 | 0% | 3,699 | 3,261 | -12% | 0 | 0 | — |
▸case-05 We are migrating user account balances from a monolithic database to a new ledger service. The legacy monolith is maintained by Core Backend, and accounting auditing reads directly from the monolith's read-replica. An external fraud detection API (Sift) evaluates every high-value transaction. Map out the existing context including systems, integrations, system owners, and dependencies. | fail→pass | 14,044 | 11,117 | -21% | 1 | 1 | 0% | 2,169 | 2,269 | +5% | 0 | 0 | — |
▸case-06 Our e-commerce portal is integrating an automated address validation feature at checkout. We use an internal address-formatting microservice (owned by Checkout Engineering) and Google Maps Geocoding API. The shipping calculation service also consumes formatted addresses from the address service. Construct a context mapping summary for this integration. | pass→pass | 16,354 | 9,674 | -41% | 1 | 1 | 0% | 2,557 | 2,215 | -13% | 0 | 0 | — |
▸case-07 A project brief states: 'We need to update our email preference center. Systems involved include our user settings Web UI, marketing automation platform (HubSpot), and transactional email gateway.' The brief does not state who owns the transactional email gateway or how user settings UI communicates with HubSpot. Provide a context mapping overview for this initiative. | pass→pass | 17,432 | 8,914 | -49% | 1 | 1 | 0% | 2,587 | 2,012 | -22% | 0 | 0 | — |
▸case-08 We are redesigning our order fulfillment pipeline. Order Entry microservice (owned by Fulfillment Team) passes orders to Warehouse Management System (WMS), which notifies our 3PL logistics partner (FedEx Supply Chain) via SFTP export. The finance reporting service reads order completion events from WMS. Generate a context map for this fulfillment pipeline. | pass→pass | 15,946 | 7,052 | -56% | 1 | 1 | 0% | 2,585 | 1,840 | -29% | 0 | 0 | — |
▸case-09 Our mobile app is adding digital gift card redemption. The gift card service is maintained by Payments Team, while user loyalty points are stored in an internal Loyalty Service (owned by Growth Team). Gift card issuing relies on an external vendor API (Blackhawk Network). Produce a context map detailing systems, integrations, ownership, and external dependencies. | fail→pass | 15,835 | 11,683 | -26% | 1 | 1 | 0% | 2,526 | 2,468 | -2% | 0 | 0 | — |
▸case-20 We are building an order processing system and need to define performance SLAs and quality targets. What latency threshold, throughput rate in requests per second, and availability percentage targets should we set for our payment authorization service during peak holiday sales? | pass→fail | 16,394 | 12,527 | -24% | 1 | 1 | 0% | 2,424 | 2,274 | -6% | 0 | 0 | — |
▸case-10 We are implementing single sign-on (SSO) across our enterprise suite. The identity provider is Okta, integrated with our internal employee directory (Active Directory, owned by IT SecOps) and our HR portal (Workday). Details about who manages the Okta tenant configuration or Workday integration settings are missing from the task notes. Prepare a context map for the SSO rollout. | fail→pass | 17,417 | 9,558 | -45% | 1 | 1 | 0% | 2,770 | 2,158 | -22% | 0 | 0 | — |
▸case-11 Our telemetry team is updating our observability stack. Systems involved: Application Services, OpenTelemetry Collector (owned by Platform Infra), Datadog Cloud SaaS, and PagerDuty alert routing. We know Platform Infra owns the collector, but ownership for PagerDuty integration rules and Datadog account administration is unspecified. Generate the context map. | pass→pass | 16,000 | 9,760 | -39% | 1 | 1 | 0% | 2,617 | 2,253 | -14% | 0 | 0 | — |
▸case-12 We need to analyze the surrounding systems for a project integrating AI search into our documentation portal. The portal is served by a static site generator, uses Algolia for indexing, and calls OpenAI API for embeddings. Web Analytics (owned by Marketing) tracks query telemetry. Ownership of the search indexing pipeline is not documented. Draft the context map. | pass→pass | 17,535 | 9,041 | -48% | 1 | 1 | 0% | 2,774 | 2,003 | -28% | 0 | 0 | — |
▸case-13 Our billing application sends invoice PDFs to customers via SendGrid API and stores invoice metadata in PostgreSQL (owned by Finance Eng). An internal PDF renderer service creates the documents. The prompt provides no information on who maintains the PDF renderer service or what contract governs SendGrid usage limits. Produce a context mapping output. | pass→pass | 13,524 | 5,847 | -57% | 1 | 1 | 0% | 2,074 | 1,424 | -31% | 0 | 0 | — |
▸case-14 We are introducing a real-time fraud scoring check during user registration. Registration Service (owned by Identity Team) sends user risk vectors to Risk Engine (owned by Fraud Team), which queries an external credit bureau API (Experian) and logs flags to Compliance DB. Prepare a context mapping report for this workflow. | pass→pass | 23,188 | 9,394 | -59% | 1 | 1 | 0% | 3,613 | 2,117 | -41% | 0 | 0 | — |
▸case-15 Our customer support team uses Zendesk. We are building a webhook endpoint in our backend (owned by Support Engineering) to sync ticket updates to our internal CRM database. A legacy analytics pipeline reads raw CRM tables directly. System owners for the legacy analytics pipeline are unknown. Construct the context map. | pass→pass | 16,799 | 9,231 | -45% | 1 | 1 | 0% | 2,487 | 2,029 | -18% | 0 | 0 | — |
▸case-16 We are evaluating our data ingestion architecture. Event Producer (owned by Data Platform) writes events to Kafka, which streams to Snowflake data warehouse and an external compliance archiving partner (Smarsh). Operational Monitoring (owned by DevOps) reads consumer lag metrics from Kafka. Map out the existing context. | pass→pass | 14,111 | 8,901 | -37% | 1 | 1 | 0% | 2,141 | 1,940 | -9% | 0 | 0 | — |
▸case-17 Our e-learning platform is connecting to Zoom API for automated webinar creation. Systems: Course Catalog (owned by Content Team), Schedule Worker (owned by Platform Team), Zoom Video Communications, and Student Notification Service. The document does not specify who owns Student Notification Service or what data protocol Schedule Worker uses to speak to Zoom. Map out the context. | pass→pass | 13,056 | 8,190 | -37% | 1 | 1 | 0% | 2,121 | 1,968 | -7% | 0 | 0 | — |
▸case-18 We are integrating Apple Pay into our mobile checkout flow. Systems: iOS Mobile Client (owned by Mobile Team), Payment Gateway Microservice (owned by Payments Team), Apple Pay Servers, and Core Ledger. Data flows between Payment Gateway and Apple Pay Servers use HTTPS/REST. Core Ledger ownership is omitted. Draft a context map. | pass→pass | 14,791 | 8,362 | -43% | 1 | 1 | 0% | 2,479 | 1,915 | -23% | 0 | 0 | — |
▸case-19 Our SaaS application allows users to export reports to Google Drive. Systems: Web Dashboard (owned by Frontend Team), Export Worker (owned by Backend Team), Google Drive API, and Usage Metering Service. The document mentions Usage Metering receives export event counts but provides no owner or API spec for it. Provide a context map. | pass→pass | 14,645 | 7,245 | -51% | 1 | 1 | 0% | 2,270 | 1,893 | -17% | 0 | 0 | — |
▸case-21 Can you design the detailed OpenAPI 3.0 REST endpoint specification, request payload JSON schema, and HTTP error response codes for a POST /v1/orders/cancel endpoint? | pass→fail | 22,828 | 22,318 | -2% | 1 | 1 | 0% | 4,592 | 4,420 | -4% | 0 | 0 | — |
▸case-22 We are choosing between PostgreSQL and MongoDB for our new time-series sensor tracking service. Can you analyze the architectural trade-offs between relational and document stores regarding ACID compliance, write throughput, and horizontal scaling? | pass→fail | 24,397 | 25,081 | +3% | 1 | 1 | 0% | 3,224 | 3,949 | +22% | 0 | 0 | — |