▸case-02 Our revenue team is experiencing data quality issues due to schema drifts across releases. Please draft an analytics instrumentation governance plan for us, including an observability runbook and a version control process for updating tracking definitions. | fail→fail | 23,622 | 17,807 | -25% | 1 | 1 | 0% | 4,142 | 3,304 | -20% | 0 | 0 | — |
▸case-01 I'm launching a new self-serve subscription flow and need to design our analytics event architecture from scratch. Please build a complete tracking specification covering event definitions, user identity mapping, privacy compliance, and schema change monitoring in a structured implementation guide. | fail→fail | 30,621 | 43,325 | +41% | 1 | 1 | 0% | 6,190 | 5,112 | -17% | 0 | 0 | — |
▸case-03 We need to optimize a dbt incremental model that processes raw billing tables into daily active user aggregates in Snowflake. How should we configure the unique_key and incremental strategy? | pass→pass | 15,474 | 13,792 | -11% | 1 | 1 | 0% | 2,991 | 2,805 | -6% | 0 | 0 | — |
▸case-04 How should our data team configure Snowflake role-based access control (RBAC) to grant RevOps analysts read access to raw event tables while hiding PII? | pass→pass | 14,366 | 15,025 | +5% | 1 | 1 | 0% | 2,868 | 3,037 | +6% | 0 | 0 | — |
▸case-05 We are setting up an Apache Airflow DAG to ingest daily Salesforce API extracts into our Redshift staging schema. What task topology and catchup settings should we use? | pass→pass | 19,331 | 21,492 | +11% | 1 | 1 | 0% | 3,652 | 4,163 | +14% | 0 | 0 | — |
▸case-06 We are building a tracking plan spreadsheet for our checkout funnel events in Google Sheets. Usually developers just list the event name and parameters. What standard columns must be included to ensure full operational ownership and accountability across engineering and product? | pass→pass | 13,950 | 12,015 | -14% | 1 | 1 | 0% | 2,313 | 2,341 | +1% | 0 | 0 | — |
▸case-07 Our mobile developers name events inconsistently (e.g., 'clickedSubmitButton', 'user_signed_up', 'PurchaseSuccess'). We are standardizing our event naming scheme. Should events be named after UI elements or user actions, and what structural style should be mandated? | pass→pass | 13,503 | 11,616 | -14% | 1 | 1 | 0% | 2,404 | 2,329 | -3% | 0 | 0 | — |
▸case-08 We are establishing data contracts between backend services and the central analytics warehouse. Engineering wants to include field names and data types, but data consumers complain schema breaking changes still break downstream models. What additional validation criteria must be part of the data contract checklist? | pass→pass | 14,624 | 14,325 | -2% | 1 | 1 | 0% | 2,525 | 2,677 | +6% | 0 | 0 | — |
▸case-09 Our SaaS platform tracks users across anonymous web sessions, logged-in web applications, and mobile devices. A developer suggests using only a global user_id for tracking. What identity fields and mapping strategy should be architected to handle pre-login activity and multi-tenant accounts? | fail→pass | 17,468 | 15,452 | -12% | 1 | 1 | 0% | 3,043 | 3,005 | -1% | 0 | 0 | — |
▸case-10 When product managers write specifications for new event tracking, they usually just define the event name and property names in Jira tickets. When engineers release the code, the payloads frequently contain unexpected types or missing properties. What practice should be added to every event definition ticket? | fail→fail | 8,790 | 9,827 | +12% | 1 | 1 | 0% | 1,624 | 2,062 | +27% | 0 | 0 | — |
▸case-11 Where should tracking plan definitions reside to prevent tracking implementations from falling out of sync with production app releases? PMs currently keep them in Notion pages. | pass→pass | 14,913 | 12,355 | -17% | 1 | 1 | 0% | 2,366 | 2,183 | -8% | 0 | 0 | — |
▸case-12 How frequently should product, analytics, and RevOps stakeholders conduct formal reviews of the tracking plan and analytics instrumentation? | pass→pass | 12,914 | 3,874 | -70% | 1 | 1 | 0% | 2,047 | 878 | -57% | 0 | 0 | — |
▸case-13 Our data team wants to create an observability runbook for monitoring tracking health. Engineering proposes logging raw HTTP status codes from the collector. What specific operational capabilities and notification mechanisms must the runbook define for anomaly detection? | fail→pass | 18,929 | 15,310 | -19% | 1 | 1 | 0% | 3,124 | 2,783 | -11% | 0 | 0 | — |
▸case-14 We operate a global e-commerce application in Europe and North America. Engineering plans to handle privacy by applying blanket SHA256 hashing to email addresses. What core consent and privacy components must be explicitly built into our tracking standards? | fail→fail | 17,135 | 17,222 | +1% | 1 | 1 | 0% | 2,713 | 3,229 | +19% | 0 | 0 | — |
▸case-15 We need to modify the schema for our 'checkout_completed' event to add multi-currency breakdown, but existing downstream pipelines rely on the existing schema. How should breaking changes and schema lifecycle be managed in the tracking standards? | pass→fail | 15,846 | 14,054 | -11% | 1 | 1 | 0% | 2,904 | 2,805 | -3% | 0 | 0 | — |
▸case-16 Our analytics engineering team is about to rebuild our core dbt data models in Snowflake. Before modifying the transformation layer, what tracking audit step should be performed across stakeholders? | fail→pass | 10,929 | 8,867 | -19% | 1 | 1 | 0% | 1,779 | 1,638 | -8% | 0 | 0 | — |
▸case-17 RevOps, Engineering, and Product cannot agree on field naming and SLA expectations for lead conversion analytics. What process mechanism should be used to establish mutual agreement and technical enforcement? | pass→pass | 13,595 | 14,643 | +8% | 1 | 1 | 0% | 2,236 | 2,803 | +25% | 0 | 0 | — |
▸case-18 In our event tracking sheet, we have columns for event_name, description, and properties. When tracking issues arise, developers cannot tell where the event originates or whether it is live in production. What missing tracking sheet attributes resolve this? | pass→pass | 9,347 | 8,148 | -13% | 1 | 1 | 0% | 1,678 | 1,679 | +0% | 0 | 0 | — |
▸case-19 Our engineers argue that event properties should be completely optional and dynamic per developer preference. What rule should our event structure standard establish regarding properties? | pass→pass | 11,541 | 11,516 | -0% | 1 | 1 | 0% | 1,905 | 1,942 | +2% | 0 | 0 | — |
▸case-20 When monitoring high-volume clickstream events (millions of events per hour), inspecting raw event logs causes performance slowdowns in our monitoring tool. What specific dashboard capability should be included in our observability framework? | pass→pass | 10,941 | 28,978 | +165% | 1 | 1 | 0% | 1,734 | 1,638 | -6% | 0 | 0 | — |
▸case-21 A user signs up on our marketing website via HubSpot, registers in our PostgreSQL auth database, and makes payments in Stripe. Analytics cannot link these user journey stages. What identity standard requirement solves this? | pass→pass | 10,035 | 8,771 | -13% | 1 | 1 | 0% | 1,741 | 1,697 | -3% | 0 | 0 | — |
▸case-22 A team lead proposes naming analytics events after static UI components (e.g., 'header_logo', 'checkout_modal'). How should the event naming policy correct this approach? | pass→pass | 11,477 | 9,477 | -17% | 1 | 1 | 0% | 1,868 | 1,990 | +7% | 0 | 0 | — |