▸case-05 Draft a Product Requirements Document (PRD) for adding social sign-on (OAuth2 via Google and Github) to our customer portal including user stories and acceptance criteria. | pass→pass | 25,614 | 28,707 | +12% | 1 | 1 | 0% | 3,508 | 4,315 | +23% | 0 | 0 | — |
▸case-01 Please analyze our user performance metrics for the workspace collaboration feature for October. Here is the dataset:
- Acquisition CAC: $45 current vs $35 target ($38 in Sept)
- Time to First Value: 14 mins current vs 8 mins target (9 mins in Sept)
- Feature Adoption Rate: 62% current vs 60% target (55% in Sept)
- Monthly Churn Rate: 4.8% current vs 3.0% target (3.2% in Sept)
Provide a full health review formatted with an overall health rating, a table mapping metrics against goals and prior period changes, jargon-free observations explaining trends, the top 3 diagnostic areas needing investigation along with likely hypotheses, and owner-assigned action items. | fail→fail | 22,990 | 24,253 | +5% | 1 | 1 | 0% | 3,011 | 3,688 | +22% | 0 | 0 | — |
▸case-02 I need a comprehensive product health review for our core SaaS dashboard for November. Inputs:
- MAU Growth: Actual +8% | Target +15% | Oct +12%
- Onboarding Completion: Actual 52% | Target 70% | Oct 68%
- Average Session Depth: Actual 4.2 pages | Target 5.0 | Oct 4.8
- D30 Retention: Actual 18% | Target 28% | Oct 27%
Please transform these metrics into a complete health assessment. I need an overall product health status, a summary table comparing actual vs target and month-over-month shifts, bulleted observations tailored for stakeholders, three priority diagnostic items with suggested root causes, and recommended next steps with owners and timelines. | fail→fail | 31,908 | 26,098 | -18% | 1 | 1 | 0% | 3,131 | 3,932 | +26% | 0 | 0 | — |
▸case-03 Write a SQL query for PostgreSQL to aggregate daily active users (DAU) and monthly active users (MAU) over the last 90 days from our raw event log table user_events. | pass→pass | 13,001 | 18,082 | +39% | 1 | 1 | 0% | 2,575 | 2,789 | +8% | 0 | 0 | — |
▸case-04 Calculate the required sample size per variant for a two-tailed A/B test with baseline conversion rate of 12%, minimum detectable effect of 15% relative lift, 80% statistical power, and alpha of 0.05. | pass→pass | 19,026 | 18,936 | -0% | 1 | 1 | 0% | 3,150 | 3,912 | +24% | 0 | 0 | — |
▸case-06 Evaluate the product health for the mobile banking app for Q3. We noticed that onboarding completion fell from 80% to 55% in July (target 75%), followed by a drop in D30 retention from 40% to 22% in August (target 35%). New signup volume remained steady at 10,000/month. Provide a complete product health assessment with observations, root causes, and recommendations. | fail→pass | 29,655 | 22,446 | -24% | 1 | 1 | 0% | 3,299 | 3,722 | +13% | 0 | 0 | — |
▸case-07 Review the health of our dev tool platform for December. Metrics: Signup conversion rate is 4.8% vs target 5.0% (-4% variance); Time to first API call is 25 mins vs target 15 mins (+66% variance); Session length is 18 mins vs target 20 mins (-10% variance); Churn rate is 6.5% vs target 2.0% (+225% variance). Produce the full health review report. | pass→pass | 26,273 | 25,900 | -1% | 1 | 1 | 0% | 2,871 | 3,832 | +33% | 0 | 0 | — |
▸case-08 Analyze October performance for our E-commerce Checkout Flow: Mobile conversion rate is 1.8% vs 3.5% target; Cart abandonment is 78% vs 60% target; Checkout load time is 4.2s vs 1.5s target. Provide the health report with action items. | fail→pass | 23,950 | 14,330 | -40% | 1 | 1 | 0% | 2,515 | 3,108 | +24% | 0 | 0 | — |
▸case-09 Summarize November health metrics for executive stakeholders for our cloud storage product: API latency p99 spiked to 850ms (target 200ms), DB lock contention caused auth timeouts for 12% of sessions, resulting in DAU dropping from 50k to 41k. Provide key observations suitable for non-technical executives. | pass→pass | 17,850 | 44,396 | +149% | 1 | 1 | 0% | 1,716 | 2,904 | +69% | 0 | 0 | — |
▸case-10 Assess overall product health for our B2B SaaS analytics tool for Q3. Total platform DAU/MAU ratio appears stable at 42% (target 40%). However, Enterprise segment engagement dropped from 50% to 28% while Free Tier activity grew from 30% to 48%. How should this be presented in the observations and health assessment? | pass→pass | 32,182 | 23,018 | -28% | 1 | 1 | 0% | 2,157 | 3,619 | +68% | 0 | 0 | — |
▸case-16 Evaluate product health for our enterprise messaging app in Q4: Voice call volume dropped 18% below target, file upload error rate rose to 4.5% (target <1%), and DAU/MAU dropped from 65% to 52% (target 60%). Provide diagnostic investigations. | pass→pass | 24,654 | 25,857 | +5% | 1 | 1 | 0% | 3,125 | 3,625 | +16% | 0 | 0 | — |
▸case-11 Assess overall product health for the fitness tracking application for January using these metrics: Paid Ad Signups: 5,000 vs 6,000 target; First Workout Completion Rate: 45% vs 70% target; Weekly Active Workouts: 3.2 vs 4.0 target; Month 1 Retention Rate: 25% vs 40% target. Structure the review across the standard product layers. | pass→pass | 21,549 | 21,766 | +1% | 1 | 1 | 0% | 2,718 | 3,653 | +34% | 0 | 0 | — |
▸case-12 Analyze health for our subscription video app for November: Monthly Recurring Revenue (MRR) grew +15% to $500k (target $450k), but daily watch time per user fell from 45 mins to 25 mins (target 40 mins) and cancellation requests increased +35%. Generate a health report clarifying how revenue metrics relate to core product health. | pass→pass | 23,114 | 29,791 | +29% | 1 | 1 | 0% | 2,838 | 4,023 | +42% | 0 | 0 | — |
▸case-13 Evaluate November health for our collaboration app: Search feature utilization is 92% against a target of 50%, but user survey feedback indicates users search repeatedly because navigation menus are broken. Should search utilization be marked as healthy green? | fail→pass | 27,026 | 18,226 | -33% | 1 | 1 | 0% | 1,642 | 2,734 | +67% | 0 | 0 | — |
▸case-14 Review the quarterly product metrics for our workflow automation tool: Core feature retention is down 25% below target, signup completion dropped by 30%, while session counts remain flat. Provide the overall health rating banner using standard product health status classifications. | fail→pass | 16,190 | 22,613 | +40% | 1 | 1 | 0% | 1,151 | 3,483 | +203% | 0 | 0 | — |
▸case-15 Generate a summary metric table for our CRM platform's October metrics: Lead Import Success Rate (Current 88%, Target 98%, Sept 95%), Contact Export Time (Current 12s, Target 3s, Sept 4s). Follow standard metric reporting table formatting. | fail→fail | 14,971 | 18,482 | +23% | 1 | 1 | 0% | 1,309 | 2,515 | +92% | 0 | 0 | — |
▸case-17 I want to assess the health of our mobile application. How should I proceed with this health evaluation? | fail→pass | 22,493 | 12,878 | -43% | 1 | 1 | 0% | 2,428 | 2,015 | -17% | 0 | 0 | — |
▸case-18 Analyze our mobile gaming app retention metrics for Q2: D1 Retention 35% vs 40% target, D7 Retention 18% vs 25% target, D30 Retention 8% vs 15% target, Resurrected User Rate 2% vs 5% target, Churn Rate 12% vs 8% target. Build the health review focus on retention performance. | pass→pass | 36,316 | 27,459 | -24% | 1 | 1 | 0% | 3,281 | 3,871 | +18% | 0 | 0 | — |
▸case-19 Evaluate October acquisition metrics for our B2B SaaS platform: Total Signups reached 12,000 vs 10,000 target (+20%), but Paid Channel CAC spiked to $220 vs $140 target (+57%), and Day 7 Activation for paid signups was 12% vs 35% target. Analyze whether acquisition is truly healthy. | pass→pass | 16,974 | 32,060 | +89% | 1 | 1 | 0% | 2,475 | 3,729 | +51% | 0 | 0 | — |
▸case-20 Assess November metrics for our productivity suite where 6 metrics missed targets: Onboarding (30% vs 50%), DAU (10k vs 20k), Session depth (2 vs 5), Export failure rate (8% vs 1%), Sharing frequency (0.5 vs 2.0), Renewal intent (40% vs 70%). Structure the priority investigation section. | fail→pass | 36,384 | 25,827 | -29% | 1 | 1 | 0% | 2,885 | 4,677 | +62% | 0 | 0 | — |
▸case-21 Evaluate product health for our news aggregator app in December: DAU/MAU is 45% vs 40% target, but Session Depth is 1.2 articles vs 3.5 target, and Feature Adoption for 'Saved Articles' is 5% vs 25% target. Generate the observations and diagnostic focus. | pass→pass | 14,975 | 22,456 | +50% | 1 | 1 | 0% | 2,202 | 3,871 | +76% | 0 | 0 | — |
▸case-22 Assess January health metrics for our healthcare portal: Appointment Bookings current is 4,500 (target 5,000, Dec 5,200), Video Consult Duration current is 12 mins (target 15 mins, Dec 14 mins). Ensure trend analysis is complete. | pass→pass | 13,926 | 18,553 | +33% | 1 | 1 | 0% | 2,365 | 3,334 | +41% | 0 | 0 | — |
▸case-23 Review health metrics for our task management tool in Q3: Active projects per team is 2.1 vs 4.0 target (-47.5%). Provide a product health review. | pass→pass | 24,522 | 24,104 | -2% | 1 | 1 | 0% | 2,146 | 4,729 | +120% | 0 | 0 | — |
▸case-24 Review product health for our developer API tool in Q3: Time to first successful API request is 45 minutes vs 10 minutes target. Onboarding completion is 38% vs 75% target. Provide a product health assessment focusing on activation. | pass→pass | 25,077 | 22,469 | -10% | 1 | 1 | 0% | 2,495 | 3,165 | +27% | 0 | 0 | — |