▸case-02 The security team receives CVE alerts from Snyk, Dependabot, and AWS GuardDuty. A critical vulnerability (CVE-2024-1234) appeared in all three feeds today. Generate a consolidated vulnerability digest for today's scan results. A teammate wants to copy all three raw alert payloads into three sequential sections. | fail→fail | 17,028 | 8,505 | -50% | 1 | 1 | 0% | 3,260 | 1,572 | -52% | 0 | 0 | — |
▸case-01 We have automated updates from three GitHub repositories (org/api-server, org/frontend, org/worker) over the last 24 hours containing 15 open PRs, 8 merged PRs, and 4 build failures. Summarize this daily activity into a single digest for engineering leads. A coworker suggested listing each repository separately with its full commit logs. | fail→pass | 10,586 | 8,354 | -21% | 1 | 1 | 0% | 1,787 | 1,448 | -19% | 0 | 0 | — |
▸case-03 Our team handles customer inquiries on both Zendesk and Intercom. Yesterday, 40 tickets were logged regarding a payment gateway outage across both platforms. Draft a daily digest of customer issues for management. An engineer recommends creating two separate bulleted lists, one for Zendesk tickets and one for Intercom tickets. | pass→pass | 8,886 | 7,746 | -13% | 1 | 1 | 0% | 1,463 | 1,212 | -17% | 0 | 0 | — |
▸case-04 We need a weekly digest of release announcements posted across Slack #launches, Discord #announcements, and Discourse forum updates. Create a unified executive summary of major product launches this week. Someone suggested printing every chat message verbatim sorted by channel name. | pass→pass | 9,447 | 9,345 | -1% | 1 | 1 | 0% | 1,575 | 1,426 | -9% | 0 | 0 | — |
▸case-05 During an incident, error logs were generated across AWS CloudWatch, Datadog APM, and Sentry. Synthesize these log streams into an incident root-cause digest for the post-mortem. A team member suggests pasting the full raw JSON stack traces from each source sequentially. | fail→fail | 17,117 | 16,455 | -4% | 1 | 1 | 0% | 2,815 | 2,655 | -6% | 0 | 0 | — |
▸case-06 We aggregate 10 technical RSS feeds covering AI engineering. Generate a daily top-3 digest for our newsletter subscribers. An intern suggests publishing all 30 full article RSS item bodies ordered alphabetically by feed URL. | pass→pass | 16,607 | 13,959 | -16% | 1 | 1 | 0% | 2,158 | 2,162 | +0% | 0 | 0 | — |
▸case-07 Five internal microservices (Auth, Billing, Search, Storage, Notifications) report status updates via PagerDuty and status page webhooks. Draft a morning operational status digest for the SRE team. A colleague suggests creating five separate status blocks showing raw uptime percentages. | pass→pass | 11,232 | 16,082 | +43% | 1 | 1 | 0% | 1,832 | 2,207 | +20% | 0 | 0 | — |
▸case-08 Half our team uses Jira for backend tasks while the design team uses Linear. Prepare an end-of-sprint digest summarizing completed work and blockers across both tools for stakeholders. A teammate suggests outputting two completely independent reports titled 'Jira Summary' and 'Linear Summary'. | pass→pass | 10,405 | 11,237 | +8% | 1 | 1 | 0% | 1,669 | 1,787 | +7% | 0 | 0 | — |
▸case-09 End-of-day sales data flows in from both Stripe and PayPal endpoints. Generate a daily revenue and transaction digest for the finance controller. A junior accountant suggests sending two unmerged CSV printouts side-by-side. | pass→pass | 13,577 | 13,357 | -2% | 1 | 1 | 0% | 2,499 | 2,229 | -11% | 0 | 0 | — |
▸case-10 We have 4 PDF whitepapers on vector database indexing algorithms (HNSW, DiskANN, FAISS, ScaNN). Produce a comparative synthesis digest highlighting key benchmarks and trade-offs. Someone suggested writing four separate book reports summarizing each paper sequentially from page 1 to end. | pass→pass | 20,734 | 20,657 | -0% | 1 | 1 | 0% | 3,302 | 3,398 | +3% | 0 | 0 | — |
▸case-11 Synthesize media mentions of our company from Google News alerts, Substack newsletters, and Twitter/X mentions into a daily PR digest. A PR manager suggested listing every tweet and email subject line in order of receipt. | fail→fail | 13,720 | 15,792 | +15% | 1 | 1 | 0% | 2,117 | 2,298 | +9% | 0 | 0 | — |
▸case-12 Clusters in us-east-1 and eu-west-1 generated 500 K8s events (OOMKilled, ImagePullBackOff, PodFailed) in the last hour. Prepare a consolidated cluster health digest for the infrastructure lead. An operator suggests listing all 500 raw event strings grouped by cluster name. | pass→fail | 12,391 | 18,967 | +53% | 1 | 1 | 0% | 2,260 | 2,232 | -1% | 0 | 0 | — |
▸case-13 Notes were taken in 4 simultaneous breakout sessions (Engineering, Product, Marketing, Sales) during our quarterly planning offsite. Create a unified executive digest for C-level leadership. A participant suggests copying each session's raw bullet points under four separate headers. | fail→fail | 13,637 | 13,676 | +0% | 1 | 1 | 0% | 2,172 | 2,193 | +1% | 0 | 0 | — |
▸case-14 Cybersecurity threat feeds from AlienVault OTX, CISA alerts, and internal honeypots detected malicious IP scans today. Produce an actionable daily threat digest for SOC analysts. A peer suggests printing the raw IOC lists feed by feed. | pass→pass | 16,534 | 17,905 | +8% | 1 | 1 | 0% | 2,728 | 2,801 | +3% | 0 | 0 | — |
▸case-15 User engagement metrics are recorded in Google Analytics (pageviews, bounce rate) and Mixpanel (feature usage, funnel conversion). Compile a weekly product digest for product managers. A marketer suggests emailing two raw dashboard export screenshots. | pass→pass | 13,703 | 12,534 | -9% | 1 | 1 | 0% | 2,127 | 2,063 | -3% | 0 | 0 | — |
▸case-16 CPU and memory usage statistics were recorded across 50 AWS EC2 instances across 3 regions (us-east-1, us-west-2, eu-central-1). Create a daily server resource digest. An administrator suggests displaying a table with 50 individual server rows. | pass→pass | 26,081 | 24,523 | -6% | 1 | 1 | 0% | 2,808 | 1,974 | -30% | 0 | 0 | — |
▸case-17 Customer reviews came in today across iOS App Store, Google Play Store, and Trustpilot. Draft a daily feedback digest for the UX team. A team member suggested printing each review chronologically without modification. | fail→pass | 14,392 | 8,942 | -38% | 1 | 1 | 0% | 2,323 | 1,313 | -43% | 0 | 0 | — |
▸case-18 We are releasing version 2.4.0 spanning 5 microservice repositories. Create a unified customer-facing release notes digest. A developer suggests combining the git log output of all 5 repos into one giant text file. | pass→pass | 12,139 | 13,851 | +14% | 1 | 1 | 0% | 1,993 | 2,125 | +7% | 0 | 0 | — |
▸case-19 Automated test runs executed across GitLab CI, GitHub Actions, and Jenkins for our nightly build pipeline. Draft a nightly test execution digest. A tester suggests listing every single passing and failing unit test name across all three tools. | pass→pass | 12,162 | 10,097 | -17% | 1 | 1 | 0% | 1,811 | 1,728 | -5% | 0 | 0 | — |
▸case-20 Monthly billing reports were exported from AWS Cost Explorer, Azure Cost Management, and Google Cloud Billing. Create a unified cloud spend digest for FinOps. A junior analyst recommends presenting three separate spreadsheets without cross-cloud totals. | pass→pass | 15,079 | 14,102 | -6% | 1 | 1 | 0% | 2,633 | 2,603 | -1% | 0 | 0 | — |
▸case-21 Candidate pipeline updates were logged today in Greenhouse (engineering roles) and Lever (sales/marketing roles). Create a daily recruitment digest for the Head of People. A recruiter suggests sending two separate raw status emails. | fail→fail | 10,761 | 7,698 | -28% | 1 | 1 | 0% | 1,729 | 1,183 | -32% | 0 | 0 | — |
▸case-22 We need a SQL query to calculate the 30-day rolling average revenue per user from our PostgreSQL orders table. A developer asked if we should digest external news feeds for this task. Write the standard SQL query for PostgreSQL. | fail→pass | 10,878 | 10,767 | -1% | 1 | 1 | 0% | 2,109 | 2,223 | +5% | 0 | 0 | — |
▸case-23 Convert the following single XML configuration file containing database connection strings into valid JSON format. A coworker asks if you should merge this with remote server logs first. Perform the single-file XML to JSON conversion directly. | pass→pass | 6,017 | 6,872 | +14% | 1 | 1 | 0% | 990 | 1,399 | +41% | 0 | 0 | — |
▸case-24 The following Python function def calculate_total(items): return sum(item.price for item in items has a syntax error due to missing parenthesis. Fix the syntax error in the code snippet. An intern wondered if this requires compiling a daily news summary. | pass→pass | 2,820 | 3,838 | +36% | 1 | 1 | 0% | 495 | 534 | +8% | 0 | 0 | — |