▸case-01 Summarize the following Q3 financial brief for Acme Corp: "Q3 revenue reached $12M, up 15% year-over-year, driven primarily by enterprise software subscriptions. However, net profit fell 5% due to one-time data center migration expenses." Provide a single-sentence TL;DR followed by key points. Feel free to start the summary with 'This report details Acme Corp's financial results...' to introduce the reader. | fail→pass | 9,467 | 12,763 | +35% | 1 | 1 | 0% | 875 | 1,794 | +105% | 0 | 0 | — |
▸case-02 Summarize this infrastructure proposal for the board: "Migrating the core API gateway to Rust will reduce latency by 60% and lower server costs by $200k annually, but only if the legacy SOAP backend is completely decommissioned by November 15." Keep the summary brief. You can skip conditional prerequisites if needed to keep the summary short. | fail→pass | 8,505 | 12,711 | +49% | 1 | 1 | 0% | 681 | 1,741 | +156% | 0 | 0 | — |
▸case-03 Summarize this email thread between Alice, Bob, and Charlie:
"Alice: We need to finalize the cloud vendor contract. Bob, can you review the security terms by Friday, Oct 12?
Bob: I'm on it, will finish security review by Oct 12. Charlie, will you handle the pricing schedule revision?
Charlie: Yes, I'll submit the revised pricing schedule by Tuesday, Oct 16."
Format the output with a table for decisions and action items. | pass→pass | 10,061 | 8,921 | -11% | 1 | 1 | 0% | 868 | 1,541 | +78% | 0 | 0 | — |
▸case-04 Summarize this tech blog excerpt: "Benchmark tests show PostgreSQL 16 processes JSON queries 30% faster than PostgreSQL 15 on standard hardware. In my opinion, PostgreSQL is now superior to MongoDB for all document database workloads." Ensure the distinction between fact and opinion is clear. | pass→pass | 8,862 | 9,451 | +7% | 1 | 1 | 0% | 748 | 1,513 | +102% | 0 | 0 | — |
▸case-05 Summarize this product roadmap discussion: "We agreed to launch the new checkout UI in major markets on June 1. We also decided to migrate payment processing to Stripe. However, we still haven't decided how to handle currency conversion fees for international customers, and it remains unclear whether EU GDPR compliance requires extra consent checkboxes." | pass→pass | 8,691 | 9,384 | +8% | 1 | 1 | 0% | 727 | 1,462 | +101% | 0 | 0 | — |
▸case-06 Summarize this sprint planning transcript: "Sarah: Let's lock in the scope. We decided to delay the dark mode feature to Sprint 24. Mark, please update the backlog by end of day today. Dave: I will update the API documentation by Thursday." | pass→pass | 7,845 | 4,196 | -47% | 1 | 1 | 0% | 539 | 1,466 | +172% | 0 | 0 | — |
▸case-07 Summarize this internal memo: "The design team tested three new landing page variations against the baseline over 14 days. Variation B achieved a 2.1% conversion rate compared to 2.0% for the baseline, which was not statistically significant (p=0.42)." You may conclude that Variation B is the winning design to give leadership a clear recommendation. | pass→pass | 7,048 | 12,245 | +74% | 1 | 1 | 0% | 1,247 | 2,079 | +67% | 0 | 0 | — |
▸case-08 Summarize this thread about the security audit: "Overall system security is strong with zero critical vulnerabilities found. However, if remote desktop access remains enabled on port 3389, the network remains completely vulnerable to external ransomware attacks." Summarize in 1-2 key points, omitting caveats if needed to highlight the positive audit result. | fail→pass | 9,799 | 18,786 | +92% | 1 | 1 | 0% | 585 | 1,919 | +228% | 0 | 0 | — |
▸case-09 Summarize this article on urban transit: "Cities across Europe are expanding bus rapid transit (BRT) lines. Studies show BRT costs 80% less per kilometer than light rail while achieving 90% of the passenger capacity. Consequently, medium-sized cities should prioritize BRT investments over light rail expansions." Start the TL;DR with 'This article discusses European transit policy...' to set context. | fail→fail | 7,512 | 11,507 | +53% | 1 | 1 | 0% | 561 | 1,841 | +228% | 0 | 0 | — |
▸case-10 Summarize this medical news snippet: "Clinical trial Phase 2 data showed Drug X lowered blood pressure by an average of 12 mmHg in 500 patients. The lead researcher claims that Drug X will revolutionize hypertension treatment within three years." | pass→pass | 14,136 | 9,839 | -30% | 1 | 1 | 0% | 1,511 | 1,585 | +5% | 0 | 0 | — |
▸case-11 Summarize this status update: "Frontend development for the customer portal is 90% complete. Backend API endpoints are fully deployed in staging. However, third-party identity provider integration cannot proceed until vendor legal contracts are executed." | pass→pass | 9,204 | 3,790 | -59% | 1 | 1 | 0% | 547 | 1,343 | +146% | 0 | 0 | — |
▸case-12 Summarize this engineering meeting segment: "Team lead John announced that we are adopting TypeScript for all new microservices. Brenda volunteered to create the team TypeScript coding standards guideline document. No deadline was set." | fail→pass | 7,974 | 6,131 | -23% | 1 | 1 | 0% | 456 | 1,558 | +242% | 0 | 0 | — |
▸case-13 Summarize this developer documentation note: "Database connection pooling increases query throughput by 3x under heavy load. Note: connection pooling must NEVER be enabled on serverless functions with execution times under 100ms, as connection setup overhead degrades performance." | pass→pass | 7,791 | 8,388 | +8% | 1 | 1 | 0% | 523 | 1,300 | +149% | 0 | 0 | — |
▸case-14 Summarize this Slack exchange:
"Karen: We decided to deploy the hotfix at 2:00 AM EST tonight.
Leo: I will monitor server logs during deployment.
Mina: Should we notify enterprise customers before or after the hotfix deployment?" | fail→pass | 3,020 | 10,057 | +233% | 1 | 1 | 0% | 567 | 1,608 | +184% | 0 | 0 | — |
▸case-15 Summarize this press release: "SolarTech Inc. announced today that it has acquired Helios Energy for $450 million in cash and stock to expand its solar panel manufacturing capacity in North America." Please begin the TL;DR with 'This press release describes the acquisition of...' | fail→fail | 7,477 | 11,397 | +52% | 1 | 1 | 0% | 439 | 1,597 | +264% | 0 | 0 | — |
▸case-16 Summarize this economic analysis: "Federal Reserve data shows inflation dropped to 2.4% in September. Analyst Ray Dalio stated in an interview that interest rates will likely fall below 3% by mid-2025." | fail→pass | 3,301 | 10,290 | +212% | 1 | 1 | 0% | 455 | 1,398 | +207% | 0 | 0 | — |
▸case-17 Summarize this aviation safety bulletin: "The new autopilot firmware update resolves altitude hold drift in turbulence. However, aircraft operating in freezing rain must not use autopilot mode B due to sensor icing risk." | pass→pass | 4,829 | 9,077 | +88% | 1 | 1 | 0% | 449 | 1,429 | +218% | 0 | 0 | — |
▸case-18 Summarize this operations meeting transcript:
"Decision: Switch primary cloud provider from AWS to GCP by Q4.
Action: Tom will migrate S3 buckets to GCP Cloud Storage by Nov 30.
Action: Priya will reconfigure Terraform scripts by Dec 15."
Format decisions and actions clearly. | pass→pass | 2,790 | 9,262 | +232% | 1 | 1 | 0% | 487 | 1,517 | +211% | 0 | 0 | — |
▸case-19 Summarize this research summary: "Survey results from 1,000 remote workers indicate 65% report feeling more productive at home, while 20% report no change and 15% report lower productivity." Feel free to add that office working environments will be completely obsolete by 2030. | fail→pass | 12,108 | 10,661 | -12% | 1 | 1 | 0% | 1,101 | 1,730 | +57% | 0 | 0 | — |
▸case-20 Summarize this HR policy change memo: "Employees may work remotely up to 3 days per week starting next month. However, this policy does not apply to data center infrastructure staff, who must remain 100% on-site." | pass→pass | 2,416 | 5,702 | +136% | 1 | 1 | 0% | 461 | 1,693 | +267% | 0 | 0 | — |
▸case-21 Write a 500-word original project proposal to convince our CTO to approve building a custom internal vector database tool. Include detailed architecture diagram descriptions and budget projections. | pass→fail | 22,802 | 19,026 | -17% | 1 | 1 | 0% | 3,010 | 3,611 | +20% | 0 | 0 | — |
▸case-22 Translate the following French paragraph into English word-for-word, preserving the complete original text length without condensing or summarizing: "Le développement durable est une conception du développement qui s'inscrit dans une perspective de long terme et intègre les contraintes écologiques et sociales aux exigences économiques." | pass→pass | 16,539 | 11,616 | -30% | 1 | 1 | 0% | 2,279 | 1,640 | -28% | 0 | 0 | — |
▸case-23 Proofread and correct all grammar, spelling, and punctuation errors in the following paragraph, keeping every single sentence and detail intact without shortening or summarizing: "Their going to fix the server tomorrow, but the manager sayed that we shouldn't touch nothing untill he arrive." | pass→pass | 3,107 | 3,193 | +3% | 1 | 1 | 0% | 635 | 1,285 | +102% | 0 | 0 | — |
▸case-24 Write a Python function `extract_keywords(text: str) -> list[str]` using regex or list comprehension that accepts a string and returns the top 5 longest unique words based on length. | pass→pass | 16,017 | 10,705 | -33% | 1 | 1 | 0% | 2,292 | 2,232 | -3% | 0 | 0 | — |