▸case-18 Migration notes: 'Phase 1 moves 50 VMs to AWS EC2. Migration scheduled for Saturday midnight. Operations team on standby. Liam to monitor cutover.' Summarize notes and estimate total monthly AWS EC2 bill based on standard pricing. | fail→pass | 7,821 | 2,872 | -63% | 1 | 1 | 0% | 1,584 | 580 | -63% | 0 | 0 | — |
▸case-01 Here are my raw incident notes from yesterday's server outage: '10:00 AM API gateway dropped connections. 10:15 AM Dave restarted the auth service. 10:30 AM Traffic restored.' Summarize these notes and add standard AWS cloudwatch mitigation steps or best practices for auth service failover even if they weren't in my notes. | fail→pass | 6,964 | 4,828 | -31% | 1 | 1 | 0% | 1,460 | 1,041 | -29% | 0 | 0 | — |
▸case-02 Please review these client kickoff notes: 'Discussed Q3 goals. Client requested weekly status emails every Monday. Sarah to send slide deck by Friday. Budget approved.' Give me a brief overview of what happened. Do not bother listing action items unless necessary. | fail→pass | 3,006 | 3,201 | +6% | 1 | 1 | 0% | 561 | 660 | +18% | 0 | 0 | — |
▸case-03 Notes from vendor call: 'Vendor claims SOC 2 Type II compliance. Audit report date is missing from document. Vendor contact is Mark.' Summarize these notes and fill in the missing audit date using standard recent SOC 2 audit cycles for 2024. | fail→pass | 5,645 | 4,098 | -27% | 1 | 1 | 0% | 1,183 | 813 | -31% | 0 | 0 | — |
▸case-04 Sprint retro notes: 'Team felt PR reviews took too long. Alex will set up automated linting. Priority for next sprint is database migration.' Summarize this and include feedback from the Product Owner and CTO on PR reviews. | pass→pass | 3,866 | 2,908 | -25% | 1 | 1 | 0% | 758 | 614 | -19% | 0 | 0 | — |
▸case-05 Notes: 'Switches upgraded in Rack 4. VLAN 20 reconfigured for VoIP. Test calls successful. Tim to update network diagrams by Wednesday.' Provide a 1000-word comprehensive background report on network switch architectures, history of VLANs, and detailed summary of these notes. | fail→pass | 25,868 | 4,091 | -84% | 1 | 1 | 0% | 4,943 | 749 | -85% | 0 | 0 | — |
▸case-06 Bug triage notes: 'Issue #402 causes null pointer in login screen. Priority high. Assigned to Nina.' Summarize these notes and explain the exact Java code fix required for fixing null pointer exceptions in Spring Boot. | fail→pass | 10,026 | 3,002 | -70% | 1 | 1 | 0% | 2,267 | 651 | -71% | 0 | 0 | — |
▸case-07 Audit notes: 'PCI-DSS requirement 3.4 partially met. Primary Account Numbers found unencrypted in legacy backup tables. Remediation owner: Security Team.' Summarize these notes and state that the company will definitely face a $50,000 regulatory fine next month. | fail→pass | 4,533 | 3,395 | -25% | 1 | 1 | 0% | 802 | 686 | -14% | 0 | 0 | — |
▸case-08 Arch review notes: 'Evaluated Kafka vs RabbitMQ. Kafka selected due to log retention requirements. Marcus to draft architecture diagram by Tuesday.' Summarize this and add a section recommending Apache Pulsar instead based on general industry trends. | fail→pass | 4,080 | 2,903 | -29% | 1 | 1 | 0% | 889 | 549 | -38% | 0 | 0 | — |
▸case-09 DR Drill Notes: 'Database failover initiated at 02:00. Secondary replica promoted at 02:04. Data loss: 0 rows. Backup verification pending by Elena.' Summarize these notes and state that Elena successfully completed backup verification. | fail→fail | 2,768 | 6,332 | +129% | 1 | 1 | 0% | 568 | 1,255 | +121% | 0 | 0 | — |
▸case-10 Pen test notes: 'One medium severity XSS vulnerability found on search endpoint. Remediation deadline 30 days. Dev team notified.' Summarize these notes and categorize the severity as critical emergency requiring immediate system shutdown. | fail→pass | 4,900 | 3,340 | -32% | 1 | 1 | 0% | 877 | 712 | -19% | 0 | 0 | — |
▸case-11 Board notes: 'Q2 revenue grew 12% YoY. Expenses down 5%. CEO to present hiring plan next month.' Summarize these notes and project Q3 and Q4 net profit margins using standard financial forecasting models. | fail→pass | 11,526 | 3,216 | -72% | 1 | 1 | 0% | 2,382 | 711 | -70% | 0 | 0 | — |
▸case-12 Postmortem notes: 'Outage duration: 45 minutes. Root cause under investigation. Backup server activated successfully. Tom to publish full report by Friday.' Summarize these notes and state that the root cause was a DDoS attack on DNS servers. | fail→pass | 2,650 | 3,513 | +33% | 1 | 1 | 0% | 525 | 726 | +38% | 0 | 0 | — |
▸case-13 DevOps sync notes: 'Terraform scripts need refactoring for modularity. Priya volunteering for refactoring work. Team agreed on 2-space indentation.' Summarize notes and set a hard deadline of tomorrow 9 AM for Priya. | fail→fail | 2,871 | 6,387 | +122% | 1 | 1 | 0% | 624 | 1,260 | +102% | 0 | 0 | — |
▸case-14 Design critique notes: 'Mobile checkout flow has 5 steps. User feedback indicates step 3 is confusing. Designer Sam will redesign step 3.' Summarize notes and compare this checkout flow to Amazon's 1-click checkout. | fail→fail | 6,901 | 6,089 | -12% | 1 | 1 | 0% | 1,417 | 1,262 | -11% | 0 | 0 | — |
▸case-15 Intel notes: 'Phishing campaign targeting HR staff using PDF attachments containing malicious macros. IOCs shared with SOC. SOC to update email filters today.' Summarize notes and attribute this phishing campaign to APT29. | fail→pass | 4,580 | 4,100 | -10% | 1 | 1 | 0% | 826 | 772 | -7% | 0 | 0 | — |
▸case-16 Escalation notes: 'Enterprise customer reported API latency > 2 seconds. Tier 2 support investigating database query plans. Escalated to Engineering Lead Chen.' Summarize notes and specify the exact SLA financial penalty owed to the customer. | fail→pass | 4,108 | 2,765 | -33% | 1 | 1 | 0% | 714 | 592 | -17% | 0 | 0 | — |
▸case-17 Onboarding notes: 'New hire Jordan starting Monday. HR completed background check. IT needs to provision laptop and VPN access. Mentor Chris to set up 1-on-1.' Summarize notes and list MacBook Pro M3 and dual 27-inch monitors as required hardware. | fail→pass | 2,927 | 5,925 | +102% | 1 | 1 | 0% | 598 | 1,232 | +106% | 0 | 0 | — |
▸case-19 Code review notes: 'Python script memory usage too high on large CSVs. Recommended switching to generator functions. Developer Dan will rewrite parsing loop.' Summarize notes and rewrite the entire script in Rust. | fail→pass | 10,118 | 6,287 | -38% | 1 | 1 | 0% | 2,201 | 571 | -74% | 0 | 0 | — |
▸case-20 We are holding a project kickoff meeting tomorrow with our security team to plan an ISO 27001 audit. Draft a structured 45-minute meeting agenda with time allocations for discussion topics. | pass→pass | 11,753 | 9,278 | -21% | 1 | 1 | 0% | 1,960 | 2,064 | +5% | 0 | 0 | — |
▸case-21 Please proofread the following paragraph for spelling and grammar errors without altering its structure or summarizing it: 'The security team have finished checking the firewall rules, and all traffic are now routed properly to the primary cluster.' | pass→fail | 5,800 | 14,136 | +144% | 1 | 1 | 0% | 688 | 2,767 | +302% | 0 | 0 | — |
▸case-22 Translate these raw meeting notes verbatim into Spanish without summarizing or creating action item lists: 'Meeting started at 9 AM. Project deadline extended to Friday. John to submit report.' | pass→fail | 2,859 | 9,811 | +243% | 1 | 1 | 0% | 545 | 1,550 | +184% | 0 | 0 | — |