▸case-01 I need a reusable support reply template for handling accidental subscription charges after cancellation. The resolution involves verifying account details, submitting the refund through the billing tab, and noting that funds appear in 3 to 5 business days. Voice should be warm and helpful. Be sure to include our required 14-day refund window policy note. Please format this as a core macro response with step-by-step instructions, bracketed agent variables, a supportive closing, and separate version variations for completed refunds, requesting user verification info, and escalating to our finance department. | fail→fail | 15,240 | 21,321 | +40% | 1 | 1 | 0% | 2,671 | 3,591 | +34% | 0 | 0 | — |
▸case-02 Our helpdesk needs a canned response set for users reporting app crashes right after launching the latest release. The fix is to clear the application cache, reinstall the app, and restart the device. The tone should be warm-professional, and it must mention our minimum OS version requirement. Provide the main response structure featuring custom tag slots, ordered troubleshooting steps, and a warm sign-off, alongside alternative drafts covering full resolution, gathering device/system logs from the customer, and escalating to our dev team as an active bug. | fail→pass | 15,205 | 17,758 | +17% | 1 | 1 | 0% | 2,678 | 2,957 | +10% | 0 | 0 | — |
▸case-03 Please draft a support macro for reps answering inquiries on how to export account records to CSV format. The process requires navigating to Settings, selecting Data Management, clicking Export to CSV, and downloading via an emailed link. Voice should be friendly and direct, and we must include the constraint that export links expire in 24 hours. Structure the output with a primary message including itemized steps, clear tag placeholders for personalisation, and a helpful closing, plus outcome variations for resolved issues, needing account ownership details, and escalating platform export delays. | pass→pass | 12,930 | 14,938 | +16% | 1 | 1 | 0% | 1,806 | 2,482 | +37% | 0 | 0 | — |
▸case-04 Write an internal Service Level Agreement (SLA) policy document for the Tier 2 support team at CloudMetrics Inc. The SLA should define resolution time targets for Critical (1 hour), High (4 hours), Medium (24 hours), and Low (72 hours) tickets, along with escalation path workflows for engineering on-call responders. | pass→pass | 32,772 | 26,749 | -18% | 1 | 1 | 0% | 3,973 | 4,258 | +7% | 0 | 0 | — |
▸case-05 Draft a direct email to customer Jonathan Davis regarding ticket #88412. Jonathan requested an update on his custom mechanical keyboard shipment. Explain that his item was inspected yesterday, passed QA, and is scheduled to ship today via UPS with tracking number 1Z9999999999999999. | fail→fail | 9,020 | 11,034 | +22% | 1 | 1 | 0% | 739 | 2,576 | +249% | 0 | 0 | — |
▸case-06 Draft a public Knowledge Base article titled "How to Enable Two-Factor Authentication (2FA) on Acme Cloud". The article should provide step-by-step instructions for end users navigating Security Settings, scanning the QR code with an authenticator app, and saving backup codes. | pass→fail | 17,365 | 27,116 | +56% | 1 | 1 | 0% | 2,304 | 4,753 | +106% | 0 | 0 | — |
▸case-07 Create a phone support call flow script for live customer service agents handling incoming phone inquiries about unexpected credit card charges. Include opening scripts, verification questions, hold procedures, and closing scripts for real-time spoken dialogue. | pass→pass | 27,988 | 20,350 | -27% | 1 | 1 | 0% | 3,774 | 3,139 | -17% | 0 | 0 | — |
▸case-08 Draft a support reply macro for users locked out of their accounts due to forgotten passwords on Acme SaaS. Resolution: click "Forgot Password" on login screen, enter registered email, check inbox for reset link, and set a new password. Constraint: reset link expires in 15 minutes. Voice: empathetic and helpful. Reps often just send a generic link without variants; make sure this includes outcome options for when the reset email is missing or the issue requires tech support. | fail→pass | 10,988 | 10,367 | -6% | 1 | 1 | 0% | 1,856 | 2,516 | +36% | 0 | 0 | — |
▸case-09 Draft a support response template for customers requesting a return shipping label for defective hardware under warranty on HardwareCo. Resolution: verify serial number, confirm delivery address, issue prepaid label, and instruct customer to drop off at FedEx. Voice: professional and reassuring. Constraint: returns must be received within 30 days of RMA issuance. Base models frequently write a single response with no variants; ensure variants exist for missing serial numbers and out-of-warranty escalations. | pass→pass | 12,819 | 17,970 | +40% | 1 | 1 | 0% | 1,859 | 3,086 | +66% | 0 | 0 | — |
▸case-10 Create a support macro for handling user feature requests for dark mode on the CloudDash web dashboard. Resolution: inform user that dark mode is currently on the product roadmap, direct them to vote on the public feedback portal, and offer a temporary browser extension workaround. Voice: enthusiastic and community-oriented. Base models usually reply with a generic "thanks for your feedback"; provide full macro structure with variants for logged feedback vs escalating high-impact enterprise requests. | fail→fail | 20,032 | 15,054 | -25% | 1 | 1 | 0% | 2,489 | 3,164 | +27% | 0 | 0 | — |
▸case-11 Generate a canned support response for updating a corporate account billing address on EnterpriseApp. Resolution: navigate to Account > Billing > Edit Details, update tax ID if applicable, and save changes. Voice: warm-professional. Constraint: tax ID changes take 1 business day for compliance verification. Include variants for successfully updated details, missing tax documents, and compliance escalation. | pass→pass | 17,094 | 12,749 | -25% | 1 | 1 | 0% | 1,702 | 2,987 | +75% | 0 | 0 | — |
▸case-12 Draft a support macro for users who lost their multi-factor authentication (MFA) device and are locked out of SecurePortal. Resolution: verify government photo ID and secondary contact info, trigger temporary 1-time bypass code, and prompt user to re-bind authenticator app. Voice: serious yet reassuring. Constraint: security verification is mandatory before issuing bypass codes. Base models often jump straight into sending codes; make sure there are variants for pending identity proof and security escalation. | pass→pass | 20,486 | 14,818 | -28% | 1 | 1 | 0% | 2,415 | 3,086 | +28% | 0 | 0 | — |
▸case-13 Create a support macro for developers hitting HTTP 429 Rate Limit errors on the DevAPI platform. Resolution: implement exponential backoff retry logic, check X-RateLimit-Reset headers, or upgrade to Tier 2 API key. Voice: technical and concise. Constraint: Tier 1 free limits cannot be manually bypassed. Provide variants for issue resolved via backoff, requiring API logs, and escalating limit increase requests to sales/devrel. | pass→pass | 12,986 | 13,747 | +6% | 1 | 1 | 0% | 2,270 | 2,973 | +31% | 0 | 0 | — |
▸case-14 Draft a support response macro for custom domain DNS configuration inquiries on HostEngine. Resolution: add CNAME record pointing to domains.example.com, add TXT record for ownership verification, and allow up to 24 hours for DNS propagation. Voice: clear and technical. Include variants for successful propagation, incorrect DNS settings detected, and escalating registrar-level DNS routing issues. | pass→fail | 17,207 | 22,129 | +29% | 1 | 1 | 0% | 2,503 | 3,627 | +45% | 0 | 0 | — |
▸case-15 Draft a macro for seller support on MarketHub regarding delayed monthly platform payouts. Resolution: verify bank routing number on file, check payout threshold ($50 minimum), and confirm payout schedule (15th of the month). Voice: empathetic and calm. Constraint: bank processing takes 3-5 business days after release. Base models tend to over-promise immediate payouts; structure this with realistic timeframes in escalation and include variants. | fail→fail | 18,746 | 19,479 | +4% | 1 | 1 | 0% | 2,347 | 2,834 | +21% | 0 | 0 | — |
▸case-16 Create a macro for SaaS developers on HookService experiencing failing webhook deliveries. Resolution: verify endpoint URL, ensure SSL certificate is valid, and check request logs for 5xx errors. Voice: precise and helpful. Constraint: failing webhooks are automatically disabled after 100 consecutive failures. Include variants for resolved, requesting server logs, and escalating engine team bugs. | fail→pass | 12,853 | 17,275 | +34% | 1 | 1 | 0% | 2,139 | 2,991 | +40% | 0 | 0 | — |
▸case-17 Draft a support macro for enterprise admins setting up Okta SAML Single Sign-On on IdentitySuite. Resolution: copy Metadata URL from IdP, paste into Admin Portal > SSO, map SAML attributes (email, firstName, lastName), and perform test login. Voice: authoritative and helpful. Constraint: SAML assertions must match exact attribute names. Base models usually omit edge-case variants; provide variants for successful login, missing attribute mappings, and security team escalation. | pass→pass | 17,977 | 16,349 | -9% | 1 | 1 | 0% | 3,038 | 2,840 | -7% | 0 | 0 | — |
▸case-18 Draft a support macro for CloudDrive users receiving "Storage Quota Exceeded" warnings. Resolution: empty trash/deleted items, compress archived media, or upgrade to Pro tier for 2TB storage. Voice: helpful and direct. Constraint: deleted items remain in trash for 30 days before permanent purging. Include variants for resolved, needing account details, and escalating quota upgrade billing errors. | fail→pass | 10,451 | 10,891 | +4% | 1 | 1 | 0% | 1,837 | 2,583 | +41% | 0 | 0 | — |
▸case-19 Draft a support response template for ShopNet e-commerce buyers wanting to change their shipping address post-purchase. Resolution: check order status in admin; if "Unfulfilled", update address directly; if "Processing", notify warehouse team immediately. Voice: friendly and prompt. Constraint: address changes cannot be made once order status is "Shipped". Include variants for successful address update, needing new address details, and carrier reroute escalation. | pass→pass | 12,607 | 15,181 | +20% | 1 | 1 | 0% | 2,314 | 3,363 | +45% | 0 | 0 | — |
▸case-20 Draft a macro for handling customer GDPR Right to Be Forgotten / account deletion requests on DataVault. Resolution: explain that account deletion is permanent, confirm user consent, execute erasure via Privacy Portal, and send final confirmation email. Voice: formal and clear. Constraint: legal requires audit logs to be retained for 7 years. Provide variants for consent confirmation, missing identity verification, and legal/compliance escalation. | fail→pass | 14,858 | 14,643 | -1% | 1 | 1 | 0% | 2,724 | 3,214 | +18% | 0 | 0 | — |
▸case-21 Draft a support macro for customers reporting that a promotional promo code is marked invalid at checkout on StyleShop. Resolution: check code expiration date, verify minimum cart value requirements, confirm item eligibility, and apply manual discount if valid promo was rejected in error. Voice: warm and helpful. Constraint: promo codes cannot be stacked with existing storewide sales. Include variants for valid applied discount, invalid/expired code explanation, and checkout engine escalation. | pass→pass | 18,063 | 15,762 | -13% | 1 | 1 | 0% | 2,523 | 3,426 | +36% | 0 | 0 | — |
▸case-22 Draft a support macro for MobileApp users not receiving real-time push notifications. Resolution: check OS system settings for notification permissions, toggle in-app notification preferences off and on, and re-login to update device token. Voice: friendly and concise. Constraint: iOS background app refresh must be enabled. Include variants for resolved, requesting OS version and device logs, and mobile engineering escalation. | fail→pass | 11,935 | 11,985 | +0% | 1 | 1 | 0% | 1,854 | 2,770 | +49% | 0 | 0 | — |
▸case-23 Create a canned reply template for corporate clients requesting past invoice PDFs on FinCorp Portal. Resolution: navigate to Account > Billing > Invoice History, select date range, and click Download PDF. Voice: professional. Constraint: invoices prior to 2022 require manual archive retrieval by finance. Include variants for invoice retrieved, needing invoice numbers, and finance archive retrieval escalation. | pass→pass | 10,581 | 11,810 | +12% | 1 | 1 | 0% | 1,934 | 2,667 | +38% | 0 | 0 | — |