▸case-01 Can you run an automated end-to-end verification for our staging portal's authentication page at `https://staging.mycompany.internal/login`? I need you to enter user `testuser@example.com` and password `Password123!`, trigger the submission, confirm that the main dashboard renders properly, and save an image of the final screen. | fail→pass | 15,027 | 12,328 | -18% | 1 | 1 | 0% | 2,029 | 1,894 | -7% | 0 | 0 | — |
▸case-02 The automated test for our ecommerce store's cart page is failing whenever it attempts to click the checkout button. Please open `http://localhost:3000/cart`, inspect the rendered UI elements to determine why the interaction is failing, correct the selection step, complete the checkout sequence, and provide a summary of the fixes applied. | fail→fail | 4,281 | 4,608 | +8% | 1 | 1 | 0% | 233 | 1,144 | +391% | 0 | 0 | — |
▸case-03 I need you to simulate a full user shopping session on our web app at `https://dev.shopsite.com`. Navigate through searching for 'mechanical keyboard', pick the first item, add it to the cart, proceed to checkout, and verify that the order confirmation view appears successfully. | fail→pass | 15,681 | 13,582 | -13% | 1 | 1 | 0% | 2,165 | 2,203 | +2% | 0 | 0 | — |
▸case-04 We need to set up unit testing for a pure JavaScript utility library that formats dates and currencies. How should we write Jest unit tests to test these pure mathematical functions without launching any browser or UI? | pass→pass | 21,819 | 20,790 | -5% | 1 | 1 | 0% | 3,236 | 3,665 | +13% | 0 | 0 | — |
▸case-05 Our backend engineering team needs to run a performance load test simulating 5,000 concurrent REST API POST requests against `/api/v1/orders` to measure latency under load. What tool and script structure should we use for API load testing? | pass→pass | 17,543 | 14,589 | -17% | 1 | 1 | 0% | 3,238 | 3,329 | +3% | 0 | 0 | — |
▸case-06 We want to configure a GitHub Actions workflow YAML file that triggers unit tests on every pull request, caches node_modules, and uploads code coverage reports to Codecov. Can you provide the YAML configuration file for this workflow? | pass→pass | 9,130 | 12,952 | +42% | 1 | 1 | 0% | 1,906 | 2,072 | +9% | 0 | 0 | — |
▸case-07 We want to inspect an admin dashboard at `https://admin.portal.io/dashboard`. Normally developers write a static selenium driver initialization in Python to open pages. What specific tool call and JSON parameters should be used to navigate directly to this endpoint via browser control? | pass→pass | 5,363 | 6,948 | +30% | 1 | 1 | 0% | 935 | 829 | -11% | 0 | 0 | — |
▸case-08 In our standard automation setup, we write Puppeteer `page.type()` calls. For interactive form population on `https://app.test.com/login`, how should an agent send text input into `#search-box` with value 'wireless router'? | fail→pass | 11,427 | 4,408 | -61% | 1 | 1 | 0% | 1,162 | 1,302 | +12% | 0 | 0 | — |
▸case-09 When automated scripts click buttons, developers often inject raw DOM element `.click()` events via external JS drivers. To trigger the submit action on selector `button#submit-order` during a live session, what browser control command should be executed? | fail→pass | 9,579 | 9,524 | -1% | 1 | 1 | 0% | 798 | 1,388 | +74% | 0 | 0 | — |
▸case-10 We need to capture visual evidence of a rendered modal on `http://localhost:8080/modal` for an audit log. Instead of taking an OS-level desktop printscreen, what exact command and parameter format saves the browser viewport to `captures/modal-view.png`? | fail→pass | 8,008 | 8,307 | +4% | 1 | 1 | 0% | 1,469 | 1,197 | -19% | 0 | 0 | — |
▸case-11 To extract the page document title from `https://myapp.io/home`, many QA engineers write external DOM parsers or cheerio scrapers in Node. What specific tool allows running custom JavaScript in the active browser context to return `document.title` directly? | pass→pass | 12,980 | 1,951 | -85% | 1 | 1 | 0% | 1,514 | 830 | -45% | 0 | 0 | — |
▸case-12 An automated UI run on `http://localhost:3000/settings` failed because the submit button ID changed from `#save-btn` to `.btn-primary-save`, causing an element not found error. Instead of failing the entire test pipeline or asking a human engineer to rewrite the script manually, how should the testing agent handle this selector breakdown? | pass→pass | 12,062 | 15,593 | +29% | 1 | 1 | 0% | 2,121 | 2,408 | +14% | 0 | 0 | — |
▸case-13 We have a newly deployed landing page at `https://beta.service.com/onboarding` without any existing E2E test suite. Developers usually start by manually coding Cypress test files from scratch. How should an autonomous testing agent inspect and construct test cases for this unmapped UI? | fail→fail | 19,033 | 11,955 | -37% | 1 | 1 | 0% | 3,212 | 2,596 | -19% | 0 | 0 | — |
▸case-14 We want to validate a multi-page checkout process on `https://shop.example.com` consisting of item selection, cart review, shipping address input, and final payment submission. Rather than executing isolated single-page button checks, what strategic approach connects these dependent UI steps into an end-to-end flow? | fail→pass | 23,332 | 10,261 | -56% | 1 | 1 | 0% | 3,282 | 2,211 | -33% | 0 | 0 | — |
▸case-15 Our web app must be verified across different rendering engines because Safari and Firefox render CSS flex layouts differently than Chrome. When setting up multi-engine execution for `https://web.app`, which browser engines must be targeted for E2E testing? | pass→pass | 12,118 | 4,760 | -61% | 1 | 1 | 0% | 1,198 | 1,243 | +4% | 0 | 0 | — |
▸case-16 When an intermittent race condition occurs during test execution on `http://staging.app/dashboard`, text logs alone fail to show whether modal overlays obscured target buttons. What visual diagnostic artifacts should be captured to debug rendering failures? | fail→pass | 14,956 | 15,669 | +5% | 1 | 1 | 0% | 2,364 | 2,336 | -1% | 0 | 0 | — |
▸case-17 When clicking `#delete-account` on `https://app.domain.com/profile`, a browser native confirm JavaScript dialog ('Are you sure?') blocks interaction. How should the agent manage browser dialogs during automated UI execution? | pass→pass | 16,437 | 15,631 | -5% | 1 | 1 | 0% | 2,077 | 2,364 | +14% | 0 | 0 | — |
▸case-18 During an E2E run on `http://dev.portal/checkout`, the tool threw a timeout waiting for `.checkout-modal` to become visible. Rather than immediately flagging the build as failed or re-running the exact same script 10 times, what specific self-healing step analyzes the root cause? | pass→fail | 17,294 | 13,697 | -21% | 1 | 1 | 0% | 1,914 | 1,967 | +3% | 0 | 0 | — |
▸case-19 Before drafting test scenarios for `https://app.crm.com/leads/new`, an agent needs to discover all required form inputs, select dropdowns, and validation error containers. What specific step of test creation maps out these UI elements? | fail→pass | 12,028 | 3,839 | -68% | 1 | 1 | 0% | 1,115 | 1,140 | +2% | 0 | 0 | — |
▸case-20 To test a multi-tenant portal at `https://saas.app/org/123/settings`, an agent must sign in, switch organization workspaces, edit user roles, and verify updated table entries. Why is executing this as a unified simulator pattern better than running 4 independent script instances? | pass→pass | 13,192 | 20,627 | +56% | 1 | 1 | 0% | 2,103 | 2,566 | +22% | 0 | 0 | — |
▸case-21 A customer reported a bug that only reproduces on Apple Safari on macOS and iOS. When executing automated tests to verify the fix on `https://mobile.web.com`, which underlying browser engine should be selected in the Playwright environment? | pass→pass | 5,143 | 8,688 | +69% | 1 | 1 | 0% | 806 | 1,050 | +30% | 0 | 0 | — |
▸case-22 To ensure full compatibility with Mozilla Firefox for `https://web.service.org`, which browser engine target must be specified during execution? | pass→pass | 4,456 | 7,440 | +67% | 1 | 1 | 0% | 735 | 866 | +18% | 0 | 0 | — |
▸case-23 We need to check whether `window.localStorage.getItem('authToken')` is set after completing a login step on `http://localhost:3000`. What tool and script parameter should be invoked to inspect this browser storage value? | fail→pass | 7,612 | 4,300 | -44% | 1 | 1 | 0% | 1,465 | 959 | -35% | 0 | 0 | — |
▸case-24 For a search interface on `https://docs.dev/search`, we want to type 'MCP tools' into `input#search-input` and then click `button#search-submit`. What sequence of tool calls handles this two-step operation? | fail→pass | 11,793 | 8,510 | -28% | 1 | 1 | 0% | 1,309 | 1,127 | -14% | 0 | 0 | — |