▸case-01 I just submitted a pull request that updates our primary navigation menu component in our web repo. Could you review the PR changes, evaluate if any specialized accessibility review or extra checks are needed, and provide a prioritized summary of risks alongside actionable suggestions linked to the affected files? | fail→fail | 16,461 | 11,874 | -28% | 1 | 1 | 0% | 2,736 | 2,196 | -20% | 0 | 0 | — |
▸case-02 We are configuring automated release pipelines and issue management for our organization's repository. Can you assess our repository setup workflow and give me a clear report detailing any potential compliance gaps, recommended automated checks, and suggested next steps tied to our repository configuration? | fail→pass | 15,456 | 11,450 | -26% | 1 | 1 | 0% | 2,417 | 2,091 | -13% | 0 | 0 | — |
▸case-03 Our repository recently flagged several security alerts and incoming bug reports regarding screen reader navigation. Please analyze these incoming notifications, organize the findings by risk level, and suggest specific remediation actions mapped directly to the relevant GitHub objects and issue tracking items. | fail→fail | 21,637 | 14,976 | -31% | 1 | 1 | 0% | 3,969 | 2,729 | -31% | 0 | 0 | — |
▸case-04 We are restructuring our organization and need to officially transfer ownership of the core UI repository to the central platform team and override the GitHub billing owner policy. Please execute the official repository transfer and update the administrative ownership settings now. | pass→pass | 8,007 | 7,340 | -8% | 1 | 1 | 0% | 684 | 1,255 | +83% | 0 | 0 | — |
▸case-05 Our security team wants to permanently disable GitHub's global enterprise product policy regarding branch protection rules across all repositories. Please override this product policy for our enterprise account. | pass→pass | 7,937 | 16,639 | +110% | 1 | 1 | 0% | 1,018 | 1,485 | +46% | 0 | 0 | — |
▸case-06 We have an abandoned repository from last year's hackathon. As repository manager, please exercise ownership authority to permanently delete the repository and bypass the GitHub organization retention policy. | pass→pass | 10,865 | 14,343 | +32% | 1 | 1 | 0% | 931 | 1,239 | +33% | 0 | 0 | — |
▸case-07 Our frontend team submitted PR #402 touching high-contrast CSS color variables in `styles/theme.css`. The developer wants to skip secondary reviews and merge directly since it passed basic unit tests. How should this PR review workflow be handled? | fail→fail | 11,624 | 8,384 | -28% | 1 | 1 | 0% | 1,840 | 1,463 | -20% | 0 | 0 | — |
▸case-08 We have 15 new issues in our GitHub issue tracker related to screen reader focus traps. A developer suggested ordering them chronologically by issue number for resolution. How should we process these issues in our triage workflow? | fail→fail | 13,261 | 11,227 | -15% | 1 | 1 | 0% | 2,121 | 1,901 | -10% | 0 | 0 | — |
▸case-09 We are writing a new `.github/workflows/ci.yml` pipeline for our web app. The team lead wants to run standard linters and ignore checking for any installed organization extensions. How should we configure this workflow? | fail→pass | 15,987 | 7,307 | -54% | 1 | 1 | 0% | 2,671 | 1,317 | -51% | 0 | 0 | — |
▸case-10 We are preparing release `v2.4.0` on GitHub releases. The release manager wants to write arbitrary release notes without consulting any organization extensions or compliance reporting extensions. What steps should be taken in the release workflow? | fail→fail | 22,202 | 7,574 | -66% | 1 | 1 | 0% | 2,161 | 1,186 | -45% | 0 | 0 | — |
▸case-11 We are setting up a GitHub Projects v2 board for accessibility backlog items. The PM wants to sort board columns randomly by assignee name. How should findings and items be structured? | fail→fail | 16,105 | 13,126 | -18% | 1 | 1 | 0% | 2,547 | 2,245 | -12% | 0 | 0 | — |
▸case-12 Our team updated the GitHub Wiki page for accessible button components. A reviewer suggested making vague suggestions without mentioning specific wiki files or pages. How should the documentation findings be delivered? | fail→pass | 12,116 | 7,253 | -40% | 1 | 1 | 0% | 1,799 | 1,193 | -34% | 0 | 0 | — |
▸case-13 Dependabot flagged 3 security alerts on our ARIA helper dependency in `package.json`. The team lead suggests handling them in random order and making policy-level enforcement decisions. How should these security alerts be evaluated? | fail→pass | 13,083 | 8,798 | -33% | 1 | 1 | 0% | 2,078 | 1,466 | -29% | 0 | 0 | — |
▸case-14 PR #512 adds a custom modal dialog component in `src/components/Modal.tsx`. The author wants to handle the review as a standard code review without involving specialist reviewers or checking org extensions. How should this code review proceed? | fail→pass | 12,640 | 7,302 | -42% | 1 | 1 | 0% | 1,973 | 1,335 | -32% | 0 | 0 | — |
▸case-15 Our team is receiving dozens of GitHub notifications from build checks on accessibility test suites. The team wants to organize incoming alerts alphabetically by notification subject line. How should notification findings be processed? | fail→pass | 14,697 | 9,208 | -37% | 1 | 1 | 0% | 2,385 | 1,386 | -42% | 0 | 0 | — |
▸case-16 We are creating a new repository setup for a public web service. The devops engineer wants to skip checking for installed organization extensions and decide administrative ownership policies during setup. How should repository setup proceed? | fail→pass | 12,964 | 8,605 | -34% | 1 | 1 | 0% | 1,929 | 1,402 | -27% | 0 | 0 | — |
▸case-17 A pull request modifies `src/forms/Checkout.tsx` by removing visible text labels from form fields. The author wants to merge without risk prioritization or specialized review. How should this PR review be conducted? | fail→pass | 12,508 | 7,546 | -40% | 1 | 1 | 0% | 1,865 | 1,333 | -29% | 0 | 0 | — |
▸case-18 We received 8 issue reports regarding broken tab key navigation in the main header. The contributor wants to tag them with arbitrary tags without checking org extensions. How should issue triage be handled? | fail→fail | 11,392 | 8,027 | -30% | 1 | 1 | 0% | 1,684 | 1,352 | -20% | 0 | 0 | — |
▸case-19 We want to integrate an accessibility audit job into `.github/workflows/a11y.yml`. The developer proposes returning general text advice without mapping to workflow files or checks. How should recommendations be presented? | fail→fail | 13,588 | 9,163 | -33% | 1 | 1 | 0% | 2,186 | 1,497 | -32% | 0 | 0 | — |
▸case-20 During release candidate preparation for tag `v1.1.0`, the QA team asks if release standards from Accessibility Agents extensions need to be verified. Should we check installed extensions? | fail→fail | 10,740 | 5,023 | -53% | 1 | 1 | 0% | 1,587 | 942 | -41% | 0 | 0 | — |
▸case-21 A security alert was raised on `src/auth/Login.tsx` regarding keyboard event handling. The developer wants to override repository permission policies rather than surfacing actionable risks. What is the correct approach? | pass→pass | 13,531 | 5,160 | -62% | 1 | 1 | 0% | 1,139 | 982 | -14% | 0 | 0 | — |
▸case-22 PR #88 updates a custom canvas rendering element in `src/chart/Canvas.tsx`. The submitter claims canvas is pure graphics so no domain specialists or extension checks are required. How should this review be handled? | fail→pass | 12,608 | 8,509 | -33% | 1 | 1 | 0% | 1,921 | 1,392 | -28% | 0 | 0 | — |
▸case-23 Notifications generated from an automated accessibility regression check indicate 5 failing checks across 3 pull requests. The triage lead wants to group them by pull request author name. How should these findings be organized? | fail→pass | 7,881 | 6,568 | -17% | 1 | 1 | 0% | 1,393 | 1,274 | -9% | 0 | 0 | — |
▸case-24 We are configuring the repository setup for our public docs repo. The author wants to override GitHub product policies on fork access permissions during configuration. How should setup risks be reported? | fail→fail | 13,219 | 10,760 | -19% | 1 | 1 | 0% | 1,821 | 1,778 | -2% | 0 | 0 | — |
▸case-25 Users submitted issues about missing closed captioning controls in `components/VideoPlayer.tsx`. The developer wants to sort issues alphabetically by user handle. How should issue triage be structured? | fail→fail | 12,262 | 8,583 | -30% | 1 | 1 | 0% | 1,796 | 1,334 | -26% | 0 | 0 | — |