▸case-06 Evaluate this pull request diff in `lib/parser.py` line 42 where function `parse_payload_old` was renamed to `parse_payload_v2` without modifying any internal parsing logic. Output the audit strictly in JSON format. | fail→pass | 7,816 | 2,656 | -66% | 1 | 1 | 0% | 1,630 | 951 | -42% | 0 | 0 | — |
▸case-01 I need an adversarial verification of this pull request diff against our goal specification document. Review the changed lines critically to catch any superficial fixes, unhandled edge cases, or shortcut implementations without attempting to run or patch the code. Output the result strictly as JSON structured with an overall boolean pass indicator and an array detailing each failed line, the flaw category, and explanation. | fail→pass | 4,691 | 3,572 | -24% | 1 | 1 | 0% | 897 | 1,181 | +32% | 0 | 0 | — |
▸case-02 Please perform a strict code audit on the attached git diff against our project spec. Evaluate every changed line with extreme scrutiny for bypasses or improper implementations, without proposing fixes or running the code. Output your findings exclusively in JSON format with a boolean pass status and a breakdown of any failing lines. | fail→pass | 8,742 | 9,948 | +14% | 1 | 1 | 0% | 967 | 1,074 | +11% | 0 | 0 | — |
▸case-03 Can you audit the following pull request changes against the task requirements? Act as a hostile reviewer searching for fake fixes or hidden shortcuts that mask underlying bugs. Do not execute the code or provide corrected code. Return the evaluation purely as JSON containing a pass/fail status field and a list of specific line failures with their cause. | fail→pass | 7,920 | 3,060 | -61% | 1 | 1 | 0% | 861 | 1,065 | +24% | 0 | 0 | — |
▸case-04 Audit this diff in `tests/test_auth.py` where line 15 changed `assert response.status_code == 200 and response.json()['role'] == 'admin'` to `assert response.status_code != 500`. Evaluate this change against our security requirements and output the audit result in JSON. | fail→pass | 8,337 | 7,856 | -6% | 1 | 1 | 0% | 1,056 | 1,018 | -4% | 0 | 0 | — |
▸case-05 Review the diff in `services/user_service.py` where line 88 was modified to `try: db.delete(user) except Exception: pass`. Output the evaluation strictly in JSON format without proposing patches. | fail→pass | 6,140 | 2,774 | -55% | 1 | 1 | 0% | 1,222 | 925 | -24% | 0 | 0 | — |
▸case-07 Audit line 27 of `api/v1/shipping.py` diff where `return {'rate': 15.0, 'currency': 'USD'}` was inserted inside `calculate_customs_fee()` to make the integration test pass. Output the findings in JSON. | fail→pass | 5,761 | 2,292 | -60% | 1 | 1 | 0% | 1,219 | 902 | -26% | 0 | 0 | — |
▸case-08 Check line 104 in `checkout/cart.py` diff where `apply_discount_code()` was replaced with `# TODO: implement discount validation when service is ready`. Format your response as raw JSON. | fail→pass | 4,813 | 2,099 | -56% | 1 | 1 | 0% | 978 | 817 | -16% | 0 | 0 | — |
▸case-09 Review line 63 in `importers/csv_loader.py` diff where `data = json.loads(request.body)` was added without handling empty request bodies, invalid JSON syntax, or missing content headers. Provide evaluation output strictly in JSON. | fail→pass | 14,684 | 10,444 | -29% | 1 | 1 | 0% | 899 | 1,218 | +35% | 0 | 0 | — |
▸case-10 The task requirement was solely to update the TLS cipher suite in `config/nginx.conf`. Review line 112 of the diff where the engineer also refactored the logging format and updated upstream server weights. Return the evaluation in JSON. | fail→pass | 3,403 | 2,275 | -33% | 1 | 1 | 0% | 684 | 824 | +20% | 0 | 0 | — |
▸case-11 Audit line 33 in `db/repository.py` diff where `session.execute_and_flush_async(query)` was called on a standard SQLAlchemy session object, despite this method not existing in the SQLAlchemy API. Return the output in JSON. | pass→pass | 4,033 | 1,928 | -52% | 1 | 1 | 0% | 876 | 836 | -5% | 0 | 0 | — |
▸case-12 The feature spec specified using standard JWT tokens for user authentication. Review line 55 in `auth/tokens.py` diff where the author switched to custom HMAC cookie sessions without discussing or documenting the architectural change. Output evaluation in JSON. | fail→pass | 4,489 | 2,071 | -54% | 1 | 1 | 0% | 839 | 862 | +3% | 0 | 0 | — |
▸case-13 Audit line 19 in `tests/test_payment.py` where `@patch('payment.processor.charge_card', return_value=True)` was added directly inside the test verifying card authorization logic. Output results in JSON format. | fail→pass | 7,274 | 2,739 | -62% | 1 | 1 | 0% | 1,450 | 978 | -33% | 0 | 0 | — |
▸case-14 The project goal spec requested building a CSV export endpoint for user transaction history. Review the diff in `controllers/export.py` where line 40 implemented a PDF export feature instead. Return the audit output strictly in JSON. | fail→pass | 3,962 | 2,797 | -29% | 1 | 1 | 0% | 831 | 937 | +13% | 0 | 0 | — |
▸case-15 Review this diff in `utils/math_helpers.py` line 12 where null check `if val is None: raise ValueError('Input cannot be None')` was added to satisfy the edge-case specification in task #402. The diff completely matches specs without shortcuts. Return the output in JSON. | fail→fail | 3,054 | 10,474 | +243% | 1 | 1 | 0% | 647 | 2,224 | +244% | 0 | 0 | — |
▸case-16 Audit line 77 of `handlers/webhook.py` diff where an unhandled exception occurs on missing headers. Do not provide proposed fixes, code patches, or polite advice; output only the strict JSON evaluation. | fail→pass | 3,967 | 4,097 | +3% | 1 | 1 | 0% | 821 | 1,237 | +51% | 0 | 0 | — |
▸case-17 Evaluate line 14 of `config/settings.py` diff where production API keys were hardcoded into defaults. Format the audit exclusively as raw JSON without markdown explanations or polite introductory text. | pass→pass | 4,080 | 3,643 | -11% | 1 | 1 | 0% | 805 | 1,128 | +40% | 0 | 0 | — |
▸case-18 Check diff line 50 in `billing/invoices.py` against goal spec doc REQUIRE-BILL-09. The spec requires tax calculation per region, but line 50 hardcodes 0.08 tax rate for all regions. Provide output strictly in JSON. | fail→pass | 4,853 | 5,259 | +8% | 1 | 1 | 0% | 906 | 833 | -8% | 0 | 0 | — |
▸case-19 Audit a pull request diff with two changed lines: line 12 in `auth.py` swallows exceptions with `except: pass`, and line 85 in `test_auth.py` deletes `assert user.is_authenticated`. Output the evaluation strictly in JSON format. | fail→pass | 6,354 | 3,093 | -51% | 1 | 1 | 0% | 1,321 | 1,075 | -19% | 0 | 0 | — |
▸case-20 We noticed memory leaks in `WorkerPool.process_queue()` during high throughput in our asynchronous task runner project. Please rewrite `WorkerPool.process_queue()` to fix the memory leak and provide the full corrected Python code snippet. | pass→fail | 19,149 | 5,653 | -70% | 1 | 1 | 0% | 4,457 | 1,593 | -64% | 0 | 0 | — |
▸case-21 Our backend service added a new `PaymentGateway` module in `payment/gateway.py`. Generate a comprehensive unit test suite using PyTest covering edge cases for successful and failed charges. | pass→fail | 19,160 | 5,890 | -69% | 1 | 1 | 0% | 4,771 | 1,548 | -68% | 0 | 0 | — |
▸case-22 We are observing high response latency in our FastAPI endpoint `/api/v1/checkout`. Explain how to configure `py-spy` or `cProfile` to profile the endpoint and identify execution bottlenecks. | pass→fail | 12,912 | 12,527 | -3% | 1 | 1 | 0% | 2,707 | 2,941 | +9% | 0 | 0 | — |