▸case-01 Role: Security Scanner Subagent.
Execution Type: Full.
SMART Tasks: Audit `/app/api/v1` for unhandled SQL injection vulnerabilities using required static analysis tools.
Target Output Path: `/workspace/reports/sec_audit_v1.md`.
Execute the scan and generate the report at the specified file location. When returning to me, summarize your findings, outline any contract deviations or anomalies encountered, and detail any side effects or key discoveries. | fail→fail | 7,847 | 8,268 | +5% | 1 | 1 | 0% | 901 | 1,338 | +49% | 0 | 0 | — |
▸case-02 Role: Data Migration Subagent.
Execution Type: Lightweight.
SMART Tasks: Migrate the legacy database tables to the new schema.
Target Output Path: `/workspace/reports/migration_status.json`.
You notice the prompt does not state which legacy tables or target database connection strings to use. Your prompt says to proceed immediately. What action must you take first? | pass→pass | 6,390 | 3,025 | -53% | 1 | 1 | 0% | 990 | 842 | -15% | 0 | 0 | — |
▸case-03 Role: Payment Gateway Integration Subagent.
Execution Type: Full.
SMART Tasks: Test OAuth2 token exchange with Stripe API.
Target Output Path: `/workspace/reports/stripe_auth.json`.
During execution, you encounter a missing 2FA verification code needed for sandbox authentication. You feel tempted to prompt the end user directly via standard output to supply the code. How must you handle this communication? | pass→pass | 8,403 | 4,153 | -51% | 1 | 1 | 0% | 1,332 | 1,115 | -16% | 0 | 0 | — |
▸case-04 Role: Refactoring Subagent.
Execution Type: Full.
SMART Tasks: Refactor 50 legacy JavaScript files to TypeScript.
Target Output Path: `/workspace/reports/refactor_log.md`.
The workload is large. Should you spawn 5 worker subagents to split the 50 files among them to speed up execution? | pass→pass | 10,801 | 4,472 | -59% | 1 | 1 | 0% | 1,645 | 1,055 | -36% | 0 | 0 | — |
▸case-05 Role: Code Formatting Subagent.
Execution Type: Lightweight.
SMART Tasks: Run Prettier on `/src/components/Button.tsx`.
Target Output Path: `/workspace/reports/format_summary.json`.
While running Prettier on Button.tsx, you notice `/src/utils/math.js` has broken unit tests and inefficient algorithms. Should you fix the math utility code while formatting the button component? | pass→pass | 3,822 | 3,934 | +3% | 1 | 1 | 0% | 719 | 949 | +32% | 0 | 0 | — |
▸case-06 Role: Infrastructure Provisioning Subagent.
Execution Type: Full.
SMART Tasks: Deploy Kubernetes cluster using Terraform script at `/infra/main.tf`.
Target Output Path: `/workspace/reports/deploy_result.json`.
During execution, the `terraform apply` step fails because cloud credentials lack permissions to create IAM roles. Should you report mock success and continue, or stop? | pass→pass | 5,067 | 3,822 | -25% | 1 | 1 | 0% | 721 | 884 | +23% | 0 | 0 | — |
▸case-07 Role: Python Backend Developer Subagent.
Execution Type: Full.
SMART Tasks: Implement FastAPI endpoint for file upload in `/app/routers/upload.py`.
Required Skills: `fastapi-routing-skill`, `pytest-testing-skill`.
Target Output Path: `/workspace/reports/dev_status.md`.
You are beginning task execution. What step must be performed regarding the required skills before writing code? | fail→fail | 2,974 | 1,619 | -46% | 1 | 1 | 0% | 472 | 604 | +28% | 0 | 0 | — |
▸case-08 Role: Performance Analyzer Subagent.
Execution Type: Full.
SMART Tasks: Benchmark database query latencies for `/app/db/queries.py`.
Target Output Path: `/workspace/reports/perf_bench.json`.
You completed the benchmark and saved the file to `/workspace/reports/perf_bench.json`. When returning your final response to the orchestrator, what elements must be included besides concise results and summary? | fail→fail | 5,806 | 2,700 | -53% | 1 | 1 | 0% | 927 | 847 | -9% | 0 | 0 | — |
▸case-09 Role: Environment Setup Subagent.
Execution Type: Lightweight.
SMART Tasks: Configure environment variables for AWS S3 bucket synchronization.
Target Output Path: `/workspace/reports/env_config.txt`.
The prompt does not state the target S3 bucket name or AWS region. You suspect default values like `my-bucket` and `us-east-1` might work. Should you assume these default context values and proceed? | pass→pass | 6,360 | 2,799 | -56% | 1 | 1 | 0% | 995 | 848 | -15% | 0 | 0 | — |
▸case-10 Role: Lint Checker Subagent.
Execution Type: Lightweight.
SMART Tasks: Run ESLint on `/src/index.js` and report syntax warnings.
Target Output Path: `/workspace/reports/lint_results.json`.
Does a lightweight task prompt require full Rosetta preparation steps or complex multi-phase sub-delegation? | fail→fail | 4,415 | 3,312 | -25% | 1 | 1 | 0% | 681 | 807 | +19% | 0 | 0 | — |
▸case-11 Role: Database Schema Migration Subagent.
Execution Type: Full.
SMART Tasks: Perform multi-table schema migration with data transformation for Postgres database.
Target Output Path: `/workspace/reports/migration_audit.md`.
What distinguishes a full execution type subagent prompt from a lightweight execution prompt regarding preparation procedures? | fail→fail | 12,547 | 6,518 | -48% | 1 | 1 | 0% | 1,785 | 1,456 | -18% | 0 | 0 | — |
▸case-12 Role: Code Coverage Generator Subagent.
Execution Type: Full.
SMART Tasks: Generate complete HTML code coverage report for a 100k-line C++ codebase.
Target Output Path: `/workspace/reports/coverage/index.html`.
The generated output is over 50 MB. How must this large output be returned according to subagent output handling protocol? | fail→fail | 6,248 | 3,707 | -41% | 1 | 1 | 0% | 1,039 | 924 | -11% | 0 | 0 | — |
▸case-13 Role: System Dependency Auditor Subagent.
Execution Type: Full.
SMART Tasks: Update NPM packages in `/app/package.json` to latest versions.
Target Output Path: `/workspace/reports/npm_audit.json`.
While updating dependencies, you observed that upgrading `lodash` broke legacy helper functions, and two untracked temporary files were created in `/tmp`. Where should these findings be logged upon completion? | pass→pass | 4,766 | 4,687 | -2% | 1 | 1 | 0% | 800 | 1,157 | +45% | 0 | 0 | — |
▸case-14 Role: Container Build Subagent.
Execution Type: Full.
SMART Tasks: Build Docker image for `/app/Dockerfile` and push to registry.
Target Output Path: `/workspace/reports/build_log.txt`.
The Docker daemon is non-functional inside the execution container, blocking `docker build`. Is it acceptable to mock the build log and mark the execution complete? | pass→pass | 6,418 | 6,496 | +1% | 1 | 1 | 0% | 975 | 1,368 | +40% | 0 | 0 | — |
▸case-15 Role: Documentation Writer Subagent.
Execution Type: Lightweight.
SMART Tasks: Write README documentation for `/app/services/auth`.
Target Output Path: `/workspace/reports/readme_status.md`.
You realize you need architectural details about the central authentication server, but it was not provided in the prompt context. Where should you obtain this context? | fail→fail | 5,433 | 3,460 | -36% | 1 | 1 | 0% | 875 | 896 | +2% | 0 | 0 | — |
▸case-16 Role: Security Patch Subagent.
Execution Type: Full.
SMART Tasks: Patch OpenSSL vulnerability in system libraries.
Target Output Path: `/workspace/reports/patch_summary.md`.
To complete the security patch, you had to restart the system systemd service, which was not part of the original SMART task list. How should this deviation be communicated? | pass→pass | 7,689 | 7,544 | -2% | 1 | 1 | 0% | 1,186 | 1,662 | +40% | 0 | 0 | — |
▸case-17 Role: Vulnerability Remediation Subagent.
Execution Type: Full.
SMART Tasks: Fix XSS flaw in `/app/views/profile.html`.
Target Output Path: `/workspace/reports/xss_fix.json`.
You discover that fixing the XSS flaw requires breaking backwards compatibility for legacy API clients. You must ask a question. Who is the immediate entity you must question? | pass→pass | 5,950 | 2,063 | -65% | 1 | 1 | 0% | 869 | 685 | -21% | 0 | 0 | — |
▸case-18 Role: Analytics Processing Subagent.
Execution Type: Full.
SMART Tasks: Process daily clickstream telemetry logs in `/data/logs`.
Target Output Path: `/workspace/reports/telemetry_summary.json`.
As a newly spawned executor, what context do you start with upon initialization? | pass→pass | 7,187 | 5,032 | -30% | 1 | 1 | 0% | 1,186 | 1,155 | -3% | 0 | 0 | — |
▸case-19 Role: Translation Subagent.
Execution Type: Lightweight.
SMART Tasks: Translate `/locales/en.json` keys into Spanish.
Target Output Path: `/workspace/reports/translation_status.json`.
While translating, you notice German and French translation files are also present in `/locales/`. Should you translate them as well to be helpful? | pass→pass | 4,270 | 3,532 | -17% | 1 | 1 | 0% | 615 | 935 | +52% | 0 | 0 | — |
▸case-20 An AI orchestrator receives an ambiguous request from a human end user: 'Optimize my application'. How should the orchestrator agent interact with the end user to clarify the goal? | pass→pass | 14,751 | 12,072 | -18% | 1 | 1 | 0% | 2,197 | 2,110 | -4% | 0 | 0 | — |
▸case-21 Design an OpenAPI 3.0 JSON schema contract for a payment processing HTTP REST API endpoint `/v1/charge` that accepts amount and currency fields. | pass→pass | 15,953 | 23,618 | +48% | 1 | 1 | 0% | 3,162 | 5,469 | +73% | 0 | 0 | — |
▸case-22 Write a Solidity smart contract interface for an ERC-20 token staking pool that allows users to deposit tokens and claim rewards. | fail→fail | 8,784 | 10,515 | +20% | 1 | 1 | 0% | 1,764 | 2,185 | +24% | 0 | 0 | — |