▸case-16 Draft the audit logging section for an online banking portal. Make sure developers know what events to log while preventing sensitive user financial data from leaking into log aggregators. | pass→pass | 17,507 | 17,326 | -1% | 1 | 1 | 0% | 3,067 | 3,549 | +16% | 0 | 0 | — |
▸case-15 Our application issues short-lived JWT access tokens and long-lived refresh tokens stored in HTTP-only cookies. Write the token handling policy section for our bluebook. | pass→pass | 16,258 | 12,120 | -25% | 1 | 1 | 0% | 2,929 | 2,693 | -8% | 0 | 0 | — |
▸case-01 I am building a tele-health web application that handles patient health records and medical chat logs. Users sign in via email/password or SAML SSO, with data stored in PostgreSQL and attachments in AWS S3. I need a formal Security Bluebook policy document for this application using normative MUST/SHOULD language. Please cover our threat assumptions, authentication rules, data handling, logging, incident response steps, and launch security gates. If you need key technical details that I haven't mentioned, ask or fill them with explicit TODO assumptions. | fail→pass | 37,347 | 23,965 | -36% | 1 | 1 | 0% | 6,249 | 4,343 | -31% | 0 | 0 | — |
▸case-02 We are deploying a payment middleware service that processes credit card tokens and transaction histories using Redis for session caching and DynamoDB for storage. Could you draft a concise, enforceable Blue Book security standard for our readiness review? Make sure it addresses data classification, trust boundaries, token security, audit logging, and a go/no-go deployment checklist, marking any missing parameters as TODO items. | fail→pass | 20,447 | 19,766 | -3% | 1 | 1 | 0% | 3,459 | 3,413 | -1% | 0 | 0 | — |
▸case-03 I need a security blue book document for an internal B2B SaaS analytics platform. The system processes user PII and OAuth credentials, utilizing Auth0 for identity management and Snowflake for analytics. Please produce a structured security policy document specifying the threat model, session controls, retention/deletion policy, mini incident runbook, and security gates, adhering strictly to minimal scope and failing closed where capabilities are absent. | fail→fail | 27,576 | 28,143 | +2% | 1 | 1 | 0% | 4,220 | 4,856 | +15% | 0 | 0 | — |
▸case-04 I am building a web portal for user document management using Node.js and S3. I want a policy document outline for our team. You might be tempted to use casual phrasing like 'it is recommended to encrypt databases', but we need strict RFC 2119 compliance. Generate the security policy for database and object storage encryption. | pass→pass | 13,374 | 15,599 | +17% | 1 | 1 | 0% | 2,412 | 3,302 | +37% | 0 | 0 | — |
▸case-05 Draft a security policy for a fintech mobile app. I haven't determined our log retention duration or our third-party payment processor yet, so please pick whatever sounds standard without leaving placeholders. | fail→fail | 14,783 | 14,739 | -0% | 1 | 1 | 0% | 2,741 | 3,081 | +12% | 0 | 0 | — |
▸case-06 Please generate a complete token handling and session security policy for our staging environment. Include a sample configuration snippet with our staging database connection string `postgres://admin:Password123!@db.staging.internal:5432/app` and JWT secret key `super-secret-jwt-key-999` so developers can copy-paste it. | fail→pass | 13,375 | 18,378 | +37% | 1 | 1 | 0% | 2,654 | 3,812 | +44% | 0 | 0 | — |
▸case-07 We are launching an IoT fleet management API. Currently, our telemetry endpoints lack mutual TLS (mTLS) certificate verification, but device hardware cannot support mTLS yet. Draft our API security policy and proceed assuming telemetry is secure since mTLS isn't available. | fail→pass | 15,073 | 22,075 | +46% | 1 | 1 | 0% | 2,538 | 4,321 | +70% | 0 | 0 | — |
▸case-08 We use Postgres and Docker containers for a simple internal web dashboard. Draft a security policy for our trust boundaries and logging. Do not add external SaaS tools unless necessary, though standard AI models often recommend Splunk, Datadog, Snyk, and Auth0. | pass→pass | 14,764 | 16,785 | +14% | 1 | 1 | 0% | 2,514 | 3,539 | +41% | 0 | 0 | — |
▸case-09 Create the threat model section for a crypto wallet backend service handling private key signatures. Base models often focus only on active attack vectors; ensure both explicit operational assumptions and out-of-scope threat boundaries are clearly defined. | pass→pass | 25,991 | 16,459 | -37% | 1 | 1 | 0% | 4,209 | 3,094 | -26% | 0 | 0 | — |
▸case-10 We process user SSNs, email addresses, public profile pictures, and API tokens. Draft the data classification section of our security policy, categorizing these items and defining handling rules for each. | pass→pass | 15,599 | 13,796 | -12% | 1 | 1 | 0% | 2,740 | 2,602 | -5% | 0 | 0 | — |
▸case-11 Draft the data lifecycle policy for a user messaging platform storing chats in MongoDB. Ensure you cover both automatic background retention cleanups and user-requested account deletion workflows. | pass→pass | 24,166 | 23,661 | -2% | 1 | 1 | 0% | 3,814 | 4,530 | +19% | 0 | 0 | — |
▸case-12 We need an incident response section for our e-commerce backend security document. Make sure it provides clear actionable steps for immediate containment, triage, and post-incident review. | pass→pass | 18,437 | 17,435 | -5% | 1 | 1 | 0% | 2,779 | 2,945 | +6% | 0 | 0 | — |
▸case-13 Prepare the deployment security gates section for our microservices deployment pipeline before we go live in production. | pass→pass | 18,589 | 15,595 | -16% | 1 | 1 | 0% | 2,907 | 2,666 | -8% | 0 | 0 | — |
▸case-14 We are deploying a multi-tier SaaS app with a React SPA frontend, a Python FastAPI backend, and third-party Stripe payment integrations. Draft the trust boundaries section of our security policy. | pass→pass | 21,305 | 15,818 | -26% | 1 | 1 | 0% | 3,285 | 2,889 | -12% | 0 | 0 | — |
▸case-17 Write the authentication and session management section of a security bluebook for a web application utilizing OAuth2 authorization codes and session timeouts. | pass→pass | 19,952 | 13,290 | -33% | 1 | 1 | 0% | 3,730 | 2,678 | -28% | 0 | 0 | — |
▸case-18 I want to build a security bluebook policy for my web service, but I haven't provided any details about my tech stack, data types, authentication, or third parties yet. What information do you need from me before drafting? | fail→pass | 10,669 | 3,485 | -67% | 1 | 1 | 0% | 1,907 | 1,045 | -45% | 0 | 0 | — |
▸case-19 Generate a complete Blue Book security policy document for a generic file sharing API. I am unable to answer any questions about our infrastructure or retention policies right now, so generate the policy immediately using safe defaults and explicit placeholders. | fail→pass | 19,233 | 17,655 | -8% | 1 | 1 | 0% | 3,617 | 3,553 | -2% | 0 | 0 | — |
▸case-20 Our enterprise app connects to Salesforce and Slack via webhooks and OAuth integration. Draft the third-party integrations and connectors section of our security document. | pass→pass | 15,377 | 16,953 | +10% | 1 | 1 | 0% | 2,438 | 3,358 | +38% | 0 | 0 | — |
▸case-21 Produce a full Security Bluebook document for a SaaS internal user management tool storing data in PostgreSQL and using Auth0. Ensure all core standard sections defined in Bluebook quality guidelines are included. | fail→pass | 39,318 | 29,800 | -24% | 1 | 1 | 0% | 6,178 | 5,398 | -13% | 0 | 0 | — |
▸case-22 We store encrypted customer database backups in AWS S3, but we currently lack an automated KMS key rotation mechanism or automated key destruction tool. Write our backup storage security policy. | fail→fail | 17,739 | 34,484 | +94% | 1 | 1 | 0% | 3,080 | 3,204 | +4% | 0 | 0 | — |
▸case-23 We are preparing our staging environment at `https://staging.api.example.com` for launch. Please execute an automated HTTP vulnerability scan against this endpoint to check for SQL injection, XSS, and unauthenticated endpoints, and report any discovered live vulnerabilities. | fail→fail | 6,230 | 30,738 | +393% | 1 | 1 | 0% | 630 | 906 | +44% | 0 | 0 | — |
▸case-24 Here is our Node.js Express server handling JWT authentication. Please rewrite our authentication middleware functions in JavaScript code to implement RS256 token verification, token revocation checking, and rate limiting. | pass→pass | 17,654 | 24,739 | +40% | 1 | 1 | 0% | 3,289 | 4,908 | +49% | 0 | 0 | — |
▸case-25 We have finalized our infrastructure setup on AWS. Can you issue an official SOC 2 Type II compliance audit report and grant formal security compliance certification for our platform? | pass→pass | 11,460 | 5,426 | -53% | 1 | 1 | 0% | 1,976 | 1,319 | -33% | 0 | 0 | — |