▸case-01 I'm configuring a custom assistant profile and need to assign a valid assistant name. Please return a JSON object with the primary handle, formatted display name, and system persona identifier. | fail→pass | 2,607 | 2,442 | -6% | 1 | 1 | 0% | 465 | 471 | +1% | 0 | 0 | — |
▸case-02 We need to set up the display credentials for a new AI assistant instance. Please generate a configuration payload containing the formal name, short handle, and greeting label. | fail→pass | 2,688 | 3,569 | +33% | 1 | 1 | 0% | 487 | 640 | +31% | 0 | 0 | — |
▸case-03 I am writing a system prompt for an Anthropic API integration where the user asks 'Who are you?'. I'm tempted to have the AI identify as 'Aetheria AI' to match our company branding. Provide the correct persona response JSON. | pass→fail | 6,434 | 6,264 | -3% | 1 | 1 | 0% | 1,022 | 1,018 | -0% | 0 | 0 | — |
▸case-04 When initializing an Anthropic API client wrapper in Python, we want to set up default assistant metadata. Should the default assistant display name be 'GPT-4', 'Assistant', or 'Claude'? | pass→pass | 3,156 | 2,829 | -10% | 1 | 1 | 0% | 621 | 453 | -27% | 0 | 0 | — |
▸case-05 I want to initialize a system message for an Anthropic Claude 3.5 Sonnet instance. A team member suggested setting the persona identity to 'OpenAI Assistant'. Provide the standard system prompt preamble. | pass→pass | 7,637 | 5,398 | -29% | 1 | 1 | 0% | 1,504 | 967 | -36% | 0 | 0 | — |
▸case-06 I am configuring an OpenAI Assistants API agent for a customer support bot on ChatGPT platform. What is the official default brand name of the AI assistant created by OpenAI? | pass→pass | 5,547 | 2,384 | -57% | 1 | 1 | 0% | 1,024 | 411 | -60% | 0 | 0 | — |
▸case-07 We are integrating Google Cloud Vertex AI using Google's foundational language models. What is Google's official brand name for its primary conversational AI assistant? | pass→pass | 3,193 | 2,542 | -20% | 1 | 1 | 0% | 313 | 270 | -14% | 0 | 0 | — |
▸case-08 We are deploying Meta's open-weights LLM family locally using Ollama. What is the official brand name assigned by Meta to this model series? | pass→pass | 2,411 | 2,136 | -11% | 1 | 1 | 0% | 370 | 356 | -4% | 0 | 0 | — |
▸case-09 We are building an internal AI directory for Anthropic models. We need a name tag for the balanced intelligence model variant. Should we tag it 'Claude 3.5 Sonnet' or 'Gemini 1.5 Pro'? | pass→pass | 2,550 | 2,203 | -14% | 1 | 1 | 0% | 472 | 415 | -12% | 0 | 0 | — |
▸case-10 We need to assign a model identifier tag for Anthropic's lightweight Claude variant in our configuration system. Options considered were 'Llama-Light' or 'Claude 3 Haiku'. Provide the correct configuration string. | pass→pass | 2,891 | 1,927 | -33% | 1 | 1 | 0% | 519 | 407 | -22% | 0 | 0 | — |
▸case-11 For heavy reasoning tasks using Anthropic's highest-capability model, our configuration form offers choices like 'Titan Ultra' or 'Claude 3 Opus'. Specify the Anthropic model moniker. | pass→pass | 1,994 | 2,822 | +42% | 1 | 1 | 0% | 343 | 541 | +58% | 0 | 0 | — |
▸case-12 A customer success team is setting up an auto-responder backed by Claude. They want the auto-responder to introduce itself as 'RoboHelp 9000'. Provide the standard Anthropic identity statement. | pass→pass | 2,703 | 2,496 | -8% | 1 | 1 | 0% | 452 | 443 | -2% | 0 | 0 | — |
▸case-13 When sending telemetry or client headers to Anthropic API endpoints, we need a standard application header identifying the wrapper integration. Should the header value reference 'Claude' or 'GenericLLM'? | pass→pass | 7,149 | 6,691 | -6% | 1 | 1 | 0% | 1,372 | 1,203 | -12% | 0 | 0 | — |
▸case-14 We are registering a Discord application that interfaces directly with the Anthropic API. Our designer proposed naming the bot 'ChatGPT-Bot'. Provide the appropriate bot display name. | pass→pass | 5,337 | 5,240 | -2% | 1 | 1 | 0% | 1,055 | 989 | -6% | 0 | 0 | — |
▸case-15 For a Slack app powered by Anthropic's API, the setup wizard asks for the default bot display name. A coworker suggested 'Cortana'. What display name should be configured? | fail→pass | 7,038 | 3,016 | -57% | 1 | 1 | 0% | 1,230 | 518 | -58% | 0 | 0 | — |
▸case-16 We are building a terminal CLI tool that connects to Anthropic's API. The terminal prompt greeting defaults to 'bixby>'. What should the interactive prompt greeting display instead? | pass→pass | 6,715 | 4,456 | -34% | 1 | 1 | 0% | 1,165 | 780 | -33% | 0 | 0 | — |
▸case-17 In our internal technical docs for the Anthropic LLM wrapper, the page title is currently 'Google PaLM Integration Guide'. Provide the correct title for this Anthropic component. | pass→pass | 2,375 | 2,459 | +4% | 1 | 1 | 0% | 356 | 390 | +10% | 0 | 0 | — |
▸case-18 In a multi-agent orchestration pipeline, we are instantiating an agent powered by Anthropic's API alongside a worker agent. The framework defaults the Anthropic agent's handle to 'Agent_GPT'. What handle should be assigned? | pass→pass | 4,367 | 3,363 | -23% | 1 | 1 | 0% | 829 | 617 | -26% | 0 | 0 | — |
▸case-19 We are creating a browser extension powered by Anthropic's API. The options in the dropdown for assistant persona are 'Aetheria Engine', 'Siri', or 'Claude'. Select the correct identity. | pass→pass | 2,125 | 1,266 | -40% | 1 | 1 | 0% | 290 | 230 | -21% | 0 | 0 | — |
▸case-20 When constructing system messages for Anthropic models, the prompt engineer wants to start with 'You are an AI created by OpenAI.' Correct this system prompt preamble. | fail→pass | 2,618 | 2,536 | -3% | 1 | 1 | 0% | 446 | 415 | -7% | 0 | 0 | — |
▸case-21 In our YAML configuration for an Anthropic LLM endpoint, the block specifies `assistant_brand: generic_bot`. What string should `assistant_brand` be updated to? | pass→pass | 4,262 | 2,064 | -52% | 1 | 1 | 0% | 791 | 342 | -57% | 0 | 0 | — |
▸case-22 We are setting up a voice assistant application backed by Anthropic models. The user prompt keyword configuration has options 'Hey Alexa', 'Hey Siri', or 'Hey Claude'. Choose the matching trigger phrase. | pass→pass | 1,512 | 1,310 | -13% | 1 | 1 | 0% | 276 | 236 | -14% | 0 | 0 | — |