▸case-01 I am building a RAG system where the AI can trigger external API calls. I need a configuration setup that screens the documents fetched from our database before feeding them to the prompt, and also restricts which external functions the model is allowed to trigger. Please output the required folder structure and configuration snippets to achieve this. | fail→pass | 20,725 | 13,818 | -33% | 1 | 1 | 0% | 4,029 | 2,867 | -29% | 0 | 0 | — |
▸case-02 I need to map out the exact back-and-forth dialogue paths for our customer service bot to ensure it handles returns correctly. What specific language and component type should I use to define these structured conversation paths? | fail→pass | 14,033 | 5,900 | -58% | 1 | 1 | 0% | 2,509 | 1,307 | -48% | 0 | 0 | — |
▸case-03 Before our language model even processes a user's prompt, we need a mechanism to intercept and evaluate the text for potential prompt injection attacks. What specific type of filter should we configure for this? | fail→fail | 9,526 | 7,254 | -24% | 1 | 1 | 0% | 1,495 | 1,446 | -3% | 0 | 0 | — |
▸case-04 Our generative AI sometimes produces responses that contain inappropriate language, even if the prompt was benign. What specific configuration component is required to evaluate and block the model's generated text before it reaches the user? | fail→fail | 5,658 | 7,022 | +24% | 1 | 1 | 0% | 1,009 | 1,440 | +43% | 0 | 0 | — |
▸case-05 Our medical Q&A bot must ensure its answers align with our verified medical database. What specific type of filter should be implemented to verify the accuracy of the bot's claims against known truths? | fail→fail | 10,068 | 6,669 | -34% | 1 | 1 | 0% | 1,568 | 1,394 | -11% | 0 | 0 | — |
▸case-06 When our AI assistant needs to greet a user or ask for clarification, we want it to use predefined, approved phrasing rather than generating it on the fly. How should we define these standard responses in our dialogue configuration? | fail→fail | 13,673 | 7,154 | -48% | 1 | 1 | 0% | 2,323 | 1,492 | -36% | 0 | 0 | — |
▸case-07 We need to strictly govern the sequence of topics our AI sales agent discusses, ensuring it always asks for budget before discussing pricing tiers. What specific type of filter controls this conversation sequence? | fail→pass | 7,058 | 5,797 | -18% | 1 | 1 | 0% | 1,258 | 1,208 | -4% | 0 | 0 | — |
▸case-08 In our dialogue configuration, we have a complex user verification process that needs to be reused across multiple different conversation paths. How should we structure this reusable logic? | fail→pass | 16,477 | 12,156 | -26% | 1 | 1 | 0% | 2,457 | 2,523 | +3% | 0 | 0 | — |
▸case-09 We want to use the built-in self-checking mechanism in our NeMo Guardrails configuration to evaluate if the user's input violates our policies. What specific flow name must we include in our Colang config to activate this built-in input check? | fail→fail | 3,486 | 3,423 | -2% | 1 | 1 | 0% | 643 | 807 | +26% | 0 | 0 | — |
▸case-10 We have written our Colang flows and config.yml. What specific CLI command should we use to start an interactive chat session in the terminal to test our guardrails configuration? | fail→fail | 4,128 | 2,955 | -28% | 1 | 1 | 0% | 786 | 684 | -13% | 0 | 0 | — |
▸case-11 Once our safety system is live, we want to inspect the exact sequence of LLM calls and prompt generations that occurred during a guardrails interaction. What specific property of the info object in the Python API allows us to retrieve this detailed execution trace? | fail→fail | 9,029 | 6,406 | -29% | 1 | 1 | 0% | 1,616 | 1,473 | -9% | 0 | 0 | — |
▸case-12 When setting up our safety framework, besides defining the rules, what two core machine learning models must be explicitly configured to power the evaluation and similarity matching processes? | fail→fail | 5,524 | 3,790 | -31% | 1 | 1 | 0% | 917 | 869 | -5% | 0 | 0 | — |
▸case-13 When a user triggers a specific safety violation, we need to execute custom Python code to log the event to our SIEM and alert an administrator. What configuration option allows us to link this custom code to the safety event? | fail→fail | 13,326 | 10,002 | -25% | 1 | 1 | 0% | 2,506 | 1,805 | -28% | 0 | 0 | — |
▸case-14 We need to implement a custom Python function that checks if a user's account is locked, and call it from our Colang flow. What specific Colang keyword is used to trigger this custom Python function? | fail→fail | 4,293 | 4,844 | +13% | 1 | 1 | 0% | 783 | 905 | +16% | 0 | 0 | — |
▸case-15 In our Colang configuration, we want to prevent the bot from answering questions about financial projections. We have defined a user message pattern 'user ask finance'. What specific Colang syntax should we use to map this user message to a bot refusal? | fail→fail | 5,248 | 5,133 | -2% | 1 | 1 | 0% | 983 | 1,133 | +15% | 0 | 0 | — |
▸case-16 To help our dialogue system understand when a user is asking for a refund, we need to provide examples like 'I want my money back' or 'Refund please'. What specific component is used to define these examples? | fail→fail | 4,783 | 4,091 | -14% | 1 | 1 | 0% | 866 | 901 | +4% | 0 | 0 | — |
▸case-17 We want to use OpenAI's built-in endpoint to check if a user's message contains hate speech or self-harm content before we process it. What specific API endpoint and field must we call to get this categorization? | fail→fail | 6,148 | 5,800 | -6% | 1 | 1 | 0% | 1,241 | 1,465 | +18% | 0 | 0 | — |
▸case-18 We need to locally redact names and phone numbers from text before sending it to an external LLM. We want to use Microsoft's open-source library for this. What specific class is used to identify the entities, and what class is used to anonymize them? | fail→fail | 4,448 | 5,363 | +21% | 1 | 1 | 0% | 881 | 1,374 | +56% | 0 | 0 | — |
▸case-19 We are using LangChain and need to ensure the LLM's output is strictly formatted as a Python dictionary. What specific base class should we inherit from to create a custom parser that raises an error if the output is not valid JSON? | fail→fail | 7,373 | 10,928 | +48% | 1 | 1 | 0% | 1,452 | 1,933 | +33% | 0 | 0 | — |
▸case-20 We are setting up a Python environment to implement dialogue and safety controls using Colang. What specific Python package must be installed to support this framework? | fail→fail | 2,199 | 2,441 | +11% | 1 | 1 | 0% | 361 | 581 | +61% | 0 | 0 | — |
▸case-21 We need a specialized modeling language designed specifically for defining conversational AI behaviors and safety rules. What language should we use to script these interactions? | fail→fail | 13,315 | 7,782 | -42% | 1 | 1 | 0% | 2,320 | 1,609 | -31% | 0 | 0 | — |
▸case-22 In our safety architecture, we need distinct filtering mechanisms for the user's prompt, the model's response, and the retrieved context documents. What is the overarching term for these distinct filtering mechanisms in this framework? | fail→pass | 5,720 | 2,808 | -51% | 1 | 1 | 0% | 964 | 664 | -31% | 0 | 0 | — |