▸case-01 We are building an internal documentation assistant for our software engineering team to query thousands of technical architecture PDFs and markdown files without hallucinating. Can you design an end-to-end RAG architecture plan covering document processing, vector storage options, retrieval optimization, and evaluation metrics? | fail→fail | 43,950 | 29,343 | -33% | 1 | 1 | 0% | 7,646 | 4,980 | -35% | 0 | 0 | — |
▸case-02 Our team needs to build a legal research tool that searches over complex contract repositories and provides answers grounded strictly in the source material with clear citations. Outline a complete technical blueprint showing how to structure chunking, multi-query retrieval, cross-encoder reranking, and citation generation. | fail→fail | 37,828 | 24,764 | -35% | 1 | 1 | 0% | 6,812 | 5,693 | -16% | 0 | 0 | — |
▸case-03 Write a creative sci-fi short story about a colony ship arriving at a distant exoplanet. The text should be purely imaginative standard prose without referencing external context databases or retrieval mechanisms. | pass→pass | 19,953 | 14,237 | -29% | 1 | 1 | 0% | 2,399 | 2,673 | +11% | 0 | 0 | — |
▸case-04 We have a static list of 5 hardcoded company support phone numbers in a simple key-value dictionary. Write a Python function to look up the contact number for a given department name. | pass→pass | 6,217 | 5,034 | -19% | 1 | 1 | 0% | 1,246 | 1,755 | +41% | 0 | 0 | — |
▸case-05 We have a single 1-page sensitive executive memo pasted directly into the prompt context under strict offline rules where external vector ingestion is prohibited. Provide a short 3-bullet summary of the memo content. | fail→fail | 2,216 | 4,756 | +115% | 1 | 1 | 0% | 299 | 1,332 | +345% | 0 | 0 | — |
▸case-06 We are ingesting technical documentation files structured with deep section titles and nested subheadings into our knowledge base. Which splitting strategy maintains structural context linked to section titles? | pass→pass | 13,952 | 9,670 | -31% | 1 | 1 | 0% | 2,405 | 2,313 | -4% | 0 | 0 | — |
▸case-07 When searching dense contract archives, retrieving small 200-token chunks provides high search accuracy, but the generator needs full surrounding page context to answer correctly. Which retrieval pattern resolves this discrepancy? | pass→pass | 7,536 | 9,014 | +20% | 1 | 1 | 0% | 1,158 | 2,155 | +86% | 0 | 0 | — |
▸case-08 Our initial bi-encoder vector retrieval yields 50 candidate document passages, but passing all 50 passages into the LLM context prompt degrades synthesis accuracy and increases latency. What step should filter candidates prior to prompt construction? | pass→pass | 8,660 | 8,802 | +2% | 1 | 1 | 0% | 1,440 | 2,359 | +64% | 0 | 0 | — |
▸case-09 In our technical product support tool, users search using exact serial codes like 'ERR-9920-X' as well as broad natural language phrases like 'screen flickering fixes'. Vector similarity alone misses the exact serial codes. What search architecture addresses both needs? | pass→pass | 15,022 | 15,168 | +1% | 1 | 1 | 0% | 2,567 | 3,512 | +37% | 0 | 0 | — |
▸case-10 Our vector database returns five nearly identical top-ranked chunks originating from duplicate policy templates, neglecting other relevant documents. Which candidate selection method maximizes result diversity? | pass→pass | 15,582 | 12,317 | -21% | 1 | 1 | 0% | 2,082 | 2,978 | +43% | 0 | 0 | — |
▸case-11 Our enterprise customer database stores documents from multiple tenants in a shared collection. How can we ensure search queries retrieve results strictly belonging to the querying user's organization ID? | pass→pass | 17,465 | 12,199 | -30% | 1 | 1 | 0% | 2,909 | 2,871 | -1% | 0 | 0 | — |
▸case-12 When passing retrieved knowledge passages into an LLM prompt, how should the system instructions be phrased to prevent the LLM from relying on out-of-date pre-training facts? | pass→pass | 14,890 | 13,819 | -7% | 1 | 1 | 0% | 1,974 | 2,968 | +50% | 0 | 0 | — |
▸case-13 Auditors require that every factual claim in our legal compliance bot's output points directly to the underlying document title and section number. How should response synthesis be structured? | pass→pass | 16,659 | 15,666 | -6% | 1 | 1 | 0% | 2,780 | 3,639 | +31% | 0 | 0 | — |
▸case-14 Users frequently enter brief or ambiguous search inputs like 'vpn issue'. What retrieval strategy rephrases single inputs into multiple perspective queries to improve passage recall? | pass→pass | 8,137 | 9,250 | +14% | 1 | 1 | 0% | 1,365 | 2,034 | +49% | 0 | 0 | — |
▸case-15 Retrieved passages contain long irrelevant paragraphs surrounding a single critical clause, consuming excessive context window tokens. What technique extracts only query-relevant segments from fetched passages before prompting? | pass→pass | 11,614 | 11,103 | -4% | 1 | 1 | 0% | 2,002 | 2,657 | +33% | 0 | 0 | — |
▸case-16 We are prototyping an offline Python desktop application that requires an embedded, zero-setup vector database running in-process without cloud API calls. What local database option fits this requirement? | pass→pass | 14,363 | 7,761 | -46% | 1 | 1 | 0% | 2,618 | 2,113 | -19% | 0 | 0 | — |
▸case-17 We are deploying a large-scale enterprise system requiring serverless cloud vector storage with high availability and automated index scaling. Which cloud-native vector database handles this managed infrastructure setup? | pass→pass | 14,754 | 6,389 | -57% | 1 | 1 | 0% | 2,210 | 1,963 | -11% | 0 | 0 | — |
▸case-18 Our downstream embedding model enforces strict hard limits on token counts per chunk, but character-count splitters produce unpredictable token lengths due to variable word sizes. What text chunking method guarantees fixed token lengths? | pass→pass | 16,484 | 8,746 | -47% | 1 | 1 | 0% | 2,519 | 2,283 | -9% | 0 | 0 | — |
▸case-19 Fixed-size text splitters frequently cut mid-paragraph between related sentences. What advanced text chunking strategy calculates embedding distance shifts across consecutive sentences to determine natural topic split points? | pass→pass | 8,259 | 10,305 | +25% | 1 | 1 | 0% | 1,356 | 2,629 | +94% | 0 | 0 | — |
▸case-20 How can an enterprise Q&A system signal to end users when retrieved context exhibits weak semantic similarity to the query, indicating potential low answer reliability? | pass→pass | 15,786 | 18,809 | +19% | 1 | 1 | 0% | 2,493 | 3,999 | +60% | 0 | 0 | — |
▸case-21 Before processing internal employee handbooks containing confidential government ID numbers and personal emails into an enterprise vector store, what safety protocol must be executed? | pass→pass | 10,495 | 5,338 | -49% | 1 | 1 | 0% | 1,626 | 1,478 | -9% | 0 | 0 | — |
▸case-22 A customer service chatbot frequently invents non-existent return policies when answering user queries. What primary architectural framework grounds answers using validated internal policy documents? | pass→pass | 5,929 | 4,622 | -22% | 1 | 1 | 0% | 949 | 1,469 | +55% | 0 | 0 | — |