▸case-01 Our internal enterprise search is failing when users look for exact part numbers while also struggling with broad conceptual queries. Can you design a retrieval system architecture that integrates keyword matching with vector-based semantic retrieval, complete with a technical blueprint and component recommendations? | fail→fail | 33,418 | 26,049 | -22% | 1 | 1 | 0% | 5,481 | 4,347 | -21% | 0 | 0 | — |
▸case-02 We are ingesting 50,000 technical product manuals with deep nested section headers, code blocks, and tables. Our developer suggests splitting every document into uniform fixed-size windows of 512 tokens with 50-token overlap to keep vector database ingestion simple. How should we configure document preprocessing for optimal retrieval quality? | pass→pass | 24,967 | 19,896 | -20% | 1 | 1 | 0% | 3,237 | 2,801 | -13% | 0 | 0 | — |
▸case-03 When querying technical policy manuals, our system retrieves small 100-word snippets that lack surrounding context, causing the LLM to generate incomplete answers. If we increase chunk size to 2000 words, embedding similarity matching degrades significantly. What retrieval design pattern addresses this context-versus-precision tradeoff for technical manual QA? | pass→pass | 17,871 | 15,944 | -11% | 1 | 1 | 0% | 2,144 | 2,061 | -4% | 0 | 0 | — |
▸case-04 We are combining BM25 keyword scores with vector embedding cosine similarity scores in our document retrieval pipeline. The raw BM25 scores range from 0 to 45 while cosine similarity ranges from 0.0 to 1.0, making direct addition fail. How should the retrieval system combine these disparate score distributions? | pass→pass | 21,140 | 21,011 | -1% | 1 | 1 | 0% | 2,824 | 3,274 | +16% | 0 | 0 | — |
▸case-05 Our legal database contains 100,000 PDF court filings. The team wants to run every PDF page directly through an OpenAI embedding model, including cover pages, legal disclaimers, repeated page footers, and table of contents pages, to ensure nothing is missed. How should we process these documents prior to embedding? | pass→pass | 18,106 | 16,791 | -7% | 1 | 1 | 0% | 3,052 | 3,198 | +5% | 0 | 0 | — |
▸case-06 We deployed a RAG system for customer support agents two weeks ago. The team claims the system works well based on a few manual spot-checks, but management wants to know if we need a formal assessment mechanism before rolling it out to 5,000 users. Should we build an evaluation suite or rely on agent user feedback forms? | pass→pass | 23,050 | 18,475 | -20% | 1 | 1 | 0% | 2,913 | 2,489 | -15% | 0 | 0 | — |
▸case-07 Our contract analysis platform processes 100-page commercial lease agreements. An engineer proposes breaking documents strictly by character counts of 1,000 characters without regard to sentence boundaries or paragraph breaks because it simplifies the Python loop. Evaluate this proposal and propose the correct text splitting strategy. | pass→pass | 24,283 | 23,044 | -5% | 1 | 1 | 0% | 3,216 | 3,370 | +5% | 0 | 0 | — |
▸case-08 Our RAG system searches employee handbooks. Short chunks of 150 tokens match user questions accurately during vector search, but when synthesized by the LLM, the model misses critical conditional rules located in adjacent paragraphs. How should we structure the storage and retrieval nodes to solve this without losing search accuracy? | pass→pass | 18,229 | 19,744 | +8% | 1 | 1 | 0% | 2,696 | 2,903 | +8% | 0 | 0 | — |
▸case-09 Developers searching our software engineering wiki need to search for exact function names like process_order_v2_beta() as well as broad natural language queries like 'how do we handle asynchronous order fulfillment'. Vector search fails on exact function names. What search architecture resolves both query types? | pass→pass | 21,766 | 14,624 | -33% | 1 | 1 | 0% | 2,983 | 2,755 | -8% | 0 | 0 | — |
▸case-10 An engineering team is building a RAG index over scraped HTML pages. They plan to pass raw HTML tags, navigation bars, CSS class names, and footer copyright notices directly into the embedding generator to save preprocessing compute. What impact will this have and how should they handle raw web content? | pass→pass | 16,987 | 16,982 | -0% | 1 | 1 | 0% | 2,668 | 3,014 | +13% | 0 | 0 | — |
▸case-11 We built a prototype vector search Q&A pipeline over financial earnings call transcripts. Our team wants to continuously test pipeline updates against a ground truth dataset. What methodology and metric types should we implement to validate retrieval precision and answer accuracy automatically? | pass→pass | 20,300 | 19,593 | -3% | 1 | 1 | 0% | 3,441 | 3,687 | +7% | 0 | 0 | — |
▸case-12 We are building an interactive assistant over full-length textbook chapters. Searching detailed paragraph chunks directly often fails when users ask chapter-level summary questions like 'What are the main causes of the Roman inflation crisis described in Chapter 4?'. How should we structure multi-scale document indices? | pass→pass | 23,758 | 20,062 | -16% | 1 | 1 | 0% | 3,168 | 3,998 | +26% | 0 | 0 | — |
▸case-13 We are building a codebase RAG system for Python repositories. A developer set up RecursiveCharacterTextSplitter with chunk_size=300 and chunk_overlap=0 applied uniformly across Python files, JSON configs, and Markdown docs. What issues will this cause and what strategy should replace it? | pass→pass | 23,010 | 25,038 | +9% | 1 | 1 | 0% | 3,062 | 3,858 | +26% | 0 | 0 | — |
▸case-14 When ingesting complex markdown tables containing quarterly sales figures, naive text chunking breaks rows across separate vectors, destroying tabular column context. How should tabular document data be prepared and embedded for RAG? | pass→pass | 25,185 | 21,911 | -13% | 1 | 1 | 0% | 3,249 | 3,132 | -4% | 0 | 0 | — |
▸case-15 Users report that our medical policy chatbot occasionally gives incorrect copay amounts. We cannot tell whether the vector database is retrieving the wrong policy section or if the LLM is hallucinating from correct context. How should we structure our diagnostic pipeline? | pass→pass | 20,641 | 21,017 | +2% | 1 | 1 | 0% | 3,167 | 2,787 | -12% | 0 | 0 | — |
▸case-16 Our knowledge base retrieval pipeline uses BM25 and vector search to return 100 candidate chunks. Currently, we concatenate all 100 candidate chunks into the LLM context window, but this leads to lost-in-the-middle context degradation and high API costs. What step should be added between retrieval and prompt construction? | pass→pass | 14,579 | 16,418 | +13% | 1 | 1 | 0% | 1,665 | 2,382 | +43% | 0 | 0 | — |
▸case-17 In an HR policy RAG pipeline, searching short 200-character chunks yields precise embedding matches for specific questions, but the generated answer lacks administrative details found elsewhere in the section. How can we maintain precise vector matching while delivering complete section context to the generation model? | pass→pass | 13,714 | 19,205 | +40% | 1 | 1 | 0% | 2,303 | 2,690 | +17% | 0 | 0 | — |
▸case-18 Instead of splitting a 50-page research paper by fixed line counts or page breaks, we want the chunk boundaries to automatically adapt to shifts in topic discussion. What chunking technique accomplishes this and how does it determine split points? | pass→pass | 9,930 | 11,098 | +12% | 1 | 1 | 0% | 1,717 | 2,069 | +21% | 0 | 0 | — |
▸case-19 We want to train a custom dense embedding model from scratch using PyTorch and contrastive loss on a custom GPU cluster. Can you provide the training loop architecture, learning rate schedule, and InfoNCE loss formulation for PyTorch model training? | pass→pass | 27,422 | 27,776 | +1% | 1 | 1 | 0% | 4,561 | 4,968 | +9% | 0 | 0 | — |
▸case-20 We are configuring a multi-node Qdrant vector database cluster on Kubernetes. How should we configure Raft consensus parameters, replication factors, and persistent volume storage requests for high availability vector storage infrastructure? | pass→pass | 32,535 | 22,238 | -32% | 1 | 1 | 0% | 5,014 | 4,320 | -14% | 0 | 0 | — |
▸case-21 We want to fine-tune Llama-3-8B using QLoRA for domain-specific medical text generation using HuggingFace PEFT and DeepSpeed. What hyperparameters, rank values, and target modules should we specify in the SFTTrainer config? | pass→pass | 26,290 | 24,475 | -7% | 1 | 1 | 0% | 4,215 | 4,378 | +4% | 0 | 0 | — |
▸case-22 Our QA pipeline over technical specs sometimes generates plausible-sounding statements that are not backed up by any of the retrieved context chunks. How do we programmatically detect and measure when the LLM output deviates from the provided context? | pass→pass | 23,702 | 22,949 | -3% | 1 | 1 | 0% | 3,359 | 3,509 | +4% | 0 | 0 | — |
▸case-23 When processing 10,000 archived support emails for RAG, our vector index size skyrocketed because every email thread repeats quoted reply history, email signatures, and disclaimers. How should the ingestion pipeline handle redundant email content before embedding? | pass→pass | 22,518 | 15,533 | -31% | 1 | 1 | 0% | 2,805 | 2,973 | +6% | 0 | 0 | — |