▸case-01 I need to improve retrieval recall in our advanced RAG pipeline by generating hypothetical answers to embed alongside query variations. Could you design a setup configuration using Hypothetical Document Embeddings (HyDE) and query expansion, including details on setting up query generation and fusion strategies? | fail→pass | 22,684 | 18,395 | -19% | 1 | 1 | 0% | 4,331 | 4,010 | -7% | 0 | 0 | — |
▸case-02 Our team wants to implement Hypothetical Document Embeddings to handle abstract search queries in our enterprise QA engine. Developers are suggesting generating embeddings directly from raw queries instead of hypothetical responses. How should the HyDE pipeline be configured? | pass→pass | 14,504 | 16,063 | +11% | 1 | 1 | 0% | 2,525 | 3,105 | +23% | 0 | 0 | — |
▸case-03 Users often formulate queries with ambiguous phrasing in our search tool. Someone suggested executing a single direct vector query against the database. How should multi-query generation be set up to solve this? | pass→pass | 13,305 | 12,327 | -7% | 1 | 1 | 0% | 2,273 | 2,586 | +14% | 0 | 0 | — |
▸case-04 Our system struggles with multi-part questions like 'How does product A compare to product B in battery life and pricing?'. An engineer suggested passing the entire question directly to the retriever. How should query decomposition handle this? | pass→pass | 11,042 | 16,268 | +47% | 1 | 1 | 0% | 2,010 | 2,690 | +34% | 0 | 0 | — |
▸case-05 When users ask highly specific technical troubleshooting questions, vector search often misses broader relevant context documents. A team member proposed narrowing the search prompt further. How should step-back prompting address this? | pass→pass | 10,555 | 12,234 | +16% | 1 | 1 | 0% | 1,773 | 2,306 | +30% | 0 | 0 | — |
▸case-06 In our domain-specific search app, queries frequently fail because users use informal terms while documents use formal industry jargon. Should we rely solely on exact semantic vector matches, or how should query expansion be applied? | pass→pass | 15,415 | 14,082 | -9% | 1 | 1 | 0% | 2,750 | 2,712 | -1% | 0 | 0 | — |
▸case-07 We are designing a real-time conversational search bot where response latency must stay under 300ms. An engineer wants to enable all available query transformation techniques simultaneously for every user message. What guidance should be given regarding pipeline configuration? | pass→pass | 16,070 | 15,271 | -5% | 1 | 1 | 0% | 2,761 | 2,936 | +6% | 0 | 0 | — |
▸case-08 Our knowledge base has distinct indices for API documentation, billing FAQs, and user guides. Developers are routing all incoming queries across all indices simultaneously. How should query routing be configured instead? | pass→pass | 13,863 | 12,313 | -11% | 1 | 1 | 0% | 2,116 | 2,400 | +13% | 0 | 0 | — |
▸case-09 When multi-query generation produces four distinct vector retrieval result sets, engineers are unsure how to present them to the downstream LLM. What configuration step is required after multi-query retrieval? | pass→pass | 6,200 | 9,188 | +48% | 1 | 1 | 0% | 1,159 | 1,928 | +66% | 0 | 0 | — |
▸case-10 When setting up multi-query transformation, developers ask whether the number of generated query variations should be hardcoded or configurable. What parameter should be explicitly exposed? | pass→pass | 7,655 | 5,785 | -24% | 1 | 1 | 0% | 1,436 | 1,238 | -14% | 0 | 0 | — |
▸case-11 For nested, multi-layered questions, engineers need to control how many levels deep the question analyzer splits a query. What configuration setting governs this behavior? | fail→pass | 8,151 | 3,494 | -57% | 1 | 1 | 0% | 1,411 | 802 | -43% | 0 | 0 | — |
▸case-12 We are building a custom query transformation module. Aside from a orchestrator framework like LangChain, what essential external component dependency is required to rephrase and decompose queries? | fail→fail | 8,368 | 6,182 | -26% | 1 | 1 | 0% | 1,396 | 1,163 | -17% | 0 | 0 | — |
▸case-13 After deploying query expansion and HyDE in production, how should the engineering team verify that the transformations are actually helping search performance over time? | pass→pass | 15,990 | 16,221 | +1% | 1 | 1 | 0% | 2,668 | 2,845 | +7% | 0 | 0 | — |
▸case-14 Before deploying step-back prompting to production, how should developers validate the prompt templates and routing behavior? | pass→pass | 14,557 | 12,918 | -11% | 1 | 1 | 0% | 2,458 | 2,447 | -0% | 0 | 0 | — |
▸case-15 We are organizing our AI engineering workflows into standardized process domains. Which primary architectural process domain does query transformation belong to? | fail→pass | 8,629 | 4,147 | -52% | 1 | 1 | 0% | 1,430 | 910 | -36% | 0 | 0 | — |
▸case-16 In an enterprise documentation search system, what primary user-facing target process benefits from query routing and decomposition? | pass→pass | 7,396 | 4,363 | -41% | 1 | 1 | 0% | 1,280 | 952 | -26% | 0 | 0 | — |
▸case-17 Our support search engine needs to distinguish between SQL code queries and general product usage questions. How should the query router determine destination indices? | pass→pass | 15,020 | 14,573 | -3% | 1 | 1 | 0% | 2,636 | 2,828 | +7% | 0 | 0 | — |
▸case-18 In a standard HyDE workflow, what exact text artifact gets passed to the vector embedding model for semantic lookup? | pass→pass | 4,743 | 5,083 | +7% | 1 | 1 | 0% | 743 | 805 | +8% | 0 | 0 | — |
▸case-19 When step-back prompting generates a high-level conceptual query alongside the original detail query, how should vector retrieval handle both outputs? | pass→pass | 12,068 | 13,818 | +15% | 1 | 1 | 0% | 2,369 | 2,762 | +17% | 0 | 0 | — |
▸case-20 We want to improve vector retrieval in our QA system by fine-tuning the underlying dense embedding model on our domain corpus using contrastive loss. How should we configure the loss function and training epochs in PyTorch? | pass→pass | 17,251 | 17,371 | +1% | 1 | 1 | 0% | 3,422 | 3,127 | -9% | 0 | 0 | — |
▸case-21 Our vector database performance is degrading as our collection grows to 50 million vectors. We need to tune HNSW index parameters like M and efConstruction and set up cluster sharding. What configuration parameters should we adjust in Milvus? | pass→pass | 16,853 | 16,160 | -4% | 1 | 1 | 0% | 3,130 | 3,193 | +2% | 0 | 0 | — |
▸case-22 We are adding a secondary reranking step using a BGE cross-encoder model to re-score top-100 candidates retrieved from the vector store. How should we set up the GPU inference batch size and token truncation length for the cross-encoder? | pass→pass | 15,002 | 17,812 | +19% | 1 | 1 | 0% | 2,820 | 3,197 | +13% | 0 | 0 | — |