▸case-01 Our RAG system for medical documents performs well on general concepts, but fails when clinicians search for exact catalog codes like 'RX-9021'. I need a step-by-step engineering plan with Python code snippets showing how to construct a dual retrieval pipeline combining dense vector embeddings with BM25 keyword matching, including a validation strategy to ensure exact identifiers and semantic context are both retrieved reliably. | pass→pass | 29,601 | 23,785 | -20% | 1 | 1 | 0% | 5,754 | 4,681 | -19% | 0 | 0 | — |
▸case-02 We are building an e-commerce catalog search service where semantic search alone frequently misses exact part numbers and model acronyms. Please draft a technical blueprint that outlines how to integrate full-text keyword indexing alongside vector similarity search, complete with clear setup steps, component architecture, and a sample script to execute unified queries. | pass→fail | 49,627 | 25,236 | -49% | 1 | 1 | 0% | 7,596 | 5,181 | -32% | 0 | 0 | — |
▸case-03 I need to upgrade our enterprise support knowledge base retrieval because pure embedding matching overlooks specific error codes and software syntax. Can you provide a comprehensive implementation guide formatted with architectural recommendations, execution phases, and verification criteria to merge lexical keyword search with semantic vector matching? | fail→fail | 50,320 | 29,496 | -41% | 1 | 1 | 0% | 8,239 | 4,696 | -43% | 0 | 0 | — |
▸case-04 We are designing a hybrid search pipeline for a legal discovery system. Engineers are debating whether to combine BM25 and dense vector score ranges using raw min-max score normalization vs Reciprocal Rank Fusion (RRF). Standard vector cosine scores range from 0 to 1, while BM25 scores are unbounded positive floats. The team wants to default to adding raw BM25 and cosine scores directly together. Draft a technical decision record and python code snippet for rank fusion. | pass→pass | 28,937 | 22,379 | -23% | 1 | 1 | 0% | 5,508 | 4,255 | -23% | 0 | 0 | — |
▸case-05 In our financial RAG assistant, we currently run a dual sparse-dense retrieval step returning the top 50 candidates from each search path. The team wants to simply take the top 5 dense results without re-ranking or score fusion. Draft a two-stage retrieval specification that improves top-5 precision for complex queries containing financial ratios and company names. | pass→pass | 27,949 | 25,464 | -9% | 1 | 1 | 0% | 4,444 | 5,060 | +14% | 0 | 0 | — |
▸case-06 Our customer support search engine uses a convex combination formula: score = alpha * dense_score + (1 - alpha) * sparse_score. The team wants to permanently set alpha to 1.0 to save compute, assuming vector search handles all queries. Provide an analysis and calibration guide for tuning the alpha hyperparameter across different query types like acronym lookups vs concept questions. | pass→pass | 19,544 | 34,143 | +75% | 1 | 1 | 0% | 3,782 | 5,390 | +43% | 0 | 0 | — |
▸case-07 We are building an online documentation search platform for developer APIs. Engineers are deciding whether to use traditional BM25 inverted index vs learned sparse representations (SPLADE) alongside dense vectors. A teammate suggests that dense vectors render all sparse representations redundant for code identifier search. Draft an architectural comparison and pipeline recommendation for developer search. | pass→pass | 33,948 | 32,075 | -6% | 1 | 1 | 0% | 4,218 | 5,287 | +25% | 0 | 0 | — |
▸case-08 In our multi-tenant SaaS application, users search across million-document workspaces. Developers suggest performing top-k hybrid search across the entire global index first, and then filtering out unauthorized tenant documents in post-processing. Draft a security and performance assessment with architectural recommendations for hybrid retrieval with tenant isolation. | pass→pass | 29,693 | 25,600 | -14% | 1 | 1 | 0% | 4,710 | 3,868 | -18% | 0 | 0 | — |
▸case-09 We implemented Reciprocal Rank Fusion with formula RRF_score = sum(1 / (k + rank)). The team set k=0, expecting higher accuracy by penalizing lower ranks heavily, but this causes top rank 1 results to dominate excessively and divides by rank value directly. Draft an engineering recommendation for selecting the smoothing constant k in RRF fusion. | pass→pass | 19,967 | 20,410 | +2% | 1 | 1 | 0% | 3,825 | 4,061 | +6% | 0 | 0 | — |
▸case-10 We deployed a hybrid search engine combining BM25 and dense embeddings for an e-commerce catalog, but management wants proof that hybrid search outperforms pure vector search. The team proposes evaluating success by counting total page views on top results. Draft a quantitative search evaluation framework with industry-standard metrics and verification procedures. | pass→pass | 27,942 | 26,299 | -6% | 1 | 1 | 0% | 4,409 | 4,341 | -2% | 0 | 0 | — |
▸case-11 Our enterprise search platform serves diverse queries ranging from exact ISBN numbers ('978-0-13-110362-7') to abstract requests ('books about system architecture principles'). Developers propose passing all queries through identical 50/50 weighted hybrid search without analyzing query characteristics. Draft an adaptive query routing pattern for hybrid search. | pass→pass | 42,432 | 21,948 | -48% | 1 | 1 | 0% | 6,593 | 4,178 | -37% | 0 | 0 | — |
▸case-12 When preparing technical manuals for hybrid search indexing, engineers want to set chunk sizes to 4,000 tokens per chunk without overlap so that vector embeddings cover entire chapters. Draft a chunking strategy analysis explaining how large chunk sizes affect BM25 term frequency scores and dense vector embedding resolution. | pass→pass | 22,750 | 21,970 | -3% | 1 | 1 | 0% | 3,916 | 3,980 | +2% | 0 | 0 | — |
▸case-13 Our real-time search endpoint must respond in under 50ms. Currently, the backend executes BM25 search sequentially, waits for the result, then executes dense vector ANN search, and finally applies rank fusion. Draft an asynchronous, parallel execution pipeline strategy to fit within the latency SLA. | pass→pass | 20,116 | 25,109 | +25% | 1 | 1 | 0% | 3,518 | 4,620 | +31% | 0 | 0 | — |
▸case-14 In our medical literature search system, developers disabled stemming and lemmatization on the keyword index to save storage, relying on dense embeddings to handle morphological variations of medical terms like 'vacuolated' vs 'vacuoles'. Draft a technical analysis on why keyword stemming/lemmatization is still necessary in hybrid retrieval pipelines. | fail→fail | 45,575 | 26,790 | -41% | 1 | 1 | 0% | 4,024 | 4,173 | +4% | 0 | 0 | — |
▸case-15 When merging top-100 results from dense vector search and top-100 results from sparse keyword search, our pipeline currently appends the two lists together into a 200-item array, creating duplicate entries for documents present in both lists. Draft a deduplication and candidate merging protocol. | pass→pass | 20,240 | 18,499 | -9% | 1 | 1 | 0% | 3,666 | 4,209 | +15% | 0 | 0 | — |
▸case-16 Our hardware diagnostic tool frequently searches for newly minted error codes like 'ERR_GPU_MEM_OVERFLOW_0x89'. Dense vector embeddings map this unknown token to a generic out-of-vocabulary background vector, causing irrelevant results. Draft an architectural solution using hybrid search to solve out-of-vocabulary (OOV) keyword failures. | fail→pass | 26,060 | 29,633 | +14% | 1 | 1 | 0% | 3,930 | 4,760 | +21% | 0 | 0 | — |
▸case-17 A company updates its product catalog with 10,000 new items every night. Developers plan to re-embed and rebuild both vector and keyword indexes from scratch every hour. Draft an incremental indexing pattern for hybrid search systems to reduce compute overhead. | fail→pass | 23,449 | 19,385 | -17% | 1 | 1 | 0% | 3,228 | 3,523 | +9% | 0 | 0 | — |
▸case-18 Our retail search app uses query synonym expansion on the BM25 query string. Developers want to also inject expanded synonym strings directly into the dense vector query embedding calculation. Draft an evaluation of query expansion strategies in hybrid search. | pass→pass | 25,416 | 22,621 | -11% | 1 | 1 | 0% | 3,965 | 3,908 | -1% | 0 | 0 | — |
▸case-19 We are tuning our vector database collection for large-scale embedding search. We want to adjust HNSW index parameters like ef_construction and M to balance indexing speed and search recall. Please provide an engineering guide for optimizing these HNSW parameters. | pass→pass | 35,017 | 19,855 | -43% | 1 | 1 | 0% | 5,190 | 3,745 | -28% | 0 | 0 | — |
▸case-20 Our relational database query log shows slow execution times on SELECT queries filtering orders by customer_id and created_at date range. Draft an SQL indexing blueprint using composite B-tree indexes to optimize these timestamp range queries. | pass→pass | 19,262 | 13,625 | -29% | 1 | 1 | 0% | 2,953 | 2,655 | -10% | 0 | 0 | — |
▸case-21 We need to fine-tune a SentenceTransformer embedding model on custom domain pairs using MultipleNegativesRankingLoss to improve vector representation quality. Draft a Python script and training loop standard for model fine-tuning. | pass→pass | 16,056 | 42,470 | +165% | 1 | 1 | 0% | 3,374 | 3,293 | -2% | 0 | 0 | — |
▸case-22 We are building an ingestion pipeline to parse large PDF contracts into text chunks for a vector database. We need recommendations on choosing optimal character length, token overlapping windows, and recursive text splitter boundary rules. Draft a technical document chunking guide. | pass→pass | 39,402 | 23,080 | -41% | 1 | 1 | 0% | 5,841 | 4,093 | -30% | 0 | 0 | — |