▸case-01 I am building a local development environment for a new RAG pipeline and need to set up a vector database. Please provide a Python script that initializes the appropriate lightweight version of the database, builds a schema allowing flexible attributes, and sets up a basic collection. | fail→pass | 17,237 | 10,093 | -41% | 1 | 1 | 0% | 3,385 | 2,497 | -26% | 0 | 0 | — |
▸case-02 We are moving our vector search to production with millions of embeddings and need high recall with fast query times. Write a configuration guide detailing which index algorithm we should use, the recommended distance metric for text embeddings, and the specific tuning parameters we need to adjust to balance speed and accuracy. | fail→fail | 16,578 | 38,172 | +130% | 1 | 1 | 0% | 2,963 | 1,674 | -44% | 0 | 0 | — |
▸case-03 Our multi-tenant application needs to isolate data per customer while ingesting massive amounts of vectors at once. Could you generate a Python code snippet using the official SDK that demonstrates how to implement a data isolation strategy and perform a massive batch ingestion operation efficiently? | fail→pass | 14,860 | 13,829 | -7% | 1 | 1 | 0% | 2,923 | 2,891 | -1% | 0 | 0 | — |
▸case-04 We are deploying our vector search infrastructure to a Kubernetes cluster for a highly available, distributed setup. Which specific deployment mode of the database should we provision, and what is the primary Python dependency we need to add to our application's requirements.txt? | fail→fail | 9,394 | 2,675 | -72% | 1 | 1 | 0% | 1,852 | 769 | -58% | 0 | 0 | — |
▸case-05 I'm building a RAG application using Langchain and need to connect it to our vector store. What specific Python package should I install via pip to integrate Langchain with our chosen vector database? | fail→fail | 7,364 | 4,075 | -45% | 1 | 1 | 0% | 1,449 | 987 | -32% | 0 | 0 | — |
▸case-06 We have a massive dataset of 100 million vectors. We want to significantly reduce the memory footprint of our index while maintaining decent recall, and we are willing to use scalar quantization. Which specific index type should we configure for our collection? | fail→fail | 11,248 | 9,918 | -12% | 1 | 1 | 0% | 2,050 | 2,161 | +5% | 0 | 0 | — |
▸case-07 Our embedding model outputs normalized vectors. We need to configure the collection's metric type to measure similarity based on the angle between vectors. Which specific metric type parameter should we set in the index parameters? | fail→fail | 7,524 | 5,679 | -25% | 1 | 1 | 0% | 1,409 | 1,305 | -7% | 0 | 0 | — |
▸case-08 During testing, our vector search is returning fast results but missing some relevant documents. We are currently using an IVF_FLAT index. Which specific search parameter must we increase at query time to improve our recall rate? | fail→fail | 5,632 | 3,974 | -29% | 1 | 1 | 0% | 971 | 1,030 | +6% | 0 | 0 | — |
▸case-09 We decided to use an HNSW index for our vector collection to get the best latency-recall tradeoff. What are the two primary index-building parameters we need to configure in the index_params dictionary? | fail→fail | 5,105 | 4,552 | -11% | 1 | 1 | 0% | 882 | 1,071 | +21% | 0 | 0 | — |
▸case-10 We have a single collection storing documents for 50 different departments. Queries always filter by department, and we want to physically isolate the data on disk to speed up these queries. What specific database feature should we configure in the schema to achieve this? | fail→fail | 10,558 | 6,366 | -40% | 1 | 1 | 0% | 1,778 | 1,528 | -14% | 0 | 0 | — |
▸case-11 In our distributed vector database setup, we want to isolate compute resources for our critical 'user_search' collection so its performance isn't degraded by heavy background batch jobs. What specific configuration option should we apply to this collection? | fail→fail | 10,045 | 5,138 | -49% | 1 | 1 | 0% | 1,657 | 1,147 | -31% | 0 | 0 | — |
▸case-12 Our application ingests JSON documents where the metadata fields vary wildly between records. We cannot define all columns upfront in the schema. How should we configure the CollectionSchema object to handle these unpredictable fields? | fail→fail | 12,240 | 8,032 | -34% | 1 | 1 | 0% | 2,345 | 1,904 | -19% | 0 | 0 | — |
▸case-13 We are setting up a staging environment on a single EC2 instance. We want a full server deployment running as a daemon, not an embedded library, but we don't have Kubernetes available. Which deployment mode should we use? | fail→fail | 11,205 | 4,860 | -57% | 1 | 1 | 0% | 1,912 | 1,159 | -39% | 0 | 0 | — |
▸case-14 We have NVIDIA GPUs available on our vector database nodes and want to leverage them to speed up index building and search operations. What specific type of index should we configure to utilize this hardware? | fail→fail | 11,599 | 10,531 | -9% | 1 | 1 | 0% | 1,938 | 2,253 | +16% | 0 | 0 | — |
▸case-15 Provide a Python snippet to initialize a Langchain vector store using our standard vector database. The database is running locally on port 19530. Assume the embeddings object is already created. | fail→pass | 5,149 | 4,071 | -21% | 1 | 1 | 0% | 964 | 1,092 | +13% | 0 | 0 | — |
▸case-16 We have 500GB of vector data stored in Parquet files on S3. We need to load this into our vector database as quickly as possible without overwhelming the standard insert API. What specific operation or API should we use? | fail→fail | 15,723 | 7,962 | -49% | 1 | 1 | 0% | 2,381 | 1,765 | -26% | 0 | 0 | — |
▸case-17 We have a small dataset of 100,000 vectors and want exact distance calculations without any quantization or compression, but we still want to use an inverted file index to speed up search compared to brute force. Which index type should we choose? | fail→fail | 8,803 | 6,427 | -27% | 1 | 1 | 0% | 1,534 | 1,536 | +0% | 0 | 0 | — |
▸case-18 Our embedding model requires calculating the straight-line Euclidean distance between vectors to determine similarity. Which metric type should we configure in the collection's index parameters? | fail→fail | 4,479 | 2,545 | -43% | 1 | 1 | 0% | 838 | 759 | -9% | 0 | 0 | — |
▸case-19 We are building a financial application using our vector database where we need to guarantee that a search immediately reflects the most recent insertions, even if it slows down the query. Which specific consistency level should we configure for the search? | fail→fail | 7,154 | 5,526 | -23% | 1 | 1 | 0% | 1,189 | 1,188 | -0% | 0 | 0 | — |
▸case-20 We need to implement a traditional full-text search engine for our blog posts using BM25 scoring. The system must support stemming, stop words, and exact phrase matching. If we choose the most popular Java-based distributed search engine built on Apache Lucene, what is the exact REST API endpoint path used to execute a search query? | fail→fail | 3,617 | 4,344 | +20% | 1 | 1 | 0% | 646 | 1,149 | +78% | 0 | 0 | — |
▸case-21 We need to run a lightweight, in-memory only approximate nearest neighbor search on a small dataset of 10,000 vectors directly within our Python process using a pure algorithm library, not a database system. We want to use the most popular library developed by Facebook AI Research. What is the exact Python class used in this library to create an exact L2 distance index? | fail→fail | 3,110 | 3,206 | +3% | 1 | 1 | 0% | 626 | 929 | +48% | 0 | 0 | — |
▸case-22 We need to store massive multi-dimensional arrays of scientific data for a machine learning training pipeline. The data will be accessed in large contiguous blocks by PyTorch. We do not need similarity search, just efficient chunked storage on disk. If we choose the Hierarchical Data Format version 5, what is the exact Python package we must pip install to interact with it? | fail→fail | 3,320 | 3,734 | +12% | 1 | 1 | 0% | 584 | 924 | +58% | 0 | 0 | — |