▸case-01 Create a technical proposal document for implementing an LLM-based summarization pipeline to shrink our massive input texts before they hit our main model. The proposal must include a section on what dependencies we need to install and how we will track the financial savings and evaluate the quality of the shrunken text. | fail→pass | 22,878 | 27,012 | +18% | 1 | 1 | 0% | 4,113 | 5,252 | +28% | 0 | 0 | — |
▸case-02 We are building a token-level reduction script for OpenAI prompts. List the exact Python package dependencies required for token counting and model architectures, assuming we aren't using any optional specialized compression libraries yet. | fail→pass | 7,234 | 5,938 | -18% | 1 | 1 | 0% | 1,248 | 1,143 | -8% | 0 | 0 | — |
▸case-03 We want to implement a specialized token-level compression technique that uses small language models to identify and remove redundant tokens. What optional dependency should we include in our requirements.txt to enable this specific technique? | fail→fail | 4,451 | 2,689 | -40% | 1 | 1 | 0% | 818 | 687 | -16% | 0 | 0 | — |
▸case-04 What specific token-level compression technique should we use to shrink prompts before sending them to GPT-4, assuming we want to use a small model to score token perplexity rather than just summarizing the text? | fail→fail | 15,750 | 10,926 | -31% | 1 | 1 | 0% | 2,777 | 2,298 | -17% | 0 | 0 | — |
▸case-05 We have a long conversation history that exceeds the context window. We want to use an LLM to condense the older messages into a shorter block of text. If we use a standard LangChain summarization chain that iteratively updates a running summary with each new document, which specific chain type should we use? | fail→fail | 5,024 | 7,525 | +50% | 1 | 1 | 0% | 769 | 1,550 | +102% | 0 | 0 | — |
▸case-06 We are passing a massive 100-page PDF to our agent, but only chapter 4 is relevant to the user's query. If we want to use a retrieval step to extract only the relevant sections before generation, what specific mathematical metric is most commonly used to measure the distance between the query embedding and the document chunk embeddings? | fail→fail | 5,920 | 5,318 | -10% | 1 | 1 | 0% | 884 | 1,303 | +47% | 0 | 0 | — |
▸case-07 We want to reduce prompt size by identifying and stripping out individual words that have low importance to the overall meaning. If we use a small causal language model to calculate the importance of each token, which specific metric, calculated as the exponentiated average negative log-likelihood, is typically used? | fail→fail | 4,295 | 5,207 | +21% | 1 | 1 | 0% | 828 | 1,158 | +40% | 0 | 0 | — |
▸case-08 We are building a RAG system and retrieving 50 documents, but that's too many to fit in the prompt. If we want to re-rank these documents to select only the top 5 most relevant ones, which specific type of model architecture is commonly used for this cross-encoding step? | fail→fail | 6,113 | 8,223 | +35% | 1 | 1 | 0% | 1,111 | 1,710 | +54% | 0 | 0 | — |
▸case-09 When configuring our prompt reduction pipeline using LLMLingua, we need to set a specific goal for how much smaller the output should be compared to the input. Which specific parameter in the LLMLingua API controls the target compression ratio or target token count? | fail→fail | 5,218 | 5,551 | +6% | 1 | 1 | 0% | 983 | 1,236 | +26% | 0 | 0 | — |
▸case-10 We are configuring our context reduction system. We want to ensure the compressed text doesn't lose critical information. If we use a BERT-based metric to evaluate the semantic similarity between the original and compressed text, which specific evaluation metric calculates precision, recall, and F1 score using pre-trained contextual embeddings? | fail→fail | 4,375 | 6,380 | +46% | 1 | 1 | 0% | 776 | 1,405 | +81% | 0 | 0 | — |
▸case-11 Our agent has a strict limit of 4000 tokens for the input context to save costs. If we are using the tiktoken library to enforce this limit for the GPT-3.5-turbo or GPT-4 models, which specific encoding standard must we load? | fail→fail | 4,316 | 4,637 | +7% | 1 | 1 | 0% | 726 | 1,155 | +59% | 0 | 0 | — |
▸case-12 We are setting up a pipeline that uses a smaller model to compress prompts for a larger model. If we choose to use a small, efficient model from the LLaMA 3 family specifically designed for fast inference on consumer hardware, which specific model size is the smallest official LLaMA 3 model available? | fail→fail | 5,508 | 5,723 | +4% | 1 | 1 | 0% | 962 | 1,431 | +49% | 0 | 0 | — |
▸case-13 To ensure our prompt reduction isn't degrading performance, we need to configure specific measurements to track the success of the pipeline. If we are evaluating a question-answering task, which specific metric measures the percentage of predictions that exactly match the reference answer string? | fail→fail | 3,931 | 4,085 | +4% | 1 | 1 | 0% | 615 | 942 | +53% | 0 | 0 | — |
▸case-14 When implementing prompt reduction, we want to measure the tradeoff between quality and compression. If we use the ROUGE metric to evaluate the overlap of n-grams between the original and compressed text, which specific ROUGE variant measures the longest common subsequence? | fail→fail | 3,747 | 3,579 | -4% | 1 | 1 | 0% | 581 | 831 | +43% | 0 | 0 | — |
▸case-15 Before deploying our context reduction system to production, we want to test it on a standard benchmark dataset designed for evaluating long-context language models. Which specific benchmark suite, introduced in 2023, is widely used for evaluating long-context understanding across multiple tasks? | fail→fail | 6,554 | 8,662 | +32% | 1 | 1 | 0% | 1,238 | 1,832 | +48% | 0 | 0 | — |
▸case-16 When deploying a token reduction strategy, we must establish the target size of the compressed text. If we are targeting the maximum context window of the standard, original GPT-4 model (not the turbo or 32k version), what is the exact token limit? | fail→fail | 2,770 | 3,057 | +10% | 1 | 1 | 0% | 476 | 827 | +74% | 0 | 0 | — |
▸case-17 After applying a reduction technique to our agent's context, we want to automatically validate the quality of the resulting output using another LLM as a judge. Which specific prompting framework, introduced by Microsoft, uses LLMs with chain-of-thought to evaluate generated text on a scale of 1 to 5? | fail→fail | 20,042 | 7,478 | -63% | 1 | 1 | 0% | 3,634 | 1,686 | -54% | 0 | 0 | — |
▸case-18 The primary business driver for shrinking our prompts is to reduce our OpenAI API bills. When we send a compressed prompt to the OpenAI API, which specific top-level field in the JSON response object contains the dictionary with the number of prompt and completion tokens billed for that request? | fail→fail | 3,273 | 3,068 | -6% | 1 | 1 | 0% | 630 | 856 | +36% | 0 | 0 | — |
▸case-19 We are building a security layer for our LLM application to prevent users from overriding system instructions. Which specific OWASP Top 10 for LLM vulnerability category covers this risk? | fail→fail | 3,964 | 4,695 | +18% | 1 | 1 | 0% | 758 | 949 | +25% | 0 | 0 | — |
▸case-20 Our agent generates JSON responses, but sometimes the LLM includes markdown formatting or conversational filler. Which specific OpenAI API parameter should we use to guarantee valid JSON output? | fail→fail | 6,585 | 5,151 | -22% | 1 | 1 | 0% | 1,308 | 1,260 | -4% | 0 | 0 | — |
▸case-21 We are building a RAG system and need to store 10 million document chunks. Which specific index type should we use in our vector database to balance search speed and recall for this scale? | fail→fail | 15,333 | 14,319 | -7% | 1 | 1 | 0% | 2,740 | 2,761 | +1% | 0 | 0 | — |
▸case-22 We are documenting the primary business processes that benefit from our new context reduction pipeline. If we are optimizing a customer support chatbot that uses a RAG architecture, which specific component of the RAG pipeline is responsible for fetching the context that will be compressed? | fail→fail | 5,182 | 4,720 | -9% | 1 | 1 | 0% | 862 | 917 | +6% | 0 | 0 | — |