▸case-01 I need to evaluate the inference performance of our new Llama 3 deployment. Please generate a comprehensive testing script or plan that measures latency and throughput. Format the output as a step-by-step guide with the necessary command-line arguments. | pass→pass | 19,959 | 16,952 | -15% | 1 | 1 | 0% | 4,099 | 3,713 | -9% | 0 | 0 | — |
▸case-02 We are trying to decide between A100 and H100 GPUs for our inference server. Can you provide a performance testing methodology to benchmark token generation rates? I want the final output to be a structured markdown table template for recording the metrics, along with the setup instructions. | pass→pass | 16,576 | 18,194 | +10% | 1 | 1 | 0% | 3,376 | 3,224 | -5% | 0 | 0 | — |
▸case-03 We want to test our Mistral 7B inference endpoint with realistic traffic patterns rather than just fixed-length strings. What dataset should we use for the benchmark run and how do we pass it to the testing script? | pass→pass | 16,161 | 15,026 | -7% | 1 | 1 | 0% | 3,072 | 2,908 | -5% | 0 | 0 | — |
▸case-04 I'm writing a custom Python script to measure the latency of our TGI endpoint. I plan to just loop 100 requests and average the time. Is there anything I should do before starting the measurement loop to ensure accurate results? | pass→pass | 16,624 | 14,552 | -12% | 1 | 1 | 0% | 2,611 | 2,664 | +2% | 0 | 0 | — |
▸case-05 We need to establish a baseline for our inference engine using synthetic data. We are going to test with random text. What parameters must we strictly control in the synthetic benchmark to ensure reproducible token generation metrics? | pass→pass | 15,364 | 14,935 | -3% | 1 | 1 | 0% | 2,670 | 2,358 | -12% | 0 | 0 | — |
▸case-06 When benchmarking our Llama 3 70B model on multiple GPUs, we are seeing out-of-memory errors at high concurrency. We are using the default engine arguments. What parameter should we tune during the benchmark to maximize throughput without OOMing? | pass→pass | 13,883 | 8,307 | -40% | 1 | 1 | 0% | 2,535 | 1,569 | -38% | 0 | 0 | — |
▸case-07 We are fine-tuning a Llama 3 8B model using LoRA on a single node with 4x A100s. We want to measure the training performance. What metrics should we track to ensure our GPU utilization is optimal during the backward pass? | pass→pass | 15,620 | 17,756 | +14% | 1 | 1 | 0% | 2,916 | 3,041 | +4% | 0 | 0 | — |
▸case-08 We deployed our new Mixtral 8x7B model and need to benchmark its performance on reasoning tasks. We want to compare it against GPT-4. What framework should we use to run these benchmarks? | pass→pass | 14,275 | 11,708 | -18% | 1 | 1 | 0% | 2,607 | 2,164 | -17% | 0 | 0 | — |
▸case-09 Our users in Europe are reporting slow responses from our US-East inference server. We need to benchmark the network performance between the client and the load balancer. What tool should we use? | pass→pass | 14,988 | 10,267 | -31% | 1 | 1 | 0% | 2,231 | 1,854 | -17% | 0 | 0 | — |
▸case-10 I'm trying to run the throughput benchmark script on a custom model loaded from HuggingFace, but it's failing to load the tokenizer and model architecture. The model requires custom modeling files. What flag am I missing in the benchmark command? | pass→pass | 6,200 | 5,163 | -17% | 1 | 1 | 0% | 948 | 837 | -12% | 0 | 0 | — |
▸case-11 We are load testing our Text Generation Inference (TGI) container using a standard HTTP load generator like hey or wrk. We want to see how it handles 50 simultaneous users. What metric exposed by TGI's /metrics endpoint is most critical to monitor for queue buildup? | pass→pass | 8,684 | 6,341 | -27% | 1 | 1 | 0% | 1,560 | 1,172 | -25% | 0 | 0 | — |
▸case-12 We are comparing an older static batching inference server with a new one that supports continuous batching. To properly demonstrate the advantage of continuous batching in our benchmark, what characteristic must our test requests have? | pass→fail | 8,972 | 7,421 | -17% | 1 | 1 | 0% | 1,510 | 1,408 | -7% | 0 | 0 | — |
▸case-13 We are benchmarking a model that was quantized to FP8, but the benchmark script seems to be running it in FP16, resulting in lower than expected throughput. What argument must be passed to the benchmark script to ensure it uses the correct precision? | pass→pass | 8,383 | 7,545 | -10% | 1 | 1 | 0% | 1,647 | 1,448 | -12% | 0 | 0 | — |
▸case-14 We have an 8x H100 node and want to benchmark a 70B model. The model won't fit on a single GPU. How do we configure the benchmark script to distribute the model across all 8 GPUs? | pass→pass | 13,078 | 14,142 | +8% | 1 | 1 | 0% | 2,731 | 2,757 | +1% | 0 | 0 | — |
▸case-15 Our application involves sending very large, identical system prompts with every request, followed by a small user query. We want to benchmark the performance improvement of caching these system prompts. What feature flag should we enable during the benchmark? | pass→pass | 9,035 | 5,891 | -35% | 1 | 1 | 0% | 1,662 | 1,133 | -32% | 0 | 0 | — |
▸case-16 For our streaming chatbot application, users complain if the text generation feels 'stuttery', even if the overall time is fast. Besides Time To First Token, what specific streaming metric must we measure during our load test to quantify this stutter? | pass→pass | 14,196 | 7,230 | -49% | 1 | 1 | 0% | 2,172 | 1,412 | -35% | 0 | 0 | — |
▸case-17 During our latency benchmarking of a small 7B model on vLLM, we noticed that CUDA graph capture takes a long time at startup and we want to test the performance without CUDA graphs to isolate the overhead. What flag disables CUDA graphs in the benchmark? | pass→pass | 4,400 | 3,810 | -13% | 1 | 1 | 0% | 906 | 684 | -25% | 0 | 0 | — |
▸case-18 We are running a load test from a client machine in a different AWS region than our inference server. To evaluate the engine's pure performance, we are currently recording the round-trip time reported by our client-side load generator. What specific Prometheus endpoint on the inference server should we scrape instead to eliminate network latency from our measurements? | pass→pass | 8,299 | 9,450 | +14% | 1 | 1 | 0% | 1,643 | 1,956 | +19% | 0 | 0 | — |
▸case-19 We are running a throughput benchmark and want to artificially constrain the inference engine to only process 16 requests concurrently, regardless of how much GPU memory is available. What argument controls this? | pass→pass | 7,299 | 5,027 | -31% | 1 | 1 | 0% | 1,485 | 852 | -43% | 0 | 0 | — |
▸case-20 We are running multiple iterations of the throughput benchmark to compare different engine versions. We want to ensure that the exact same random requests are sampled from the dataset in every run. What parameter must we set? | pass→pass | 3,541 | 3,502 | -1% | 1 | 1 | 0% | 643 | 513 | -20% | 0 | 0 | — |
▸case-21 Our benchmark nodes don't have internet access, but we have the model weights stored on a shared NFS drive. We are passing the local path to the script, but it still hangs trying to reach huggingface.co. What environment variable must we set to 1 to force the library to skip network checks? | pass→pass | 4,641 | 3,347 | -28% | 1 | 1 | 0% | 830 | 658 | -21% | 0 | 0 | — |
▸case-22 We want to benchmark an AWQ-quantized model. The benchmark script is failing to recognize the quantization method automatically. What flag do we need to provide to explicitly state the quantization format? | pass→pass | 7,547 | 4,130 | -45% | 1 | 1 | 0% | 1,425 | 772 | -46% | 0 | 0 | — |