▸case-01 We are building an HTTP endpoint in Python to serve token-by-token LLM completions to web frontends. The team currently returns a standard JSON response with application/json after collecting all generated tokens in memory, which creates high latency before the user sees any text. Configure a streaming response endpoint using FastAPI that pushes tokens incrementally as they are generated. | pass→pass | 11,846 | 23,863 | +101% | 1 | 1 | 0% | 2,294 | 4,048 | +76% | 0 | 0 | — |
▸case-02 We deployed a streaming inference endpoint behind Nginx. Right now, clients receive all response tokens at once in a single large batch at the end of generation instead of receiving them token-by-token. A developer suggests increasing proxy_buffer_size. Provide the correct Nginx location block configuration to fix token delivery delay for streaming HTTP endpoints. | pass→fail | 10,833 | 11,013 | +2% | 1 | 1 | 0% | 2,010 | 2,478 | +23% | 0 | 0 | — |
▸case-03 We are configuring a C++ model config file (config.pbtxt) in Triton Inference Server for an audio transcribing model that yields multiple output tensors over time for a single input request. Standard Triton models map one request to exactly one response. Configure the model transaction policy in config.pbtxt so Triton allows multiple responses per request. | pass→pass | 10,226 | 12,712 | +24% | 1 | 1 | 0% | 1,730 | 2,776 | +60% | 0 | 0 | — |
▸case-04 We are designing a .proto file for a real-time predictive service. The client sends one feature vector request, and the server continuously yields prediction update messages as new sensor chunks arrive. A developer wrote 'rpc StreamPredictions (PredictRequest) returns (PredictResponse);'. Update the RPC definition to enable server-side streaming. | pass→pass | 3,969 | 5,006 | +26% | 1 | 1 | 0% | 691 | 1,238 | +79% | 0 | 0 | — |
▸case-05 In a FastAPI application using Hugging Face Transformers, a developer uses TextIteratorStreamer inside an async endpoint by calling next(streamer) directly on the main event loop inside an async generator loop. This causes the entire web server to freeze and block concurrent incoming requests while model inference executes. Provide the corrected pattern for yielding tokens asynchronously without blocking the event loop. | pass→pass | 18,052 | 16,757 | -7% | 1 | 1 | 0% | 2,757 | 3,384 | +23% | 0 | 0 | — |
▸case-06 We need to deploy a 70B parameter LLM for streaming inference using vLLM on a node with 4 NVIDIA A100 GPUs. The base script initializes LLMEngine without specifying multi-GPU sharding, causing out-of-memory errors on GPU 0. Provide the initialization arguments for vLLM to parallelize the model tensor weights across all 4 GPUs. | pass→pass | 8,149 | 13,567 | +66% | 1 | 1 | 0% | 1,613 | 2,863 | +77% | 0 | 0 | — |
▸case-07 Our monitoring team needs to track latency metrics for a token streaming LLM service. They currently measure only total endpoint request duration (http_request_duration_seconds). However, this fails to capture initial user responsiveness or smooth generation cadence. Specify the two core latency metrics required specifically for evaluating streaming text generation performance. | pass→pass | 7,312 | 13,082 | +79% | 1 | 1 | 0% | 1,305 | 1,989 | +52% | 0 | 0 | — |
▸case-08 We are serving dynamic long-prompt LLM requests via vLLM streaming. Under high workload, short streaming requests experience huge initial delays because long prompt prefill processing blocks the engine iteration cycle. Configure the vLLM engine flags to split large prompt prefill steps into smaller chunks across iterations. | pass→pass | 13,847 | 13,859 | +0% | 1 | 1 | 0% | 2,071 | 2,434 | +18% | 0 | 0 | — |
▸case-09 We are setting up a WebSocket endpoint for LLM text generation streaming. The client application needs a clear, deterministic signal to know when the model has finished producing text tokens so it can re-enable the UI prompt submission button, rather than relying on socket closure. Provide the streaming protocol convention for marking the end of generation over the connection. | pass→pass | 15,449 | 17,478 | +13% | 1 | 1 | 0% | 2,363 | 3,721 | +57% | 0 | 0 | — |
▸case-10 We are deploying a text generation model on TorchServe. The team wrote a custom handler extending BaseHandler that returns a list of strings at the end of the postprocess method, but TorchServe sends the entire result in a single HTTP batch. Modify the handler output pattern to enable response streaming chunking in TorchServe. | fail→fail | 19,912 | 19,548 | -2% | 1 | 1 | 0% | 3,659 | 3,936 | +8% | 0 | 0 | — |
▸case-11 We are integrating real-time computer vision inference into an Apache Kafka consumer pipeline in Python using kafka-python. High input message spikes cause the consumer thread to fetch thousands of messages into RAM, leading to Out-Of-Memory crashes because model inference takes 50ms per frame. Provide the stream consumer control pattern to prevent memory saturation. | pass→pass | 25,385 | 32,361 | +27% | 1 | 1 | 0% | 4,841 | 6,396 | +32% | 0 | 0 | — |
▸case-12 When running a multi-GPU vLLM streaming inference container inside Docker using PyTorch distributed IPC, worker processes crash with 'Bus error (core dumped)' during high-concurrency streaming. Explain the required Docker run or compose parameter adjustment to fix memory crashes. | pass→pass | 11,024 | 12,095 | +10% | 1 | 1 | 0% | 1,977 | 2,569 | +30% | 0 | 0 | — |
▸case-13 We are writing a Python client script to stream request inputs and receive output tokens back from a Triton Inference Server using tritonclient.grpc. The client developer is calling client.infer() synchronously inside a loop. Provide the Triton gRPC client API call specifically designed for bidirectional streaming. | fail→fail | 7,723 | 7,749 | +0% | 1 | 1 | 0% | 1,408 | 1,709 | +21% | 0 | 0 | — |
▸case-14 A frontend developer is attempting to read an SSE streaming LLM endpoint using standard fetch() with await response.json(). This causes the frontend to wait until the request finishes before updating the DOM. Provide the JavaScript browser standard code construct designed for parsing SSE streams token by token. | fail→fail | 11,531 | 12,507 | +8% | 1 | 1 | 0% | 2,122 | 2,778 | +31% | 0 | 0 | — |
▸case-15 We are building an online real-time speech transcription pipeline using continuous 500ms audio buffer chunks fed into a Whisper speech-to-text model. Raw non-overlapping chunks cause words split at chunk boundaries to be misrecognized or dropped entirely. Provide the audio signal preprocessing pattern to fix boundary word loss during streaming transcription. | pass→pass | 19,592 | 22,506 | +15% | 1 | 1 | 0% | 3,379 | 4,431 | +31% | 0 | 0 | — |
▸case-16 A vLLM streaming inference service fails to start on container boot with 'RuntimeError: Engine failed to initialize CUDA context' because PyTorch default memory allocators compete with vLLM KV-cache pre-allocation on a shared GPU environment. Configure the vLLM engine parameter to reserve a specific safe fraction (such as 85%) of total GPU VRAM for model weights and KV cache. | pass→pass | 5,872 | 27,791 | +373% | 1 | 1 | 0% | 1,198 | 5,075 | +324% | 0 | 0 | — |
▸case-17 We are building a Text-to-Speech streaming HTTP service where audio samples must begin playing on the client before the full sentence audio is completely synthesized. The existing endpoint generates the complete WAV file in memory and sends a Response(content=wav_bytes, media_type='audio/wav'). Convert this to a streaming chunked endpoint yielding raw audio chunks. | pass→pass | 17,202 | 21,742 | +26% | 1 | 1 | 0% | 3,239 | 4,508 | +39% | 0 | 0 | — |
▸case-18 When an HTTP client disconnects or aborts a token-streaming request mid-generation, our Python FastAPI inference server continues spending GPU compute cycles completing the entire token generation loop in the background. Provide the request state monitoring pattern inside the streaming loop to abort model generation immediately upon client disconnect. | pass→pass | 16,244 | 19,738 | +22% | 1 | 1 | 0% | 2,755 | 3,857 | +40% | 0 | 0 | — |
▸case-19 We are writing a custom Python backend for Triton Inference Server that yields multiple output tensors per request in decoupled mode. The current implementation returns pb_utils.InferenceResponse at the end of execute(). What API calls on Triton's InferenceResponseSender object must be invoked for each incremental streaming payload and upon final completion? | pass→pass | 11,925 | 13,909 | +17% | 1 | 1 | 0% | 2,352 | 3,205 | +36% | 0 | 0 | — |
▸case-20 We are deploying Hugging Face Text Generation Inference (TGI) via Docker. Clients need to receive individual tokens as server-sent events using the OpenAI-compatible API path. Provide the request body JSON parameter that forces TGI to return an event stream instead of waiting for generation completion. | pass→pass | 5,343 | 4,580 | -14% | 1 | 1 | 0% | 761 | 1,191 | +57% | 0 | 0 | — |
▸case-21 We need a Python script using Apache Arrow to convert a 50GB dataset of Parquet files on disk into compressed Feather format for static offline batch training. Write a script that processes the Parquet files on disk. | pass→pass | 18,335 | 19,415 | +6% | 1 | 1 | 0% | 3,380 | 4,103 | +21% | 0 | 0 | — |
▸case-22 We are training a PyTorch model across 8 GPUs using torch.nn.parallel.DistributedDataParallel. Configure PyTorch TorchDistributed logger to aggregate and save training loss metrics to a local TensorBoard log file during epoch training iterations. | pass→pass | 16,260 | 18,087 | +11% | 1 | 1 | 0% | 3,143 | 4,153 | +32% | 0 | 0 | — |
▸case-23 We have an existing ResNet-50 .onnx model file saved on disk. Run offline INT8 static quantization using onnxruntime.quantization to produce an optimized resnet50_quantized.onnx file saved to local storage. | pass→pass | 16,801 | 19,970 | +19% | 1 | 1 | 0% | 3,459 | 4,785 | +38% | 0 | 0 | — |