▸case-01 We are instrumenting OpenAI and Anthropic API calls in our Python service. We want to attach an attribute to our spans specifying the provider vendor name. A developer suggested setting `llm.provider = 'openai'`. What standard semantic convention attribute key should be used instead? | fail→fail | 4,976 | 3,987 | -20% | 1 | 1 | 0% | 999 | 1,041 | +4% | 0 | 0 | — |
▸case-02 When instrumenting an LLM completion request span, an engineer logged the target model string under `request.model_name`. What is the standardized OpenTelemetry trace attribute key for the requested model name? | fail→fail | 4,219 | 3,190 | -24% | 1 | 1 | 0% | 769 | 953 | +24% | 0 | 0 | — |
▸case-03 Our team wants to capture the maximum generation tokens parameter in trace spans. Someone suggested using `llm.max_tokens`. What is the standard semantic attribute key for this request parameter? | fail→fail | 5,636 | 5,098 | -10% | 1 | 1 | 0% | 1,072 | 1,088 | +1% | 0 | 0 | — |
▸case-04 To track why an LLM finished generating tokens (such as hitting a stop sequence versus length limit), an engineer proposed adding `llm.stop_reason`. What standard semantic attribute key should be used on the span? | fail→pass | 6,631 | 4,219 | -36% | 1 | 1 | 0% | 1,022 | 1,086 | +6% | 0 | 0 | — |
▸case-05 For tracking input token usage in trace spans, a developer created `usage.input_tokens`. What is the standardized OpenTelemetry semantic attribute key for prompt token count? | fail→pass | 5,535 | 5,851 | +6% | 1 | 1 | 0% | 1,074 | 1,384 | +29% | 0 | 0 | — |
▸case-06 We are initializing telemetry in an AI microservice. What central OpenTelemetry SDK object manages tracer creation lifecycle and resource configuration? | fail→fail | 5,374 | 2,402 | -55% | 1 | 1 | 0% | 1,056 | 677 | -36% | 0 | 0 | — |
▸case-07 In our telemetry setup, which OpenTelemetry SDK component receives finished spans from active tracers and dispatches them to exporters? | fail→fail | 3,954 | 3,473 | -12% | 1 | 1 | 0% | 676 | 845 | +25% | 0 | 0 | — |
▸case-08 We are setting up distributed tracing for an agent pipeline and need to export telemetry over standard gRPC or HTTP to an OpenTelemetry Collector. Which trace exporter protocol extension should be configured? | fail→fail | 6,912 | 7,368 | +7% | 1 | 1 | 0% | 1,474 | 1,775 | +20% | 0 | 0 | — |
▸case-09 Our operations team hosts a Jaeger backend for trace visualization. Which specific trace exporter integration is supported alongside OTLP for sending span data directly to Jaeger? | fail→fail | 5,514 | 6,236 | +13% | 1 | 1 | 0% | 990 | 1,450 | +46% | 0 | 0 | — |
▸case-10 During local development of prompt templates, we want trace spans dumped directly to stdout without running a local collector container. Which built-in exporter component should be configured? | fail→fail | 4,968 | 3,778 | -24% | 1 | 1 | 0% | 872 | 943 | +8% | 0 | 0 | — |
▸case-11 We installed `opentelemetry-api` in our Python LLM application, but calling `trace.get_tracer_provider()` yields a non-operational tracer provider. Which core package must be installed to supply the full operational implementation? | fail→fail | 3,645 | 3,423 | -6% | 1 | 1 | 0% | 684 | 918 | +34% | 0 | 0 | — |
▸case-12 We are looking for an optional ecosystem library that provides automated instrumentation wrappers for AI framework calls (such as LangChain or LlamaIndex) emitting standard traces. Which library provides this capability? | fail→fail | 8,762 | 6,252 | -29% | 1 | 1 | 0% | 1,587 | 1,511 | -5% | 0 | 0 | — |
▸case-13 Our multi-agent architecture uses two independent HTTP services. Spans created in the downstream service appear as isolated trace roots. What OpenTelemetry mechanism must be enabled to pass trace identifiers in request headers? | fail→fail | 7,818 | 6,866 | -12% | 1 | 1 | 0% | 1,386 | 1,582 | +14% | 0 | 0 | — |
▸case-14 In a production high-throughput LLM gateway, sending every span synchronously to a remote collector increases latency. Which SpanProcessor implementation queues spans and exports them asynchronously on a background thread? | fail→fail | 3,047 | 4,949 | +62% | 1 | 1 | 0% | 506 | 1,031 | +104% | 0 | 0 | — |
▸case-15 Our production LLM application generates hundreds of spans per second, exceeding telemetry storage budgets. What OpenTelemetry configuration technique should be implemented to record only a representative percentage of traces? | fail→fail | 10,078 | 7,551 | -25% | 1 | 1 | 0% | 1,971 | 1,680 | -15% | 0 | 0 | — |
▸case-16 When instantiating our global tracer provider, we want service name, deployment environment, and version metadata attached to all spans automatically. Which OpenTelemetry object handles these application-level metadata key-value pairs? | fail→fail | 5,439 | 4,886 | -10% | 1 | 1 | 0% | 991 | 1,220 | +23% | 0 | 0 | — |
▸case-17 An agent task function attached a massive prompt text to a span attribute, causing high memory consumption. What OpenTelemetry SDK configuration feature caps attribute counts and maximum attribute length? | fail→fail | 5,993 | 6,448 | +8% | 1 | 1 | 0% | 1,162 | 1,458 | +25% | 0 | 0 | — |
▸case-18 When capturing token metrics for an LLM response span, a developer logged the output token count under `usage.output_tokens`. What standard OpenTelemetry semantic convention attribute key should be used for completion tokens? | pass→pass | 5,565 | 5,129 | -8% | 1 | 1 | 0% | 1,057 | 1,298 | +23% | 0 | 0 | — |
▸case-19 When instrumenting an LLM request span, an engineer set the temperature parameter under `llm.temperature`. What standardized OpenTelemetry attribute key should be used for request temperature? | fail→fail | 3,146 | 3,015 | -4% | 1 | 1 | 0% | 566 | 842 | +49% | 0 | 0 | — |
▸case-20 We are fine-tuning a LLaMA model on customer support transcripts using LoRA adapters. What learning rate and epoch count should we select to prevent overfitting? | fail→fail | 12,965 | 11,088 | -14% | 1 | 1 | 0% | 2,257 | 2,471 | +9% | 0 | 0 | — |
▸case-21 We are setting up a vector database index for a retrieval pipeline with 10 million embeddings. Should we pick HNSW or IVF index type to optimize recall versus search latency? | fail→fail | 14,301 | 12,954 | -9% | 1 | 1 | 0% | 2,586 | 2,548 | -1% | 0 | 0 | — |
▸case-22 We need to evaluate textual overlap between our summarization model output and ground truth references. Should we compute ROUGE-L or BLEU score for summarization assessment? | fail→fail | 11,038 | 9,703 | -12% | 1 | 1 | 0% | 1,945 | 2,079 | +7% | 0 | 0 | — |