▸case-01 We are building a Python LLM application and need an open-source, local-server observability solution that uses OpenTelemetry to visualize traces without transmitting data to a third-party SaaS cloud. We are considering deploying a Prometheus and Jaeger collector stack. What specific Python tool and server should we use instead for LLM trace visualization? | fail→fail | 14,578 | 7,810 | -46% | 1 | 1 | 0% | 2,596 | 1,681 | -35% | 0 | 0 | — |
▸case-02 When configuring Python telemetry for an application making direct OpenAI API calls to send OpenTelemetry spans to a local Phoenix instance, we were going to install `opentelemetry-instrumentation-openai`. Which PyPI package is the standard auto-instrumentation package maintained for this ecosystem? | fail→fail | 6,492 | 4,109 | -37% | 1 | 1 | 0% | 1,278 | 1,015 | -21% | 0 | 0 | — |
▸case-03 We want our LLM application traces to conform to an open vendor-neutral telemetry standard rather than a custom proprietary log schema, so we can export them to standard collector backends. What tracing standard should our instrumentation adhere to? | fail→fail | 9,918 | 5,427 | -45% | 1 | 1 | 0% | 1,817 | 1,297 | -29% | 0 | 0 | — |
▸case-04 We have an extensive LangChain pipeline and are writing manual Python wrappers around every chain step to extract input and output payloads. How should we capture intermediate step spans automatically? | fail→pass | 14,289 | 7,674 | -46% | 1 | 1 | 0% | 2,536 | 1,648 | -35% | 0 | 0 | — |
▸case-05 Our RAG service uses LlamaIndex for document retrieval and query engine execution. We are writing custom event listener callbacks to track index lookup spans. What automated approach should we use instead? | fail→fail | 11,039 | 5,188 | -53% | 1 | 1 | 0% | 2,237 | 1,078 | -52% | 0 | 0 | — |
▸case-06 Our AI agent executes custom Python business logic loops outside of standard framework chains like LangChain or LlamaIndex. We plan to print JSON trace logs to stdout and parse them later. How should we record these custom execution steps into our trace server? | fail→fail | 19,971 | 13,558 | -32% | 1 | 1 | 0% | 3,804 | 2,831 | -26% | 0 | 0 | — |
▸case-07 We need to run automated quality and safety evaluations on model responses in our testing environment. We were planning to use regex pattern matching and exact text BLEU scores. What evaluation method should we set up for flexible semantic quality assessment? | fail→fail | 12,943 | 11,764 | -9% | 1 | 1 | 0% | 2,251 | 2,459 | +9% | 0 | 0 | — |
▸case-08 Our engineering team wants to visually inspect high-dimensional vector representations of context chunks to spot clusters and outliers locally. We were planning to dump vector arrays to CSV and write custom Matplotlib scripts. What capability should we use within our observability UI? | fail→fail | 10,722 | 8,365 | -22% | 1 | 1 | 0% | 1,770 | 1,742 | -2% | 0 | 0 | — |
▸case-09 As user query patterns evolve over time, the semantic distribution of production queries may diverge from our indexing dataset. We were planning to calculate daily average word counts as a proxy metric. What specific monitoring feature should we configure? | fail→fail | 14,009 | 9,281 | -34% | 1 | 1 | 0% | 2,077 | 1,630 | -22% | 0 | 0 | — |
▸case-10 To diagnose why our RAG pipeline provides unhelpful answers, we need to measure whether retrieved context passages contain information relevant to the user query. We were going to track HTTP status codes from our vector database. What feature should we implement? | fail→fail | 10,286 | 8,956 | -13% | 1 | 1 | 0% | 1,851 | 1,570 | -15% | 0 | 0 | — |
▸case-11 Before releasing a prompt modification to production, we want to run benchmark evaluations across multiple prompt versions against a fixed set of test inputs. We were going to save text outputs in separate Git branches and manually diff them. What feature manages these evaluation runs? | fail→fail | 11,001 | 7,134 | -35% | 1 | 1 | 0% | 1,845 | 1,531 | -17% | 0 | 0 | — |
▸case-12 In our high-volume production service, generating traces for every single request creates excessive network and storage load. We were thinking of adding time sleep delays to slow down traffic. What configuration option should we apply to capture a representative subset of traces? | fail→fail | 10,242 | 7,822 | -24% | 1 | 1 | 0% | 1,656 | 1,802 | +9% | 0 | 0 | — |
▸case-13 We need to configure our local Phoenix observability setup so that collected traces can be forwarded downstream to external telemetry endpoints. We were planning to write a script that periodically scrapes the UI endpoints. What configuration option governs trace destination routing? | fail→fail | 13,335 | 10,652 | -20% | 1 | 1 | 0% | 2,463 | 2,428 | -1% | 0 | 0 | — |
▸case-14 List the two primary Python packages required to run a local Phoenix server and auto-instrument OpenAI requests. We were planning to install `arize-sdk` and `openai-tracing`. | fail→fail | 5,996 | 3,691 | -38% | 1 | 1 | 0% | 938 | 863 | -8% | 0 | 0 | — |
▸case-15 How frequently should automated LLM-as-judge evaluation workflows be executed against application outputs? A developer suggested running evaluations only once during initial repository setup. | fail→pass | 13,149 | 11,921 | -9% | 1 | 1 | 0% | 2,181 | 2,202 | +1% | 0 | 0 | — |
▸case-16 What continuous operational practice should be maintained to ensure vector search relevance does not degrade as query topics shift over time? An engineer suggested re-embedding the entire corpus every night. | fail→pass | 12,587 | 9,847 | -22% | 1 | 1 | 0% | 2,071 | 1,841 | -11% | 0 | 0 | — |
▸case-17 When instrumenting a multi-step agent containing prompt templates, vector retrievers, and tool executions, should we instrument only the final LLM text generation step to save effort? | fail→pass | 13,565 | 10,816 | -20% | 1 | 1 | 0% | 2,226 | 2,189 | -2% | 0 | 0 | — |
▸case-18 To ensure our RAG architecture delivers high quality outputs, what specific aspect of document fetching should we continuously evaluate instead of relying solely on end-to-end API response latency? | fail→fail | 10,429 | 6,374 | -39% | 1 | 1 | 0% | 1,755 | 1,295 | -26% | 0 | 0 | — |
▸case-19 We want to spin up an interactive web interface on a developer laptop to inspect traces in real time while debugging local code. We were considering deploying a Docker container with Prometheus and Grafana. What component should we launch locally? | fail→pass | 9,865 | 5,219 | -47% | 1 | 1 | 0% | 1,842 | 1,128 | -39% | 0 | 0 | — |
▸case-20 We are fine-tuning Llama 3 8B using QLoRA for domain-specific text generation. What initial values should we choose for the learning rate and LoRA rank (r)? A colleague suggested using Phoenix trace sampling to optimize training hyperparameters. | fail→fail | 16,652 | 13,298 | -20% | 1 | 1 | 0% | 2,778 | 2,665 | -4% | 0 | 0 | — |
▸case-21 We are setting up a Qdrant vector database index with 10 million vectors. What recommended ranges should we set for m and ef_construct parameters to balance indexing speed and search recall? Someone suggested setting up Phoenix embedding drift monitoring to configure these HNSW parameters. | fail→fail | 15,813 | 15,163 | -4% | 1 | 1 | 0% | 2,876 | 3,036 | +6% | 0 | 0 | — |
▸case-22 Write a system prompt for a financial assistant chatbot that restricts answers to financial queries and formats tables in Markdown. Should we embed Phoenix evaluation metrics directly into the system prompt text? | fail→fail | 11,560 | 10,272 | -11% | 1 | 1 | 0% | 1,987 | 1,978 | -0% | 0 | 0 | — |