▸case-01 We are evaluating machine translation SOTA models scoring around 42-45 BLEU-4. A teammate suggests using BLEU-4 alone to evaluate fluent generation and wants to know why top models seem stuck in a narrow score range. Provide a structured decomposition of BLEU-4, analyzing its constituent signal components, reward/penalty mechanisms, ceiling effects at SOTA, and gaming pathologies. | pass→pass | 33,605 | 32,085 | -5% | 1 | 1 | 0% | 4,980 | 3,582 | -28% | 0 | 0 | — |
▸case-02 In evaluating code LLMs using Pass@k (where k=1 or k=10), a developer assumes Pass@1 is just accuracy over generated samples. Decompose the Pass@k metric formula into its underlying mathematical signals, detailing what it rewards or penalizes, its score distribution floor/ceiling behavior, and potential pathologies when models generate unparseable code vs failing edge tests. | fail→fail | 36,978 | 33,880 | -8% | 1 | 1 | 0% | 5,290 | 5,315 | +0% | 0 | 0 | — |
▸case-03 Our summarization benchmark uses ROUGE-L to rank top summarization models, but models generating long repetitive text still achieve high scores. Decompose the ROUGE-L metric into its constituent mathematical components, explain its reward drivers, analyze its ceiling effects, and explain the repetition gaming pathology. | pass→pass | 50,536 | 43,191 | -15% | 1 | 1 | 0% | 5,532 | 7,409 | +34% | 0 | 0 | — |
▸case-04 An NER evaluation report presents a single micro-F1 score of 0.92 across 20 entity types. The lead engineer wants to understand why rare entity types are ignored by high-scoring models. Perform a structured decomposition of the F1 metric into its signal components, analyzing what precision and recall reward/penalize, ceiling effects on frequent classes, and floor limitations on rare classes. | pass→pass | 37,537 | 21,075 | -44% | 1 | 1 | 0% | 5,234 | 4,110 | -21% | 0 | 0 | — |
▸case-05 An LLM leaderboard uses the Bradley-Terry based ELO rating system from pairwise human preferences. A product manager thinks ELO is an absolute score of reasoning quality. Decompose the ELO metric into its constituent mathematical signals, state what pairwise win probabilities reward and penalize, analyze ceiling effects when models saturate the voter pool, and detail gaming pathologies like verbosity bias. | pass→pass | 33,894 | 43,245 | +28% | 1 | 1 | 0% | 4,477 | 5,159 | +15% | 0 | 0 | — |
▸case-06 Evaluating reasoning models on GSM8K using Exact Match (EM). High-capability models frequently fail EM because of minor formatting variations (e.g. '$10' vs '10 dollars'). Provide a structured metric decomposition of Exact Match, analyzing its binary signal, rewards/penalties, ceiling/floor limits, and sensitivity pathologies. | pass→pass | 31,418 | 24,309 | -23% | 1 | 1 | 0% | 4,163 | 3,439 | -17% | 0 | 0 | — |
▸case-07 An evaluation pipeline uses perplexity on a test corpus to rank foundation models, but models trained on web scrapes containing test set contamination show unrealistically low perplexity. Decompose perplexity into its mathematical signal components, what it rewards, its score bounds, and data contamination pathologies. | pass→pass | 47,721 | 34,579 | -28% | 1 | 1 | 0% | 4,160 | 3,240 | -22% | 0 | 0 | — |
▸case-08 A semantic evaluation pipeline uses BERTScore with Cosine Similarity of contextual embeddings. Developers notice that highly fluent hallucinations still score over 0.85. Decompose BERTScore into its constituent signals, state its reward/penalty mechanisms, and analyze its high-baseline ceiling pathology. | pass→pass | 40,423 | 29,847 | -26% | 1 | 1 | 0% | 4,708 | 4,469 | -5% | 0 | 0 | — |
▸case-09 Evaluation of image captioning systems uses METEOR. The team wants to understand why METEOR ranks outputs differently than BLEU. Decompose METEOR into its mathematical signal components, reward/penalty mechanics, and its discriminative power. | pass→pass | 42,000 | 29,034 | -31% | 1 | 1 | 0% | 6,432 | 3,290 | -49% | 0 | 0 | — |
▸case-10 A search engine evaluation uses nDCG@10 to grade neural retrieval models. The team observes that swapping ranks 8 and 9 barely moves the score, whereas missing rank 1 tanks it. Decompose nDCG@K into its constituent signals, detail its position discount and relevance gain rewards, and analyze score sensitivity at top vs bottom ranks. | pass→pass | 43,958 | 61,031 | +39% | 1 | 1 | 0% | 4,481 | 4,288 | -4% | 0 | 0 | — |
▸case-11 An automated LLM judge computes head-to-head Win Rate between Model A and Model B on 500 prompts. The product lead notes Model A wins 70% of the time, but team suspects position bias and self-enhancement bias. Perform a metric decomposition of Win Rate, analyzing its constituent signals, implicit weights, ceiling effects, and LLM-as-a-Judge pathologies. | fail→fail | 57,918 | 105,153 | +82% | 1 | 1 | 0% | 8,022 | 4,305 | -46% | 0 | 0 | — |
▸case-12 A safety moderation model is evaluated using AUROC on an imbalanced dataset where bad content is 0.1% of traffic. The AUROC is 0.98, but the deployment team reports massive false positive cascades. Decompose AUROC into its constituent signal components, reward/penalty structure, and analyze why it exhibits severe ceiling distortion under extreme class imbalance. | pass→pass | 33,402 | 20,378 | -39% | 1 | 1 | 0% | 5,277 | 2,864 | -46% | 0 | 0 | — |
▸case-13 An AI evaluation team measures human annotation reliability using Cohen's Kappa across two annotators labeling binary sentiment. Although observed agreement is 90%, Cohen's Kappa drops to 0.20 due to extreme class prevalence. Decompose Cohen's Kappa into its underlying signal components, detail what it rewards/penalizes, and analyze the prevalence paradox pathology. | pass→pass | 32,860 | 25,534 | -22% | 1 | 1 | 0% | 5,614 | 4,333 | -23% | 0 | 0 | — |
▸case-14 A time-series forecasting model for inventory demand is evaluated using MAPE. When actual demand is near zero, MAPE shoots to infinity or produces extreme outliers. Decompose MAPE into its mathematical formula signals, state what it rewards and penalizes, and analyze its division-by-zero pathology and asymmetric penalty. | pass→pass | 25,929 | 20,911 | -19% | 1 | 1 | 0% | 3,968 | 3,135 | -21% | 0 | 0 | — |
▸case-15 An open-ended text generation benchmark uses MAUVE to compare model outputs with human text distributions. Decompose MAUVE into its underlying statistical signals, detail how trade-off curves measure divergence, explain what it rewards/penalizes, and analyze its sensitivity to vector quantization hyperparameters. | fail→pass | 44,391 | 33,557 | -24% | 1 | 1 | 0% | 8,247 | 5,491 | -33% | 0 | 0 | — |
▸case-16 Speech-to-text models are evaluated using Word Error Rate (WER), which is based on normalized Levenshtein distance. Decompose WER into its constituent insertion, deletion, and substitution error signals, explain its reward/penalty weighting, and analyze the pathology where WER exceeds 100%. | pass→pass | 24,312 | 16,104 | -34% | 1 | 1 | 0% | 4,665 | 2,978 | -36% | 0 | 0 | — |
▸case-17 A medical AI diagnostic system evaluates predicted class probabilities using the Brier Score. Decompose the Brier Score into its mathematical signals, state what it rewards/penalizes, and analyze its ceiling/floor behavior for rare events. | pass→pass | 25,906 | 20,344 | -21% | 1 | 1 | 0% | 3,942 | 3,876 | -2% | 0 | 0 | — |
▸case-18 An MT benchmark uses ChrF instead of BLEU to evaluate morphologically rich languages like Finnish or Turkish. Decompose ChrF into its mathematical signal components, explain how character n-gram precision and recall function, and contrast its granularity and ceiling behavior against word-level metrics. | pass→pass | 31,947 | 29,068 | -9% | 1 | 1 | 0% | 5,225 | 4,560 | -13% | 0 | 0 | — |
▸case-19 An enterprise search engine measures retrieval performance using Mean Reciprocal Rank (MRR). Decompose MRR into its mathematical constituent signal, state what rank position rewards/penalizes, analyze its floor and ceiling effects, and explain why it fails for multi-answer retrieval scenarios. | pass→pass | 24,376 | 20,542 | -16% | 1 | 1 | 0% | 3,570 | 3,176 | -11% | 0 | 0 | — |
▸case-20 An e-commerce recommendation system uses Precision@10 and Recall@10 to rank candidate items. Decompose Precision@K and Recall@K into their constituent signals, contrast their reward structures, and analyze slate size floor/ceiling effects when users only engage with 1 or 2 items. | fail→fail | 36,371 | 29,265 | -20% | 1 | 1 | 0% | 5,817 | 5,166 | -11% | 0 | 0 | — |
▸case-21 We are training a new transformer language model from scratch and want to design a custom loss function that combines focal loss with cross-entropy to handle rare tokens during backpropagation. Write the PyTorch code for this custom training loss module and derive its gradients for parameter updates. | pass→pass | 33,911 | 43,143 | +27% | 1 | 1 | 0% | 6,411 | 8,516 | +33% | 0 | 0 | — |
▸case-22 We ran Model A and Model B on 1,000 benchmark test samples and obtained mean scores of 0.742 and 0.758 respectively. Perform a paired bootstrap resampling test with 10,000 iterations in Python to calculate the p-value and 95% confidence interval for the difference in means to check statistical significance. | pass→pass | 17,874 | 33,164 | +86% | 1 | 1 | 0% | 3,879 | 5,836 | +50% | 0 | 0 | — |
▸case-23 We need to hire human annotators to grade model outputs for medical accuracy. Draft a 5-point Likert rating scale rubric with clear qualitative definitions for each score level (1 = completely inaccurate, 5 = fully accurate) and instructions for resolving edge cases. | pass→pass | 25,284 | 23,305 | -8% | 1 | 1 | 0% | 3,172 | 2,998 | -5% | 0 | 0 | — |