▸case-13 A complex reasoning benchmark requires exploring multiple potential paths, evaluating intermediate decision steps, and backtracking when a path leads to a dead end. Linear Chain-of-Thought prompting fails because it follows a single unbranching sequence. Which prompting framework generalizes CoT by structuring intermediate steps as a tree search with lookahead and backtracking? | pass→pass | 5,120 | 8,037 | +57% | 1 | 1 | 0% | 837 | 2,696 | +222% | 0 | 0 | — |
▸case-01 We need to align an open-weights LLM using human preference pairs (chosen vs rejected responses). The research lead wants to avoid training and hosting a separate reward model or running complex PPO actor-critic loops. What alignment methodology directly optimizes human preference probabilities using a simple cross-entropy-style objective without a separate reward model? Explain the core mechanism. | pass→pass | 13,074 | 25,045 | +92% | 1 | 1 | 0% | 2,321 | 3,868 | +67% | 0 | 0 | — |
▸case-02 An engineering team wants to expand total parameter capacity to over 100 billion parameters while keeping the active compute cost and FLOPs per generated token comparable to a much smaller 14 billion parameter model. Which architectural mechanism routes each token to a subset of feed-forward networks? | pass→pass | 4,414 | 7,388 | +67% | 1 | 1 | 0% | 750 | 2,644 | +253% | 0 | 0 | — |
▸case-03 A high-throughput inference service runs a 70B LLM. To reduce generation latency without altering the target model weights or suffering any accuracy degradation, the lead proposes using a smaller 2B draft model to generate candidate tokens that the 70B model verifies in parallel. What inference optimization technique is this? | pass→pass | 3,299 | 5,883 | +78% | 1 | 1 | 0% | 542 | 2,363 | +336% | 0 | 0 | — |
▸case-04 A developer wants an LLM agent to alternate step-by-step between reasoning about research papers and invoking external search APIs to retrieve missing citations before reaching a final answer. What prompting paradigm explicitly interleaves reasoning traces with action execution? | pass→pass | 5,010 | 6,094 | +22% | 1 | 1 | 0% | 805 | 2,423 | +201% | 0 | 0 | — |
▸case-05 A reasoning agent evaluates complex math word problems using Chain-of-Thought prompting. To improve accuracy over single-path greedy decoding, the researcher wants to generate multiple independent reasoning paths at non-zero temperature and select the final answer based on the most frequent outcome. What technique is this? | pass→pass | 5,152 | 4,708 | -9% | 1 | 1 | 0% | 755 | 2,155 | +185% | 0 | 0 | — |
▸case-06 A generative vision team wants to generate 1024x1024 images using diffusion models but faces severe GPU VRAM bottlenecks when running hundreds of denoising steps directly in pixel space. What architectural formulation runs the diffusion process inside a low-dimensional compressed feature representation produced by a perceptual autoencoder? | pass→pass | 11,618 | 13,057 | +12% | 1 | 1 | 0% | 2,010 | 3,736 | +86% | 0 | 0 | — |
▸case-07 When generating images from text prompts using diffusion models, researchers need a mechanism to adjust the trade-off between text prompt adherence and sample diversity at inference time without training a separate discriminative classifier network. What technique jointly trains conditional and unconditional score estimators? | pass→pass | 6,172 | 7,851 | +27% | 1 | 1 | 0% | 1,134 | 2,868 | +153% | 0 | 0 | — |
▸case-08 An AI safety lab aims to align a model by having the model critique and revise its own draft responses based on a set of written principles, generating synthetic preference datasets for fine-tuning. What Anthropic-developed alignment method uses explicit principles for self-critique? | pass→pass | 3,604 | 3,815 | +6% | 1 | 1 | 0% | 557 | 2,025 | +264% | 0 | 0 | — |
▸case-09 A research team needs to select a standard benchmark to assess a new 70B parameter model's general knowledge and problem-solving capability across 57 diverse academic and professional subjects in a multiple-choice format. Which standard benchmark should they report? | pass→pass | 4,442 | 4,669 | +5% | 1 | 1 | 0% | 695 | 2,173 | +213% | 0 | 0 | — |
▸case-10 To evaluate a model's code generation capabilities, developers want to test whether generated Python functions successfully pass functional unit test suites rather than comparing n-gram overlap with ground truth solutions. Which benchmark measures pass@k functional correctness for Python docstrings? | pass→pass | 5,359 | 6,496 | +21% | 1 | 1 | 0% | 877 | 2,432 | +177% | 0 | 0 | — |
▸case-11 A vision lab trained a new diffusion model for synthetic satellite imagery. To quantitatively measure both the realism and feature-space diversity of generated images compared to real dataset images using an Inception-v3 backbone, which metric should they compute? | fail→pass | 7,334 | 8,488 | +16% | 1 | 1 | 0% | 1,324 | 2,845 | +115% | 0 | 0 | — |
▸case-12 An LLM system frequently makes multi-digit calculation errors when solving physics word problems using natural language Chain-of-Thought. To eliminate arithmetic mistakes, the system prompts the LLM to construct executable Python code blocks for computation and delegate execution to a Python interpreter. What technique is this? | pass→pass | 5,243 | 7,149 | +36% | 1 | 1 | 0% | 845 | 2,482 | +194% | 0 | 0 | — |
▸case-14 An LLM API service uses static batching, which forces short generation requests to wait until the longest output sequence in the batch finishes generating before memory is freed. What serving strategy dynamically iterates token-by-token, inserting new incoming requests and evicting completed sequences at each iteration step? | pass→pass | 5,436 | 5,551 | +2% | 1 | 1 | 0% | 797 | 2,300 | +189% | 0 | 0 | — |
▸case-15 A scientific organization wants their AI assistant to answer detailed research questions using internal document repositories updated hourly. To avoid frequent expensive model retraining and reduce factual hallucination, which technique grounds LLM answers by querying a vector store and injecting matching document snippets into the prompt context? | pass→pass | 5,369 | 5,834 | +9% | 1 | 1 | 0% | 736 | 2,312 | +214% | 0 | 0 | — |
▸case-16 An AI research lab wants to benchmark their model's ability to maintain context, handle follow-up edits, and follow multi-step user prompts across a two-turn conversation flow using an LLM-as-a-judge approach. What standard multi-turn benchmark evaluation set is designed for this scenario? | pass→pass | 7,733 | 9,427 | +22% | 1 | 1 | 0% | 1,321 | 3,005 | +127% | 0 | 0 | — |
▸case-17 A text-to-image research group needs an automated score to quantify how well the content of a generated image matches a textual prompt like "a brass telescope on a mahogany table". Which metric measures cosine similarity between text and image embeddings in a shared multimodal latent space? | pass→pass | 6,331 | 5,544 | -12% | 1 | 1 | 0% | 935 | 2,386 | +155% | 0 | 0 | — |
▸case-18 An engineer needs to deploy a 70B parameter model on a single 24GB GPU. In FP16 precision, the model weights take 140GB VRAM. To drastically shrink the VRAM footprint down to ~35GB or ~20GB while keeping model topology intact, which memory reduction technique converts high-precision float weights into 8-bit or 4-bit integer representations? | pass→pass | 6,997 | 10,707 | +53% | 1 | 1 | 0% | 1,242 | 3,334 | +168% | 0 | 0 | — |
▸case-19 In diffusion model theory, what mathematical concept describes estimating the vector gradient of the log probability density of the data distribution with respect to the input space, allowing the reverse process to navigate toward real data modes? | pass→pass | 7,621 | 10,714 | +41% | 1 | 1 | 0% | 1,207 | 3,395 | +181% | 0 | 0 | — |
▸case-20 Prior to public deployment, an AI safety team hires researchers to intentionally write adversarial inputs, jailbreak prompts, and edge-case instructions to discover security flaws and safety vulnerabilities in an LLM. What is this security testing process called? | pass→pass | 4,449 | 4,442 | -0% | 1 | 1 | 0% | 632 | 2,041 | +223% | 0 | 0 | — |
▸case-21 We are training a ResNet-50 model from scratch on a custom 10-class dataset of industrial gear defect photos. We need to decide whether to use CrossEntropyLoss or MSE Loss for multi-class single-label classification, and whether SGD with momentum or AdamW is typically standard for training CNN image classifiers. What are the recommended standard loss function and optimizer setups for this task? | pass→pass | 15,645 | 18,086 | +16% | 1 | 1 | 0% | 2,576 | 4,413 | +71% | 0 | 0 | — |
▸case-22 A telecommunications company wants to predict customer churn based on tabular data containing monthly billing amounts, contract duration, and call logs (50 columns, 500,000 rows). The data team asks whether they should use gradient boosted decision trees like XGBoost/LightGBM or a 100-layer Transformer for tabular binary classification. Which architecture is standard for structured tabular data? | pass→pass | 12,886 | 13,501 | +5% | 1 | 1 | 0% | 2,187 | 3,563 | +63% | 0 | 0 | — |
▸case-23 An archival project needs to perform automated keyphrase extraction on 1,000,000 scanned historical document transcriptions without using any neural networks or API calls. The team wants to rank terms based on how frequently they appear in a specific document relative to their frequency across the entire corpus. What classical statistical NLP algorithm should they implement? | pass→pass | 7,106 | 10,981 | +55% | 1 | 1 | 0% | 1,236 | 3,272 | +165% | 0 | 0 | — |