▸case-03 In an evaluation section for HumanEval, authors write: 'We computed pass@1 and pass@10 metrics for HumanEval python coding problems. Models generated 200 samples per prompt at temperature 0.8. Code execution was evaluated in an isolated Docker container.' Generate a comprehensive evaluation protocol extraction for HumanEval from this text. | fail→pass | 16,727 | 20,660 | +24% | 1 | 1 | 0% | 2,041 | 3,176 | +56% | 0 | 0 | — |
▸case-01 We published a paper evaluating DeepMath-7B on the GSM8K benchmark. Section 4 states: 'We evaluated our model on GSM8K using 8-shot chain-of-thought prompting. Answers were generated using nucleus sampling with top_p=0.95 and top_k=50. We evaluated accuracy by checking if the target integer appeared in the final sentence.' Extract a structured evaluation protocol specification for GSM8K from this text. Focus on identifying all evaluation protocol parameters and highlight any parameter gaps or missing settings. | pass→pass | 23,054 | 16,634 | -28% | 1 | 1 | 0% | 2,268 | 2,357 | +4% | 0 | 0 | — |
▸case-02 A paper reports MMLU evaluation results for ModelX: 'ModelX was tested on MMLU across all 57 subjects using 5-shot prompting with subject-specific prefix prompts. Exact match on option letters (A, B, C, D) was calculated via log-likelihood comparison across target completion tokens.' Please construct a structured breakdown of the evaluation protocol parameters used for MMLU, specifying parameter categories and completeness. | fail→fail | 15,382 | 16,480 | +7% | 1 | 1 | 0% | 1,886 | 2,279 | +21% | 0 | 0 | — |
▸case-04 An evaluation section states: 'For AlpacaEval 2.0, we used gpt-4-1106-preview as an automated judge to compare model outputs against GPT-4 completions. Outputs were evaluated in randomized pairwise order to compute win rates.' Please extract all evaluation protocol parameters into a structured specification and assess protocol completeness. | pass→pass | 11,721 | 13,133 | +12% | 1 | 1 | 0% | 2,156 | 1,765 | -18% | 0 | 0 | — |
▸case-05 A paper evaluates a coding agent on SWE-bench Lite: 'Our agent was deployed on SWE-bench Lite using Claude 3.5 Sonnet. The agent executed bash commands inside the provided SWE-bench environment using a maximum budget of 30 environment turns per task.' Extract the complete evaluation protocol parameters from this experimental description. | fail→pass | 10,997 | 16,721 | +52% | 1 | 1 | 0% | 707 | 2,299 | +225% | 0 | 0 | — |
▸case-06 A paper describes TruthfulQA evaluation: 'We evaluated TruthfulQA in two modes: TruthfulQA-MC1 choosing the single correct answer via log-likelihoods, and TruthfulQA-Generative using GPT-3 judge models to score truthfulness and informativeness.' Extract the evaluation protocol specifications across both modes and evaluate parameter completeness. | pass→pass | 16,577 | 14,320 | -14% | 1 | 1 | 0% | 1,961 | 2,860 | +46% | 0 | 0 | — |
▸case-07 Section 5 reads: 'We evaluated Llama-3-8B on MATH benchmark using standard greedy decoding and reported exact match accuracy.' Extract the evaluation protocol parameters for the MATH benchmark evaluation and assess the completeness of the parameters. | pass→pass | 12,440 | 17,526 | +41% | 1 | 1 | 0% | 1,377 | 2,180 | +58% | 0 | 0 | — |
▸case-08 A paper states: 'We evaluated reasoning capabilities on BBH (BIG-bench Hard) using 3-shot chain-of-thought prompting. Model predictions were parsed to extract the final answer line.' Provide a structured extraction of all evaluation protocol parameters and state which parameters are ambiguous or missing. | pass→pass | 14,670 | 18,061 | +23% | 1 | 1 | 0% | 1,577 | 2,548 | +62% | 0 | 0 | — |
▸case-09 In Section 3, authors state: 'Unlike the standard 5-shot MMLU setup, we evaluated Llama-2-70B on MMLU using 0-shot chain-of-thought prompting with the phrase Let us think step by step, extracting option letters from the generated reasoning text.' Extract the evaluation protocol parameters and compare this protocol against the official MMLU benchmark standard. | pass→pass | 15,046 | 20,701 | +38% | 1 | 1 | 0% | 1,832 | 3,226 | +76% | 0 | 0 | — |
▸case-10 An evaluation setup notes: 'For the MATH dataset evaluation, we implemented a custom LaTeX SymPy parser that converts equations into canonical symbolic forms before testing exact equivalence, rather than relying on raw string exact match.' Extract the evaluation protocol parameters from this setup. | fail→fail | 8,841 | 13,630 | +54% | 1 | 1 | 0% | 661 | 1,741 | +163% | 0 | 0 | — |
▸case-11 A paper reports: 'We evaluated Python code generation on MBPP using 3-shot prompting on the 427-problem sanitized split (MBPP-Sanitized). Unit test assertions were executed with a 3-second timeout per problem.' Extract all evaluation protocol parameters for MBPP. | pass→pass | 9,322 | 16,189 | +74% | 1 | 1 | 0% | 615 | 1,994 | +224% | 0 | 0 | — |
▸case-21 Section 2.2 describes model pre-training: 'We pre-trained Model-Z for 100k steps using AdamW optimizer with beta1=0.9, beta2=0.95, peak learning rate 3e-4 decayed via cosine schedule to 3e-5, weight decay 0.1, gradient clipping 1.0, and batch size 4M tokens.' Extract the pre-training hyperparameters. | pass→pass | 4,097 | 15,240 | +272% | 1 | 1 | 0% | 770 | 1,674 | +117% | 0 | 0 | — |
▸case-12 A paper evaluates RewardBench: 'We scored candidate response pairs on RewardBench using ArmoRM-Llama3-8B-v0.1. Preference predictions were determined by comparing scalar reward values, treating absolute scalar differences under 0.01 as ties.' Extract the evaluation protocol specification and parameter completeness. | pass→pass | 13,599 | 14,616 | +7% | 1 | 1 | 0% | 1,421 | 1,914 | +35% | 0 | 0 | — |
▸case-13 A paper describes Arena-Hard-Auto evaluation: 'We evaluated candidate models against GPT-4-0314 baseline responses on Arena-Hard v0.1 using Claude-3-Opus as the judge. Judgment prompts followed the default Arena-Hard pairwise comparison format.' Extract the structured evaluation protocol parameters. | fail→pass | 9,168 | 13,584 | +48% | 1 | 1 | 0% | 628 | 1,884 | +200% | 0 | 0 | — |
▸case-14 An experimental section states: 'HellaSwag was evaluated in 10-shot setting using per-token normalized log-likelihood across completion choices. Prompt contexts included trailing space whitespace separators before candidate endings.' Extract the evaluation protocol parameters for HellaSwag. | pass→pass | 8,489 | 16,557 | +95% | 1 | 1 | 0% | 614 | 2,095 | +241% | 0 | 0 | — |
▸case-15 A paper describes AgentBench evaluation: 'We benchmarked autonomous agents on AgentBench OS environment tasks. Each task execution had a hard timeout of 120 seconds and a maximum limit of 15 tool execution turns. Docker containers were reset to initial state between tasks.' Extract all evaluation protocol parameters. | fail→fail | 8,305 | 9,258 | +11% | 1 | 1 | 0% | 511 | 1,738 | +240% | 0 | 0 | — |
▸case-16 A paper states: 'For the DROP reading comprehension benchmark, we evaluated 1-shot performance using the standard F1 and Exact Match metrics after normalizing numbers to words and removing punctuation from model outputs.' Extract the evaluation protocol parameters. | fail→pass | 9,925 | 11,255 | +13% | 1 | 1 | 0% | 498 | 1,308 | +163% | 0 | 0 | — |
▸case-17 A paper evaluates MathVista: 'We evaluated GPT-4V on MathVista by resizing input images to 1024x1024 before feeding them with 0-shot CoT text prompts. Final numerical answers were parsed using regular expressions and scored via exact match.' Extract the evaluation protocol parameters. | fail→pass | 4,031 | 16,085 | +299% | 1 | 1 | 0% | 532 | 2,101 | +295% | 0 | 0 | — |
▸case-18 Section 4.1 states: 'For BIG-bench Hard (BBH), we used 3-shot chain-of-thought prompts formatted with explicit Answer: suffixes. Model predictions were extracted by matching text following Answer: using case-insensitive string matching.' Extract the evaluation protocol specification. | fail→pass | 8,160 | 14,512 | +78% | 1 | 1 | 0% | 584 | 1,670 | +186% | 0 | 0 | — |
▸case-19 A paper reports TriviaQA evaluation: 'We evaluated TriviaQA in the open-domain setting using a Contriever dense retriever to supply top-5 passages of 200 tokens each. Predictions were evaluated using length-normalized exact match against target aliases.' Extract the evaluation protocol specification. | fail→pass | 12,189 | 15,335 | +26% | 1 | 1 | 0% | 750 | 2,034 | +171% | 0 | 0 | — |
▸case-20 In Section 3 of our paper on Llama-Math, we write: 'Llama-Math is a 7B parameter decoder-only transformer with 32 layers, 32 attention heads, hidden dimension 4096, and SwiGLU activation functions trained with RoPE embeddings.' Please extract the model architecture parameters from this specification into a structured summary table. | pass→pass | 4,194 | 12,345 | +194% | 1 | 1 | 0% | 653 | 1,310 | +101% | 0 | 0 | — |
▸case-22 Table 3 presents evaluation results across benchmarks: GSM8K: 78.4%, MMLU: 65.2%, HumanEval: 52.1%, MATH: 34.8%. Summarize the benchmark performance numbers across these four datasets in a markdown summary table. | pass→pass | 11,566 | 3,481 | -70% | 1 | 1 | 0% | 462 | 691 | +50% | 0 | 0 | — |