▸case-14 During meta-analysis of object detection models on COCO 2017 minival, two candidate papers used non-standard custom IoU threshold metrics (e.g. mIoU@0.85) that cannot be aligned with standard mAP@50:95. A team member proposes omitting these papers entirely from the output JSON payload. How should incompatible or excluded paper entries be retained in the final output schema? | fail→pass | 13,968 | 6,013 | -57% | 1 | 1 | 0% | 2,402 | 1,367 | -43% | 0 | 0 | — |
▸case-01 I have gathered raw score tuples and experimental condition logs across several reinforcement learning papers. Please consolidate them into unified structured comparison tables. Output a JSON payload containing dataset and metric specific tables that report method performance, confidence bounds, run counts, evaluation comparability status, source papers, and SOTA highlights, alongside a separate log listing any entries that were excluded. | fail→pass | 35,186 | 16,917 | -52% | 1 | 1 | 0% | 5,895 | 2,828 | -52% | 0 | 0 | — |
▸case-02 Please take these extracted evaluation metrics and condition metadata objects from recent NLP studies and merge them into a unified performance overview. Format the response as JSON containing individual tables split by dataset and metric—with method rows capturing scores, interval estimates, iteration counts, source references, and best-performing baselines—and include a dedicated list detailing any omitted submissions and why. | fail→pass | 13,574 | 15,862 | +17% | 1 | 1 | 0% | 2,833 | 2,682 | -5% | 0 | 0 | — |
▸case-03 We need to merge our score extraction records and condition catalogs from computer vision literature into standardized benchmark summaries. Could you generate a JSON object with dataset-metric tables detailing method scores, confidence intervals, comparability flags, fairness notes, and SOTA summary statistics, as well as an array documenting entries left out of the primary comparison? | fail→pass | 39,398 | 14,176 | -64% | 1 | 1 | 0% | 4,925 | 2,339 | -53% | 0 | 0 | — |
▸case-04 We have an unparsed text excerpt from a Machine Learning paper describing ImageNet top-1 accuracy results: 'Our model achieved 84.2% accuracy on ImageNet-1k with 300 epochs of training using AdamW optimizer.' Parse this prose text and extract the raw metric score, training budget, and dataset name into key-value pairs. | pass→pass | 8,017 | 2,831 | -65% | 1 | 1 | 0% | 632 | 877 | +39% | 0 | 0 | — |
▸case-19 For speech recognition error rates on the Switchboard benchmark, an implementation team considers stringifying numeric values like "12.4" in the JSON table output to prevent IEEE 754 floating point rounding display artifacts. What data type should be used for the score field in method rows? | fail→pass | 8,952 | 3,925 | -56% | 1 | 1 | 0% | 1,437 | 1,012 | -30% | 0 | 0 | — |
▸case-05 Here is a list of experimental setup notes from five GPU benchmark reports, detailing hardware specs like '2x NVIDIA A100-80GB', 'batch size 64', and 'FP16 precision'. Standardize these hardware setup descriptions into a uniform condition catalog object listing compute resources and hyperparameter settings. | pass→fail | 14,163 | 20,359 | +44% | 1 | 1 | 0% | 2,929 | 3,848 | +31% | 0 | 0 | — |
▸case-06 We have a clean benchmark table represented in JSON and want to publish it in an IEEE conference paper. Convert this structured table data into LaTeX source code using the booktabs package with proper \toprule, \midrule, and \bottomrule formatting. | pass→pass | 15,834 | 18,490 | +17% | 1 | 1 | 0% | 3,135 | 3,926 | +25% | 0 | 0 | — |
▸case-07 We are aggregating BLEU score evaluation tuples for the WMT14 English-to-German translation benchmark from recent neural machine translation papers. Some baselines report 95% confidence bounds as standard deviation ranges while others report explicit lower and upper bounds. When generating the structured comparison JSON table, an engineer suggests representing confidence bounds as a nested object like {"min": 28.1, "max": 29.5}. How should confidence intervals be structured in the method performance rows? | fail→pass | 16,121 | 10,898 | -32% | 1 | 1 | 0% | 2,913 | 1,561 | -46% | 0 | 0 | — |
▸case-08 For the Cityscapes semantic segmentation benchmark, we are building a comparison table for mean Intersection over Union (mIoU) scores across vision transformers. We also have word error rate (WER) scores on LibriSpeech where lower values indicate superior performance. How does the target benchmark JSON table schema represent whether larger numeric values indicate better model performance? | fail→pass | 9,751 | 8,290 | -15% | 1 | 1 | 0% | 1,942 | 936 | -52% | 0 | 0 | — |
▸case-09 In an LLM reasoning evaluation study on the GSM8K dataset, several papers evaluated models using 8-shot chain-of-thought prompting while older models used 0-shot prompting. An analyst wants to attach unfair setup warnings as individual text strings inside each method row object. Where and in what structure should general fairness comparison warnings be placed in the comparison table schema? | fail→pass | 12,420 | 6,437 | -48% | 1 | 1 | 0% | 2,281 | 1,471 | -36% | 0 | 0 | — |
▸case-10 When compiling competitive results on the ImageNet-1K classification benchmark, we need to highlight the overall top-performing state-of-the-art result directly in the table summary metadata. A developer proposed creating a separate sub-object called sota_summary_statistics. How should state-of-the-art method details be represented at the table level? | fail→pass | 20,353 | 4,190 | -79% | 1 | 1 | 0% | 2,511 | 1,152 | -54% | 0 | 0 | — |
▸case-11 When comparing molecular graph generation models on the QM9 dataset, paper A reports average scores across 10 random seeds (num_runs: 10), but paper B only reports a single point estimate without disclosing the number of experimental runs. What value should be populated for the run count field when paper metadata does not specify run repetitions? | pass→pass | 11,851 | 4,839 | -59% | 1 | 1 | 0% | 1,809 | 1,125 | -38% | 0 | 0 | — |
▸case-12 In an offline reinforcement learning benchmark on D4RL HalfCheetah-Medium, Method X was trained on standard dataset v2, whereas Method Y was trained on non-standard filtered v0 data. When building the comparison table JSON, how should the schema explicitly flag whether Method Y's test setup is directly comparable to the standard evaluation protocol? | fail→pass | 18,845 | 7,678 | -59% | 1 | 1 | 0% | 2,555 | 1,740 | -32% | 0 | 0 | — |
▸case-13 While compiling protein structure prediction scores on the CASP14 dataset, Method Alpha used template-free modeling while Method Beta had access to experimental MSA sequence databases. We need to document this training asset discrepancy in Method Beta's table entry. Which row-level string field stores specific details regarding setup differences or dataset caveats? | fail→pass | 11,553 | 2,746 | -76% | 1 | 1 | 0% | 1,855 | 850 | -54% | 0 | 0 | — |
▸case-15 For paper submissions excluded from the primary tabular benchmark on the HumanEval python coding benchmark due to target data contamination, what exact key fields must be present in each item of the exclusion log array? | fail→pass | 38,542 | 2,883 | -93% | 1 | 1 | 0% | 1,583 | 831 | -48% | 0 | 0 | — |
▸case-16 In a medical image segmentation comparison table for the ISIC 2018 skin lesion dataset, we need to track publication provenance for each baseline algorithm. Should publication details be nested inside a bibliography object like {"citation": {"paper": "Smith et al.", "year": 2022}} or flattened into the row schema? | fail→pass | 13,291 | 5,620 | -58% | 1 | 1 | 0% | 2,352 | 1,253 | -47% | 0 | 0 | — |
▸case-17 When combining results from five papers on the GLUE Benchmark SST-2 sentiment task, different authors refer to the same architecture as 'BERT-Base', 'bert_base_uncased', and 'BERT (Devlin 2018)'. How are standardized method rows presented inside each dataset table object in the final schema? | fail→pass | 23,842 | 7,218 | -70% | 1 | 1 | 0% | 2,914 | 1,742 | -40% | 0 | 0 | — |
▸case-18 We are generating benchmark comparisons across three distinct evaluation datasets (SQuAD v2.0, NarrativeQA, and DROP) and two metrics (Exact Match and F1 score). Should the JSON response combine all datasets into a single monolithic flat table with dataset tags on each row, or separate them into distinct table structures? | fail→fail | 15,716 | 5,850 | -63% | 1 | 1 | 0% | 2,829 | 1,477 | -48% | 0 | 0 | — |
▸case-20 When creating a comparison table for zero-shot accuracy on the MMLU benchmark, how are the dataset title and evaluation metric identified in the table schema? | pass→pass | 29,590 | 4,785 | -84% | 1 | 1 | 0% | 2,140 | 1,188 | -44% | 0 | 0 | — |
▸case-21 We have raw score extraction tuples containing target method names, metric values, and confidence limits alongside condition catalog records detailing hardware and hyperparameter setups. What overall JSON structure is produced when consolidating these input records for benchmark reporting? | fail→pass | 12,824 | 6,185 | -52% | 1 | 1 | 0% | 2,392 | 1,534 | -36% | 0 | 0 | — |
▸case-22 When logging publication dates for baseline methods in an NLP model comparison table, an analyst suggests formatting publication years as string ranges like "2024-Q1" or string tokens like "2024". What primary primitive data type is required for the year attribute in method rows? | fail→pass | 4,800 | 2,495 | -48% | 1 | 1 | 0% | 789 | 775 | -2% | 0 | 0 | — |
▸case-23 In an audio synthesis benchmark on LibriTTS, a comparison table has multiple potential fairness warnings: one regarding unequal training batch sizes and another regarding fine-tuning compute budgets. How are multiple fairness notes represented at the table level? | fail→pass | 13,744 | 4,926 | -64% | 1 | 1 | 0% | 2,294 | 1,307 | -43% | 0 | 0 | — |
▸case-24 When aggregating paper results where a method reports a mean accuracy score but no variance or confidence interval bound was provided in the source text, how should the confidence_interval field be populated in the JSON output row? | fail→fail | 10,741 | 6,588 | -39% | 1 | 1 | 0% | 1,824 | 1,443 | -21% | 0 | 0 | — |
▸case-25 When consolidating extracted score records with condition catalog entries for a ResNet-50 baseline on CIFAR-100, where does specific non-standard evaluation context (such as custom data augmentation) belong in the final table row schema? | fail→pass | 17,561 | 5,749 | -67% | 1 | 1 | 0% | 2,952 | 1,305 | -56% | 0 | 0 | — |
▸case-26 During benchmark synthesis for a drug discovery task on Tox21, a baseline method is dropped because its evaluation split used a random 80/20 partition instead of the standard scaffold split. What field in the output schema documents why this method was removed from the main table? | fail→pass | 12,373 | 2,408 | -81% | 1 | 1 | 0% | 2,010 | 807 | -60% | 0 | 0 | — |
▸case-27 For perplexity evaluation on WikiText-103 where lower scores represent superior performance, what value should be assigned to the table-level metric directionality flag? | pass→pass | 6,959 | 2,545 | -63% | 1 | 1 | 0% | 1,096 | 655 | -40% | 0 | 0 | — |