▸case-01 I finished running my model training script once, but I need to make sure the performance metrics are stable. Could you run the evaluation across several different random seeds and tell me if the results hold up consistently? | fail→fail | 6,969 | 11,556 | +66% | 1 | 1 | 0% | 877 | 489 | -44% | 0 | 0 | — |
▸case-02 Here is the benchmark pipeline for our new algorithm. Please execute it multiple times using varied seed values and provide a breakdown of how reliable and steady the output metrics are across runs. | fail→fail | 3,462 | 15,343 | +343% | 1 | 1 | 0% | 481 | 249 | -48% | 0 | 0 | — |
▸case-03 We published initial findings for this experiment, but reviewers want confirmation of statistical stability. Can you rerun the code under different initialization seeds and evaluate whether the findings are robust? | fail→fail | 10,491 | 16,306 | +55% | 1 | 1 | 0% | 775 | 219 | -72% | 0 | 0 | — |
▸case-04 I want to optimize the hyperparameters (learning rate, batch size, dropout) of my ResNet50 model using Optuna on a single validation set to maximize top-1 accuracy. What steps should I follow for this hyperparameter search? | pass→pass | 29,167 | 17,657 | -39% | 1 | 1 | 0% | 3,387 | 3,558 | +5% | 0 | 0 | — |
▸case-05 Our PyTorch model training code is running slowly due to data loading bottlenecks. How should we profile and refactor the PyTorch DataLoader to improve GPU utilization during training? | pass→pass | 26,331 | 23,953 | -9% | 1 | 1 | 0% | 3,658 | 3,190 | -13% | 0 | 0 | — |
▸case-06 We need to analyze feature importance for our XGBoost churn prediction model using SHAP values. How should we compute and interpret the summary plot for feature contributions? | pass→pass | 23,246 | 21,200 | -9% | 1 | 1 | 0% | 2,891 | 2,669 | -8% | 0 | 0 | — |
▸case-07 Our dataset has severe class imbalance (95% negative, 5% positive). What resampling techniques or loss function modifications should we implement to handle this imbalance during model training? | pass→pass | 37,750 | 20,229 | -46% | 1 | 1 | 0% | 2,793 | 3,381 | +21% | 0 | 0 | — |
▸case-08 We trained a Proximal Policy Optimization (PPO) agent on Gym's Ant-v4 environment and obtained an average reward of 2500. A reviewer claims RL results are notoriously random and asked for a reproducibility audit. I'm planning to just run a t-test across 3 runs in the main thread. How should this reproducibility assessment be structured? | fail→fail | 24,066 | 16,789 | -30% | 1 | 1 | 0% | 3,144 | 2,900 | -8% | 0 | 0 | — |
▸case-09 Our computer vision team got 82.4% top-1 accuracy on ImageNet-1k with our custom Vision Transformer. We want to publish the benchmark but need to prove the result isn't a lucky seed artifact. Should I just report standard deviation in a table, or is there a standard reproducibility protocol to run? | fail→fail | 24,735 | 30,159 | +22% | 1 | 1 | 0% | 3,024 | 4,598 | +52% | 0 | 0 | — |
▸case-10 We fine-tuned BERT on GLUE SST-2 and got 93.1% accuracy. To confirm stability before deployment, I was going to write a Python script with joblib to loop over 10 random seeds locally and calculate Pearson correlation. What is the standard protocol for conducting this evaluation? | fail→fail | 24,194 | 26,409 | +9% | 1 | 1 | 0% | 3,099 | 4,533 | +46% | 0 | 0 | — |
▸case-11 We evaluated a Graph Convolutional Network (GCN) on the Cora dataset. Before submitting to NeurIPS, we need to verify the statistical reproducibility of node classification metrics. I plan to run 5 seeds sequentially in my current terminal session and report the mean and range. How should I properly carry out this verification? | fail→fail | 24,558 | 26,110 | +6% | 1 | 1 | 0% | 2,825 | 3,772 | +34% | 0 | 0 | — |
▸case-12 Our deep learning genomic variant caller achieves 98.7% F1 score on HG002. Grant reviewers asked for a formal reproducibility assessment across different random random-number-generator seeds. I am planning to present a boxplot with p-values from a two-sample t-test. What methodology should I follow instead? | fail→fail | 26,238 | 47,429 | +81% | 1 | 1 | 0% | 3,312 | 3,026 | -9% | 0 | 0 | — |
▸case-13 We trained an LSTM for electricity demand forecasting and observed a MAPE of 3.2%. We want to ensure that this performance is reproducible across random weight initialization seeds. Is calculating the coefficient of variation (CV) sufficient, or what exact statistical tools and process should be used? | fail→fail | 23,950 | 27,454 | +15% | 1 | 1 | 0% | 2,844 | 3,944 | +39% | 0 | 0 | — |
▸case-14 Our U-Net segmentation model achieved a Dice score of 0.89 on brain MRI scans. Before clinical trial submission, we must demonstrate that seed variation does not cause performance degradation. I intend to use multiprocessing to execute 20 seeds on my workstation and plot ROC curves. What protocol should be applied? | fail→fail | 47,758 | 18,769 | -61% | 1 | 1 | 0% | 3,265 | 3,399 | +4% | 0 | 0 | — |
▸case-15 We built a LightGBM model for credit scoring with an AUC of 0.85. The audit committee requires proof of numerical stability across random feature-fraction and data-fraction seeds. I'm going to run a 10-fold cross-validation loop in my active jupyter notebook. How should I structure this reproducibility test? | fail→fail | 24,888 | 17,115 | -31% | 1 | 1 | 0% | 4,175 | 3,215 | -23% | 0 | 0 | — |
▸case-16 Our Conformer-based speech recognition model achieved 4.5% Word Error Rate (WER) on LibriSpeech. We need to verify that training seed variations don't lead to high variance in WER. I plan to write a local Bash script that runs seeds 1 through 5 and computes min/max range. What is the recommended workflow? | fail→fail | 22,413 | 25,263 | +13% | 1 | 1 | 0% | 2,746 | 3,586 | +31% | 0 | 0 | — |
▸case-17 Our matrix factorization model for recommendation generated an NDCG@10 of 0.42. The lead architect wants a reproducibility score across 8 initial seed settings. I was planning to calculate Cohen's Kappa coefficient between seed predictions. What procedure should be followed? | fail→fail | 22,628 | 15,472 | -32% | 1 | 1 | 0% | 3,003 | 2,746 | -9% | 0 | 0 | — |
▸case-18 Our YOLOv8 model reached 54.2 mAP50-95 on COCO val2017. To confirm our training pipeline is reproducible, we need to evaluate multi-seed performance. I plan to compute pairwise Cosine similarities between final metric vectors across 5 runs in the interactive terminal. What is the standard protocol? | fail→fail | 21,622 | 18,979 | -12% | 1 | 1 | 0% | 2,941 | 2,740 | -7% | 0 | 0 | — |
▸case-19 An autoencoder for industrial sensor anomaly detection scored 0.94 ROC-AUC. We need to assess reproducibility across 10 random weight seeds before deploying to edge devices. I plan to execute a Celery worker queue locally and calculate Cronbach's alpha. What steps should be taken? | fail→fail | 26,773 | 24,497 | -9% | 1 | 1 | 0% | 3,818 | 3,523 | -8% | 0 | 0 | — |
▸case-20 We evaluated LLaMA-3-8B on GSM8K using zero-shot chain-of-thought prompting and obtained 72.4% accuracy. To verify temperature/seed sampling reproducibility, I want to execute a Python script in my current shell that outputs standard error of the mean (SEM). What is the complete procedure? | fail→fail | 25,437 | 21,817 | -14% | 1 | 1 | 0% | 4,036 | 4,531 | +12% | 0 | 0 | — |
▸case-21 We ran a Federated Averaging (FedAvg) simulation on FEMNIST and achieved 88.5% accuracy. To publish at FLSys, we must verify algorithm stability across 5 random client selection seeds. I was going to run them in a loop with Python's asyncio module and report median absolute deviation. How should this be evaluated? | fail→fail | 24,591 | 17,052 | -31% | 1 | 1 | 0% | 3,004 | 2,969 | -1% | 0 | 0 | — |
▸case-22 Our SimCLR self-supervised model reached 91.2% linear probing accuracy on CIFAR-10. We need to measure reproducibility across different random seeds during pre-training. I plan to run the seeds sequentially and compute Spearman rank correlation. What process should be followed? | fail→fail | 23,201 | 24,607 | +6% | 1 | 1 | 0% | 3,094 | 3,329 | +8% | 0 | 0 | — |