▸case-01 We need to optimize the max_depth and learning_rate for a XGBoost fraud classifier on credit transaction data. Should we run a SHAP interaction summary or set up a GridSearchCV workflow in scikit-learn? | pass→pass | 12,299 | 6,229 | -49% | 1 | 1 | 0% | 2,014 | 1,371 | -32% | 0 | 0 | — |
▸case-02 We are selecting the optimal decision threshold for a hospital readmission model to balance precision and recall across test benchmarks. Should we generate local waterfall plots or evaluate precision-recall curves? | pass→pass | 9,958 | 6,593 | -34% | 1 | 1 | 0% | 1,664 | 1,351 | -19% | 0 | 0 | — |
▸case-07 A risk analyst notices that running perturbation-based local explanations twice on the exact same loan application yields slightly different feature weights. They ask if the model is broken. How should this behavior be explained? | pass→pass | 10,481 | 10,477 | -0% | 1 | 1 | 0% | 1,856 | 2,222 | +20% | 0 | 0 | — |
▸case-03 We suspect our customer churn model is achieving 0.99 AUC because a feature recorded post-cancellation is present in training data. Should we run a global feature importance report or conduct a prediction-time leakage audit? | pass→pass | 11,324 | 10,220 | -10% | 1 | 1 | 0% | 1,933 | 1,812 | -6% | 0 | 0 | — |
▸case-04 A credit scoring model denied applicant #4092. The product manager wants to see exactly how much each input feature shifted this applicant's base score to the final logit. We are considering showing only a top-5 global feature rank table. What specific local visualization accurately attributes each feature's marginal contribution for this individual? | pass→pass | 13,183 | 10,199 | -23% | 1 | 1 | 0% | 2,257 | 2,043 | -9% | 0 | 0 | — |
▸case-05 Our gradient boosting model predicts unusually high risk when age is high but income is medium. Developers want to inspect whether two features interact non-linearly across the dataset. Should we look at univariate partial dependence plots or bivariate SHAP interaction values? | pass→pass | 10,084 | 9,466 | -6% | 1 | 1 | 0% | 1,778 | 1,855 | +4% | 0 | 0 | — |
▸case-06 Executives need to understand why the supply chain delay predictor flags certain shipments as high risk. The engineering team proposed sending them raw TreeSHAP matrix arrays. How should model insights be packaged for non-technical stakeholders? | fail→pass | 15,622 | 15,954 | +2% | 1 | 1 | 0% | 2,388 | 2,916 | +22% | 0 | 0 | — |
▸case-08 We need local feature attributions for a 500,000-row Random Forest model. A colleague suggests using KernelSHAP without sampling. What model-specific explainer should be recommended to handle tree ensembles efficiently? | pass→pass | 7,897 | 9,757 | +24% | 1 | 1 | 0% | 1,483 | 1,970 | +33% | 0 | 0 | — |
▸case-09 A healthcare bias auditor wants to verify if protected demographic variables are disproportionately driving high-risk scores in a triage classifier. What approach provides transparent verification of feature reliance? | pass→pass | 13,309 | 17,506 | +32% | 1 | 1 | 0% | 2,103 | 3,472 | +65% | 0 | 0 | — |
▸case-10 An analyst asserts that because 'account_age' is the #1 feature globally in a random forest model, it must be the primary driver behind every single individual prediction. How should this claim be evaluated? | pass→pass | 10,644 | 14,210 | +34% | 1 | 1 | 0% | 1,741 | 2,666 | +53% | 0 | 0 | — |
▸case-11 In a housing price model, 'sqft_living' and 'num_rooms' have a 0.95 correlation coefficient. When calculating marginal feature importance using marginal expectations, the explainer evaluates unrealistic synthetic samples (e.g., 500 sqft with 10 rooms). What caveat applies to model explainers under high feature correlation? | pass→pass | 8,811 | 10,165 | +15% | 1 | 1 | 0% | 1,472 | 1,827 | +24% | 0 | 0 | — |
▸case-12 To show the marginal effect of feature X on predictions when feature X is strongly correlated with feature Y, an engineer suggests standard Partial Dependence Plots (PDP). What alternative plot handles correlated features without extrapolating to improbable feature combinations? | pass→pass | 7,545 | 7,108 | -6% | 1 | 1 | 0% | 1,201 | 1,361 | +13% | 0 | 0 | — |
▸case-13 We are interpreting a PyTorch deep neural network used for medical image classification and want axiomatic path-based feature attributions relative to a black baseline image. Which feature attribution method satisfies completeness and implementation invariance for differentiable models? | pass→pass | 10,156 | 9,193 | -9% | 1 | 1 | 0% | 1,865 | 1,973 | +6% | 0 | 0 | — |
▸case-14 When using Integrated Gradients on an image classifier, changing the reference baseline from an all-black image to a neutral gray image dramatically shifts pixel importance scores. How should baseline selection be documented when presenting local attributions? | pass→pass | 15,936 | 16,020 | +1% | 1 | 1 | 0% | 2,570 | 2,747 | +7% | 0 | 0 | — |
▸case-15 An engineer calculates permutation feature importance on the training dataset to identify key signals for a complex overfitted decision tree. Why is training-set permutation importance problematic for model explanation? | pass→pass | 9,216 | 8,347 | -9% | 1 | 1 | 0% | 1,379 | 1,519 | +10% | 0 | 0 | — |
▸case-16 A compliance officer demands a feature attribution framework where the sum of feature attributions plus the baseline expected value strictly equals the model output score. Which mathematical property guarantees this additive consistency? | pass→pass | 5,482 | 5,785 | +6% | 1 | 1 | 0% | 1,006 | 1,130 | +12% | 0 | 0 | — |
▸case-17 A loan applicant denied by an automated credit model asks: 'What is the minimum change in my income and credit debt required to get approved?' Which interpretability technique directly answers this question? | pass→pass | 5,168 | 6,993 | +35% | 1 | 1 | 0% | 951 | 1,486 | +56% | 0 | 0 | — |
▸case-18 To explain a 1000-tree ensemble, a developer trains a single shallow decision tree on the ensemble's predictions. The surrogate tree achieves 70% accuracy in mimicking the ensemble. Should this surrogate tree be presented as a reliable explanation? | pass→pass | 10,458 | 10,487 | +0% | 1 | 1 | 0% | 1,706 | 1,965 | +15% | 0 | 0 | — |
▸case-19 A data science team generates a SHAP beeswarm plot for a churn model. Red dots (high feature values) for 'customer_support_calls' cluster on the right side of the zero axis (positive SHAP values). What does this visualization convey? | pass→pass | 4,328 | 6,584 | +52% | 1 | 1 | 0% | 819 | 1,192 | +46% | 0 | 0 | — |
▸case-20 A chest X-ray classification model detects pneumonia. Clinicians need a visual heatmap showing which spatial regions of the lung image activated the convolutional layers most strongly. Which explainer technique is appropriate? | pass→pass | 6,061 | 9,952 | +64% | 1 | 1 | 0% | 1,049 | 1,944 | +85% | 0 | 0 | — |
▸case-21 We are comparing a black-box proprietary REST API endpoint against a local XGBoost model. We need a single explanation framework that can generate local explanations for both without internal model access. What class of explainers is required for the external API? | pass→pass | 5,722 | 7,974 | +39% | 1 | 1 | 0% | 1,021 | 1,707 | +67% | 0 | 0 | — |
▸case-22 A security auditor demonstrates that adding imperceptible noise to an input image completely changes its Saliency Map without changing the predicted class label. What known limitation of feature attribution maps does this expose? | pass→pass | 8,435 | 8,551 | +1% | 1 | 1 | 0% | 1,457 | 1,614 | +11% | 0 | 0 | — |