▸case-01 We trained an XGBoost classifier on tabular customer churn data. We need to evaluate global feature importance, but relying on default built-in gain or split counts gives skewed rankings toward high-cardinality continuous variables. Write Python code using an appropriate model interpretation library to compute exact SHAP values for the tree ensemble efficiently. | pass→pass | 12,736 | 28,604 | +125% | 1 | 1 | 0% | 2,502 | 4,377 | +75% | 0 | 0 | — |
▸case-02 We have a scikit-learn Pipeline combining a custom imputer, TF-IDF vectorizer, and an SVM classifier. We need to explain individual instance predictions locally without access to model gradients or tree structures using LIME. Write Python code to generate a local surrogate explanation for a single prediction row. | fail→fail | 14,396 | 20,619 | +43% | 1 | 1 | 0% | 2,800 | 3,444 | +23% | 0 | 0 | — |
▸case-03 We are interpreting a PyTorch ResNet image classification model. Standard vanilla gradient saliency maps show noisy, saturated attribution masks where non-zero pixels fail to explain confidence drops. Write Python code using Captum to calculate attributions with Integrated Gradients to handle saturation. | pass→pass | 16,765 | 20,505 | +22% | 1 | 1 | 0% | 3,334 | 4,449 | +33% | 0 | 0 | — |
▸case-04 In our Random Forest regressor for real estate valuation, built-in Gini impurity feature importances flag random high-cardinality noise columns as top predictors. Write Python code using scikit-learn inspection tools to evaluate permutation feature importances reliably on a held-out validation dataset. | pass→pass | 16,676 | 19,004 | +14% | 1 | 1 | 0% | 3,274 | 4,169 | +27% | 0 | 0 | — |
▸case-05 Our gradient boosting loan model has strongly correlated features like annual income and monthly salary. Standard 1D Partial Dependence Plots force the model to evaluate unrealistic combinations. Write Python code using Accumulated Local Effects (ALE) to analyze feature effects while respecting feature correlation. | pass→pass | 16,225 | 27,338 | +68% | 1 | 1 | 0% | 3,128 | 5,826 | +86% | 0 | 0 | — |
▸case-06 A customer was denied a loan by a LightGBM binary classifier. The compliance team needs to provide actionable feedback explaining the minimal feature adjustments required for the customer to flip the outcome from rejected to approved. Write Python code using DiCE to generate actionable counterfactual explanations. | pass→pass | 22,952 | 19,813 | -14% | 1 | 1 | 0% | 3,830 | 4,242 | +11% | 0 | 0 | — |
▸case-07 We have a Hugging Face BERT sentiment model. We want token-level attributions for a given input sentence using Captum in PyTorch. Write Python code to compute attributions relative to reference baseline tokens. | pass→pass | 17,813 | 19,400 | +9% | 1 | 1 | 0% | 3,376 | 4,275 | +27% | 0 | 0 | — |
▸case-08 We are initializing a SHAP explainer for a tabular dataset with 200,000 training samples. Passing the entire dataset as the background distribution causes severe memory and speed bottlenecks. Write Python code to summarize the background dataset using SHAP summary functions. | fail→fail | 15,259 | 15,963 | +5% | 1 | 1 | 0% | 2,309 | 3,432 | +49% | 0 | 0 | — |
▸case-09 We are setting up LimeTabularExplainer for a scikit-learn model trained on tabular data that includes categorical features like education level and zip code. If categorical features are treated as continuous, LIME samples invalid fractional values. Write Python code configuring the explainer to respect categorical feature indices. | pass→pass | 16,760 | 26,955 | +61% | 1 | 1 | 0% | 2,619 | 4,511 | +72% | 0 | 0 | — |
▸case-10 Our production inference backend serves 1,000 requests per second. Calculating exact SHAP values synchronously inside the real-time API endpoint causes timeout failures. Provide a Python architecture snippet using FastAPI BackgroundTasks or Celery to decouple explainability computation from prediction serving. | pass→pass | 19,340 | 19,187 | -1% | 1 | 1 | 0% | 2,977 | 4,055 | +36% | 0 | 0 | — |
▸case-11 We trained a PyTorch neural network for 5-class image classification. We want to extract target-specific attributions for predicted class 3 using Captum IntegratedGradients. Write Python code that explicitly targets class index 3. | pass→pass | 15,660 | 12,905 | -18% | 1 | 1 | 0% | 2,328 | 2,333 | +0% | 0 | 0 | — |
▸case-12 In our risk scoring model trained with LightGBM, stakeholders require that higher debt ratio must never decrease the predicted default risk score, enforcing guaranteed monotonic behavior. Provide Python code setting up LightGBM model training with this interpretability constraint. | fail→pass | 14,092 | 22,122 | +57% | 1 | 1 | 0% | 2,563 | 3,732 | +46% | 0 | 0 | — |
▸case-13 Our regulatory compliance department rejects local linear models like LIME because linear coefficients do not provide clear IF-THEN coverage boundaries. Provide Python code using Alibi AnchorTabular to generate high-precision IF-THEN rule explanations with explicit coverage guarantees for a tabular prediction. | pass→pass | 18,819 | 17,457 | -7% | 1 | 1 | 0% | 3,576 | 3,621 | +1% | 0 | 0 | — |
▸case-14 We suspect that age and income have a non-additive interaction effect in our XGBoost marketing conversion model. Standard 1D SHAP summary plots conflate main effects with interaction terms. Write Python code using SHAP to extract explicit pairwise interaction values. | pass→pass | 16,740 | 25,920 | +55% | 1 | 1 | 0% | 3,284 | 5,528 | +68% | 0 | 0 | — |
▸case-15 When calculating feature attributions using Integrated Gradients on a PyTorch model, we need to mathematically verify that our integration approximation is accurate according to the completeness axiom. Write Python code using PyTorch and Captum to assert that the attribution sum equals the prediction difference. | pass→pass | 13,521 | 16,494 | +22% | 1 | 1 | 0% | 2,797 | 3,573 | +28% | 0 | 0 | — |
▸case-16 We have an LSTM model predicting financial fraud from time-series transaction sequences of shape (batch, time_steps, features). Permuting timesteps independently breaks temporal structure. Write Python code using Captum IntegratedGradients to compute feature attributions while preserving 3D sequence dimensions. | pass→pass | 14,214 | 21,647 | +52% | 1 | 1 | 0% | 2,642 | 4,588 | +74% | 0 | 0 | — |
▸case-17 We trained a GradientBoostingRegressor in scikit-learn. We want to inspect partial dependence for features age and education on validation data using scikit-learn built-in inspection utilities. Write Python code to generate and plot partial dependence. | pass→pass | 10,904 | 12,208 | +12% | 1 | 1 | 0% | 2,211 | 2,654 | +20% | 0 | 0 | — |
▸case-18 We need to tune hyperparameters like learning_rate, num_leaves, and max_depth for a LightGBM classifier on customer retention data using Optuna. Write Python code setting up the Optuna objective function, trial parameter suggestions, and study optimization run. | pass→pass | 19,970 | 16,414 | -18% | 1 | 1 | 0% | 3,258 | 3,862 | +19% | 0 | 0 | — |
▸case-19 We are setting up multi-GPU training for a ResNet PyTorch model across 4 GPUs using PyTorch DistributedDataParallel. Write Python code to initialize the process group, wrap the model in DistributedDataParallel, and set up DistributedSampler. | pass→pass | 16,169 | 12,515 | -23% | 1 | 1 | 0% | 3,550 | 2,974 | -16% | 0 | 0 | — |
▸case-20 We want to log training metrics, loss curves, model parameters, and model artifacts to an MLflow tracking server during PyTorch model training. Write Python code to configure MLflow experiment tracking within the training loop. | pass→pass | 16,389 | 15,913 | -3% | 1 | 1 | 0% | 3,332 | 3,851 | +16% | 0 | 0 | — |
▸case-21 We are using Captum IntegratedGradients on a PyTorch vision model to explain image classifications. A standard all-black zero baseline tensor creates false attributions for dark image regions. Write Python code that defines alternative baseline images, such as Gaussian noise or blurred versions, to test baseline sensitivity. | pass→pass | 18,142 | 29,170 | +61% | 1 | 1 | 0% | 3,557 | 4,900 | +38% | 0 | 0 | — |
▸case-22 We have a credit card fraud detection LightGBM model trained on a highly imbalanced dataset (0.1% positive class). Calculating SHAP base values using an unweighted random sample of the training set results in an expected value near 0.001, making all positive prediction explanations appear disproportionately massive. Write Python code to establish a balanced baseline background dataset for SHAP interpretation. | pass→pass | 18,546 | 20,995 | +13% | 1 | 1 | 0% | 3,440 | 4,322 | +26% | 0 | 0 | — |
▸case-23 We computed local SHAP value arrays of shape (N_samples, N_features) for a LightGBM tabular classification model test set. We need to aggregate these local feature attributions into a global feature ranking and plot a summary plot showing feature impact distribution. Write Python code using SHAP visualization tools to create this global summary plot. | pass→pass | 12,547 | 11,922 | -5% | 1 | 1 | 0% | 2,496 | 2,923 | +17% | 0 | 0 | — |