▸case-01 I am building a batch data drift detector for a credit scoring model with continuous numerical features (such as income and credit score) that do not follow Gaussian distributions. A team member suggested using a Student's t-test to compare the baseline distribution with weekly production batches. Provide a Python script using scipy.stats to perform a proper statistical drift test on these continuous features and return p-values. | pass→pass | 17,026 | 19,530 | +15% | 1 | 1 | 0% | 3,487 | 4,162 | +19% | 0 | 0 | — |
▸case-02 We need to monitor data drift for categorical features (such as customer subscription plan tier and region code) in our production recommendation pipeline. An engineer proposed applying a 2-sample Kolmogorov-Smirnov test directly on the categorical string codes. Write a Python function using scipy.stats that correctly assesses categorical drift between baseline and target datasets. | pass→pass | 21,645 | 27,029 | +25% | 1 | 1 | 0% | 3,436 | 4,574 | +33% | 0 | 0 | — |
▸case-03 We calculated the Population Stability Index (PSI) for our model's input features across baseline and current monthly inference data. The resulting PSI values are 0.04 for feature A, 0.18 for feature B, and 0.31 for feature C. Detail the standard industry interpretation thresholds for these PSI values and specify which features require immediate drift remediation. | pass→pass | 8,617 | 8,789 | +2% | 1 | 1 | 0% | 1,552 | 1,838 | +18% | 0 | 0 | — |
▸case-04 Our real-time transaction fraud model processes streaming telemetry. We want to detect sudden shifts in model prediction error without hardcoding a fixed moving window size, which either reacts too slowly to sudden changes or introduces noise during stable periods. Write a Python snippet using the river library that implements an adaptive streaming drift detector for binary prediction outcomes. | pass→pass | 10,567 | 20,836 | +97% | 1 | 1 | 0% | 1,919 | 3,538 | +84% | 0 | 0 | — |
▸case-05 We are monitoring an LLM embedding space (1536-dimensional vectors from OpenAI embeddings) for retrieval semantic drift. Running 1536 univariate KS tests causes high false positive rates due to multiple testing and ignores cross-feature correlations. Write a Python snippet that computes a single drift score across high-dimensional embeddings using a kernel-based distance metric. | pass→pass | 20,203 | 20,916 | +4% | 1 | 1 | 0% | 4,077 | 4,724 | +16% | 0 | 0 | — |
▸case-06 Our MLOps team is debating whether to compare daily inference data against the previous day's inference batch (rolling baseline) or against the original training dataset (gold-standard baseline). Explain the risk of baseline drift associated with rolling baselines and recommend the appropriate baseline configuration for drift monitoring. | pass→pass | 17,613 | 26,771 | +52% | 1 | 1 | 0% | 2,495 | 4,550 | +82% | 0 | 0 | — |
▸case-07 We are implementing the Drift Detection Method (DDM) for streaming classification monitoring based on model error rates. Describe the mathematical threshold conditions used by DDM to transition between Normal, Warning, and Drift states based on minimum error rate p_min and minimum standard deviation s_min. | pass→pass | 16,304 | 13,846 | -15% | 1 | 1 | 0% | 2,594 | 3,009 | +16% | 0 | 0 | — |
▸case-08 Write a Python script using the Evidently library (v0.4+) to generate a data drift report comparing reference DataFrame `ref_df` and current DataFrame `curr_df`. Export the results to an HTML report file named `drift_report.html`. | fail→pass | 8,212 | 9,265 | +13% | 1 | 1 | 0% | 1,642 | 2,161 | +32% | 0 | 0 | — |
▸case-09 In our healthcare risk prediction system, input patient demographics remain identical, but historical diagnostic labels Y given demographic inputs X have changed due to updated clinical guidelines (i.e., P(Y|X) changed while P(X) remained constant). Distinguish whether this is Covariate Shift, Concept Drift, or Prior Probability Shift, and explain why feature distribution drift tests miss it. | pass→pass | 11,276 | 12,635 | +12% | 1 | 1 | 0% | 2,147 | 2,564 | +19% | 0 | 0 | — |
▸case-10 We need a Python microservice using `prometheus_client` that periodically calculates feature PSI scores and exports them for alerting. Define the appropriate Prometheus metric type for tracking feature PSI across different feature names and show how to set its value. | pass→pass | 14,165 | 20,200 | +43% | 1 | 1 | 0% | 2,746 | 3,772 | +37% | 0 | 0 | — |
▸case-11 When running 2-sample KS tests on millions of production records daily, extremely small p-values (< 0.001) are returned even for minute, operationally meaningless shifts in feature values. Provide a Python strategy to mitigate sample size sensitivity during statistical drift testing on large-scale datasets. | pass→pass | 20,875 | 19,809 | -5% | 1 | 1 | 0% | 3,946 | 4,279 | +8% | 0 | 0 | — |
▸case-12 In our loan default prediction system, ground truth outcomes take 12 months to observe. We need to estimate production model performance degradation (e.g., ROC AUC drop) in real time without immediate target labels. Describe how Confidence-based Performance Estimation (CBPE) calculates estimated performance metrics. | pass→pass | 17,145 | 22,711 | +32% | 1 | 1 | 0% | 3,453 | 5,087 | +47% | 0 | 0 | — |
▸case-13 We want to monitor continuous regression model predictions for mean shift using the Page-Hinkley change detection algorithm. Write a Python function implementing the Page-Hinkley test that maintains cumulative difference sums and triggers an alert when the statistic exceeds threshold delta. | pass→pass | 23,600 | 22,974 | -3% | 1 | 1 | 0% | 4,757 | 4,953 | +4% | 0 | 0 | — |
▸case-14 Design an automated MLOps workflow policy triggered by model drift metrics. Describe the decision path for handling minor feature drift (PSI between 0.1 and 0.25) vs severe feature drift (PSI > 0.25) in high-risk financial applications. | fail→fail | 33,497 | 48,380 | +44% | 1 | 1 | 0% | 5,463 | 7,895 | +45% | 0 | 0 | — |
▸case-15 Our incoming inference data stream suddenly has missing values (NaN) jump from 0.1% to 25% for a key feature due to an upstream API payload change. Explain how a drift detector should distinguish missingness drift from feature distribution drift, and write a Python check for missing value rate changes. | pass→pass | 22,797 | 27,626 | +21% | 1 | 1 | 0% | 4,405 | 5,725 | +30% | 0 | 0 | — |
▸case-16 We want a symmetric, bounded distance metric to evaluate probability distribution drift between model prediction probabilities on baseline vs production. Write a Python function using scipy.spatial.distance to compute Jensen-Shannon divergence between baseline and target probability distributions. | pass→pass | 19,879 | 19,066 | -4% | 1 | 1 | 0% | 3,839 | 3,929 | +2% | 0 | 0 | — |
▸case-17 Univariate tests fail to catch complex non-linear feature interaction shifts (e.g., age and income shifting together while univariate distributions look normal). Describe how Classifier Two-Sample Test (C2ST) works for multivariate drift detection and write a Python procedure using scikit-learn. | pass→pass | 28,945 | 25,839 | -11% | 1 | 1 | 0% | 4,700 | 5,659 | +20% | 0 | 0 | — |
▸case-18 We are monitoring 200 input features daily using 2-sample KS tests with alpha = 0.05. Without correction, we experience frequent false alarms due to the multiple testing problem. Show how to apply False Discovery Rate (FDR) Benjamini-Hochberg correction in Python using statsmodels. | pass→pass | 15,206 | 22,319 | +47% | 1 | 1 | 0% | 3,129 | 3,980 | +27% | 0 | 0 | — |
▸case-19 In an e-commerce conversion prediction model, purchase labels arrive 7 days after inference. Explain how to set up a delayed-label target drift monitoring pipeline that compares baseline target distribution P(Y) with delayed ground truth P(Y_true). | pass→pass | 22,235 | 26,728 | +20% | 1 | 1 | 0% | 3,840 | 5,499 | +43% | 0 | 0 | — |
▸case-20 We are training a Random Forest classifier offline and need a Python script to perform 5-fold cross-validated grid search over max_depth and n_estimators using scikit-learn. Output the best hyperparameters and test score. | pass→pass | 8,750 | 8,358 | -4% | 1 | 1 | 0% | 1,893 | 2,117 | +12% | 0 | 0 | — |
▸case-21 We need to optimize a trained PyTorch image classification model by converting it to ONNX format and applying static INT8 quantization using ONNX Runtime. Provide the Python script to execute this conversion. | pass→pass | 20,087 | 18,199 | -9% | 1 | 1 | 0% | 3,986 | 4,236 | +6% | 0 | 0 | — |
▸case-22 We are updating our Feast feature store repository to add a new entity and feature view for user transaction aggregations. Provide the Feast Python definition file (`features.py`) defining the Field, Entity, and FeatureView. | pass→pass | 11,727 | 7,795 | -34% | 1 | 1 | 0% | 2,427 | 1,958 | -19% | 0 | 0 | — |
▸case-23 Write a scikit-learn preprocessing pipeline using ColumnTransformer that applies StandardScaler to numerical columns ('age', 'balance') and OneHotEncoder to categorical columns ('country', 'gender'). | pass→pass | 5,536 | 7,465 | +35% | 1 | 1 | 0% | 1,178 | 1,870 | +59% | 0 | 0 | — |