▸case-01 We are building an automated prediction monitor for continuous numeric features in our credit scoring model. Our pipeline team suggested using a Chi-Square test to compare the baseline feature distribution against real-time prediction inputs. Recommend the appropriate statistical test for continuous feature drift and state why Chi-Square is inadequate. | pass→pass | 14,811 | 26,532 | +79% | 1 | 1 | 0% | 2,624 | 3,126 | +19% | 0 | 0 | — |
▸case-02 In our fraud prediction monitoring system, ground truth labels take up to 30 days to finalize due to dispute processing. The lead engineer wants to calculate real-time precision and recall on every incoming prediction batch. Explain how prediction monitoring should be configured when ground truth feedback is delayed, and specify what drift metrics to rely on instead of immediate supervised performance. | pass→pass | 17,930 | 16,841 | -6% | 1 | 1 | 0% | 2,984 | 3,062 | +3% | 0 | 0 | — |
▸case-03 We need to monitor categorical features like device_type and state_code in our e-commerce recommendation prediction monitor. A team member proposed running a two-sample Kolmogorov-Smirnov test on these string features. Provide the correct statistical method for categorical feature drift detection. | fail→pass | 15,034 | 16,919 | +13% | 1 | 1 | 0% | 2,759 | 3,411 | +24% | 0 | 0 | — |
▸case-04 For our sentiment analysis prediction monitor, the pipeline compares today's prediction outputs strictly against yesterday's prediction outputs in a rolling daily loop without retaining a fixed reference set. Identify the risk of this setup (such as slow drift masking) and specify how reference baselines should be structured for drift monitoring. | pass→pass | 15,513 | 19,468 | +25% | 1 | 1 | 0% | 2,529 | 3,588 | +42% | 0 | 0 | — |
▸case-05 We are designing the JSON logging schema for an inference service feeding our prediction monitor. A junior developer suggests logging only `{"prediction": 0.85}` to save bandwidth. Specify the essential fields required in the inference log payload to enable feature drift analysis and model lineage tracking. | pass→pass | 15,031 | 14,057 | -6% | 1 | 1 | 0% | 2,671 | 2,745 | +3% | 0 | 0 | — |
▸case-06 When our prediction monitoring worker consumes log streams from an updated inference service, some incoming payload messages are missing required feature fields due to API schema changes. The current monitor crashes silently. What validation mechanism and error handling state should the monitoring worker implement for invalid payloads? | pass→pass | 16,294 | 18,894 | +16% | 1 | 1 | 0% | 2,688 | 3,712 | +38% | 0 | 0 | — |
▸case-07 Our loan default model receives inputs where applicant average income has increased due to inflation, but the relationship between income and default probability remains unchanged. The monitoring alert fired for concept drift. Explain whether this scenario represents feature drift or concept drift, and define the mathematical difference between them. | pass→pass | 13,317 | 14,815 | +11% | 1 | 1 | 0% | 2,051 | 2,485 | +21% | 0 | 0 | — |
▸case-08 In our risk scoring prediction monitor, we calculate Population Stability Index (PSI) for credit score predictions. The current threshold alerts on any PSI above 0.01. Provide standard industry PSI threshold guidance for categorizing slight, moderate, and significant distribution shifts. | pass→pass | 11,552 | 23,187 | +101% | 1 | 1 | 0% | 1,997 | 2,365 | +18% | 0 | 0 | — |
▸case-09 In a production BERT text classification model monitoring system, high numbers of unseen domain slang tokens are passing through tokenization without being tracked. The developer claims standard latency monitoring is sufficient. Detail how an input validation monitor should quantify out-of-vocabulary (OOV) token rate and why it matters for NLP model quality. | pass→pass | 20,016 | 22,692 | +13% | 1 | 1 | 0% | 3,132 | 4,269 | +36% | 0 | 0 | — |
▸case-10 Our fraud model prediction monitor checks if individual feature values fall within simple minimum and maximum bounds from training. However, anomalous combinations of normal feature values (e.g., low age combined with high credit limit) are slipping through undetected. Recommend a multivariate anomaly detection approach for incoming inference requests. | pass→pass | 16,312 | 19,200 | +18% | 1 | 1 | 0% | 2,754 | 3,596 | +31% | 0 | 0 | — |
▸case-11 An ML engineer set up an automated prediction monitor that runs a Kolmogorov-Smirnov test on micro-batches of 10 inference requests every 5 seconds. The monitor generates hundreds of false drift alerts daily. Explain why micro-batching causes false positives in statistical drift tests and recommend appropriate windowing strategies. | pass→pass | 19,800 | 20,170 | +2% | 1 | 1 | 0% | 3,159 | 4,029 | +28% | 0 | 0 | — |
▸case-12 A DevOps dashboard reports 99.9% uptime, 20ms p95 latency, and zero HTTP 500 errors for an image classification API, so the team marked prediction monitoring complete. Explain why system metrics alone are insufficient for prediction monitoring and list three ML-specific health metrics that must be monitored. | pass→pass | 14,395 | 14,711 | +2% | 1 | 1 | 0% | 2,394 | 2,749 | +15% | 0 | 0 | — |
▸case-13 Our binary classification prediction monitor only records final class labels (0 or 1). However, the model has recently shifted from returning high-confidence predictions (0.95 or 0.05) to uncertain predictions (0.51 or 0.49), while class output ratios remain 50/50. Explain how monitoring prediction probability distributions reveals calibration loss before class balance changes. | pass→pass | 17,151 | 20,342 | +19% | 1 | 1 | 0% | 3,048 | 4,024 | +32% | 0 | 0 | — |
▸case-14 An automated prediction monitor is connected directly to a CI/CD pipeline to trigger retrain jobs on AWS SageMaker whenever a single 5-minute window exhibits feature drift. This results in constant redundant retraining runs. Design a robust policy for triggering retrain pipelines from prediction monitor alerts. | pass→pass | 22,183 | 20,046 | -10% | 1 | 1 | 0% | 3,755 | 3,961 | +5% | 0 | 0 | — |
▸case-15 To evaluate overall feature store drift, our prediction monitor averages p-values across 50 individual univariate Kolmogorov-Smirnov tests. Explain why averaging univariate p-values is statistically unsound and recommend a proper distance metric or method for measuring drift across numerical feature distributions. | pass→pass | 18,251 | 20,774 | +14% | 1 | 1 | 0% | 3,205 | 4,002 | +25% | 0 | 0 | — |
▸case-16 In our tabular prediction pipeline, missing feature values are silently imputed with zeros before being passed to both the model and the drift monitor. Consequently, when an upstream data pipeline bug caused a feature to become 90% missing, feature drift went undetected. Recommend how missingness should be monitored. | pass→pass | 16,867 | 18,121 | +7% | 1 | 1 | 0% | 2,725 | 3,229 | +18% | 0 | 0 | — |
▸case-17 We are choosing a prediction distribution divergence metric for our multi-class recommendation model. The team lead proposed Kullback-Leibler (KL) divergence, but occasionally production predictions have zero probability for rare classes, causing division by zero or infinite values. Recommend an alternative bounded, symmetric divergence metric. | pass→pass | 12,352 | 15,847 | +28% | 1 | 1 | 0% | 2,411 | 3,520 | +46% | 0 | 0 | — |
▸case-18 An engineer wants to monitor prediction drift on an object detection model by running a two-sample Kolmogorov-Smirnov test on raw pixel arrays from incoming JPEG images. Explain why raw pixel statistical tests fail for computer vision prediction monitoring and provide two valid approaches for monitoring image model predictions. | pass→pass | 15,063 | 17,633 | +17% | 1 | 1 | 0% | 2,473 | 3,248 | +31% | 0 | 0 | — |
▸case-19 When feature distribution changes (covariate shift) occur in a real-time churn prediction model without immediate labels, a developer asserts we can calculate exact model F1-score using feature drift metrics alone. Clarify what can and cannot be determined about performance under covariate shift without ground truth labels. | pass→pass | 14,359 | 15,258 | +6% | 1 | 1 | 0% | 2,507 | 2,955 | +18% | 0 | 0 | — |
▸case-20 Our prediction monitoring setup sends high-priority PagerDuty alerts whenever any secondary feature undergoes minor statistical drift. On-call engineers are turning off alerts due to noise. Propose a tiered alerting structure categorizing low, medium, and high severity prediction monitoring events. | fail→pass | 16,615 | 20,225 | +22% | 1 | 1 | 0% | 2,797 | 3,839 | +37% | 0 | 0 | — |
▸case-21 We are configuring an automated Optuna hyperparameter optimization script for training a LightGBM model on historical tabular data. Recommend the search space and cross-validation split strategy for optimizing learning rate and max depth. | pass→pass | 17,704 | 17,614 | -1% | 1 | 1 | 0% | 3,265 | 3,824 | +17% | 0 | 0 | — |
▸case-22 We need to export a PyTorch Transformer model to ONNX format and apply INT8 dynamic quantization using ONNX Runtime for edge deployment. Provide the export command and quantization function calls. | pass→pass | 29,343 | 11,505 | -61% | 1 | 1 | 0% | 2,458 | 2,730 | +11% | 0 | 0 | — |
▸case-23 Our data engineering team is writing a PySpark SQL job to compute rolling historical aggregations (30-day average transaction count) and store them in Feast offline storage. Provide the Spark SQL query structure and offline feature table definition. | pass→pass | 15,422 | 17,381 | +13% | 1 | 1 | 0% | 2,914 | 3,693 | +27% | 0 | 0 | — |