▸case-22 Should we retrain our deep learning model every hour on live production streaming data to keep it updated with zero lag? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-11 When deploying a newly trained recommendation model to production, we want to route 5% of live traffic to the new version initially and gradually scale up based on real-time metrics, rolling back automatically if error rates spike. What deployment strategy should be implemented? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-20 Our data team wants a pipeline orchestrator that models data assets explicitly (tables, ML models) rather than just task DAG execution order, with native metadata tracking. What modern orchestrator tool fits this asset-aware paradigm? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-04 When raw training datasets arrive in an S3 bucket, our AWS pipeline must automatically launch a multi-step ML training workflow without running expensive continuous polling tasks. What native AWS event orchestration service should trigger this workflow? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-05 Our data engineering team wants to manage object storage datasets using Git-like branching, merging, and commits without duplicating terabytes of data. Which open-source data versioning tool provides Git semantics directly over object stores like S3 or GCS? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-06 We are observing silent model degradation caused by unexpected missing columns and schema drift in incoming training batches. What specialized open-source Python framework should we integrate into our data ingestion pipeline to define data assertions and generate validation suites? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-07 Our distributed model training jobs on AWS run for 12 hours and consume substantial EC2 spend. We want to cut compute costs by up to 70% using non-guaranteed capacity, but need to handle unexpected node terminations. What compute purchasing option and recovery mechanism should we configure in SageMaker? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-01 Our team is building a Kubernetes-native machine learning platform that needs centralized experiment tracking and feature management. Please produce a comprehensive system design breakdown, operational runbook steps, and concrete validation checks to confirm data lineage and experiment logging integrity. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-08 Our fraud detection ML model requires both offline batch features for training and ultra-low latency online features for live REST prediction. What dedicated MLOps architecture component handles sync between batch and online stores? | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-15 Our data science team needs a centralized system to track trained model artifacts, transition models through Staging and Production states, and require sign-offs before deployment. What core MLOps component manages this lifecycle? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-16 We need to scale PyTorch model training across 16 GPU nodes on a Kubernetes cluster with minimal communication bottleneck. What distributed training framework or library should be configured? | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-23 We want to store raw training datasets directly in our Git repository alongside python code using Git LFS for version control. Is this the recommended approach for terabyte-scale dataset storage in production MLOps? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-13 We want our Kubernetes ML deployment configurations to be declaratively stored in Git, so that any merge to main automatically synchronizes the cluster state without running manual kubectl commands. What deployment methodology should we adopt? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-10 We are migrating our machine learning pipelines to Google Cloud Platform and need a fully managed, serverless orchestration engine that supports Kubeflow Pipelines DSL. Which native GCP service should we use? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-19 When launching ML training pods on Kubernetes, GPU workloads are colliding and causing out-of-memory errors on shared nodes. How should GPU resources be explicitly requested in the Kubernetes pod specification? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-09 Our high-throughput vision model trained in PyTorch is experiencing high inference latency in production. Our developer suggests keeping the native PyTorch Python process for serving and simply increasing worker threads. Is there an open-source cross-platform runtime format and engine that should be used to optimize inference performance instead? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-03 We need a Kubernetes-native serverless model serving component on top of our cluster to handle auto-scaling down to zero and standardize custom inference protocol across frameworks. Should we write custom Flask microservices on standard Deployment manifests or leverage a specialized Kubernetes CRD tool? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-02 We are deploying high-throughput model inference endpoints on Azure and need to minimize cloud spend while supporting canary releases. Can you outline a cost-optimized deployment strategy, detail the pipeline automation workflow, and specify practical testing steps to verify traffic splitting and failover recovery? | pass→pass | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-24 Can we replace our experiment tracking tool entirely by logging hyperparameters and loss values directly into stdout application log files in CloudWatch? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-14 We are building an MLOps platform that processes protected health information (PHI) under HIPAA guidelines. What essential security controls must be applied to model artifacts and training data stored in cloud storage buckets? | pass→pass | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-21 Our online model inference microservice on Triton or KServe is experiencing high CPU consumption due to executing single-item predictions under high request traffic. What serving optimization technique aggregates concurrent requests into a single matrix calculation? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-12 Our production NLP endpoint has been running unchanged for 6 months, but user performance complaints are rising despite fast API latency. What monitoring capability and specific statistical metrics should we implement to catch degradation in input feature distributions? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-17 Our ML training pipeline needs access to external database credentials and API tokens. Storing these credentials in environment variables inside Docker images or Git commits is forbidden. How should secret access be managed? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-18 To keep our production recommendation model updated, our engineer proposes configuring S3 object creation events to immediately launch a full model retraining job whenever any new CSV file is uploaded to our raw landing bucket. Is triggering full retraining directly on raw file upload the recommended MLOps pattern, or how should automated retraining be triggered? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |