▸case-01 We are setting up an automated CI/CD pipeline for our machine learning services on Kubernetes. Can you provide a step-by-step implementation guide and a verification checklist to ensure models automatically train, test, and deploy safely to production? | fail→fail | 30,243 | 26,059 | -14% | 1 | 1 | 0% | 4,504 | 4,607 | +2% | 0 | 0 | — |
▸case-02 Our production recommendation model on AWS is experiencing silent performance decay. I need a concrete action plan with validation steps to establish real-time monitoring for data drift, feature freshness, and prediction latency. | fail→fail | 22,381 | 26,817 | +20% | 1 | 1 | 0% | 4,038 | 4,508 | +12% | 0 | 0 | — |
▸case-03 We need to architect a centralized feature store and model registry infrastructure for our team. Please outline the necessary setup procedures, best practice workflows, and verification tests to validate our deployment. | fail→fail | 39,163 | 23,593 | -40% | 1 | 1 | 0% | 8,224 | 3,979 | -52% | 0 | 0 | — |
▸case-04 I am writing a research paper on novel deep learning architectures and need to derive the exact mathematical gradients for a custom cross-entropy loss function with label smoothing. Please provide the complete step-by-step calculus derivation for backpropagation. | pass→pass | 22,482 | 28,435 | +26% | 1 | 1 | 0% | 4,145 | 5,681 | +37% | 0 | 0 | — |
▸case-05 We are designing a customer portal for our SaaS application using React and Tailwind CSS. Please write a responsive navigation bar and dashboard layout component with collapsible sidebar functionality in TypeScript. | pass→pass | 30,822 | 26,522 | -14% | 1 | 1 | 0% | 5,766 | 5,526 | -4% | 0 | 0 | — |
▸case-06 Our office building network experiencing intermittent connection drops on the physical Ethernet switches. Can you write a troubleshooting guide for configuring Cisco VLAN trunking protocols and physical Cat6 patch panel cabling? | pass→pass | 26,865 | 15,881 | -41% | 1 | 1 | 0% | 3,672 | 3,056 | -17% | 0 | 0 | — |
▸case-07 We need to automate a daily batch ML inference workflow that pulls data from Amazon S3, runs PyTorch model inference, and writes predictions back to Snowflake. Engineers usually write custom cron scripts, but we want a robust pipeline structure. Outline the implementation workflow and verification steps. | fail→fail | 25,357 | 22,995 | -9% | 1 | 1 | 0% | 3,647 | 3,673 | +1% | 0 | 0 | — |
▸case-08 Our team of six data scientists currently logs hyperparameter tuning experiments into shared Google Sheets, leading to missing metrics and unrepeatable runs. Outline a setup for centralized experiment tracking with step-by-step workflow and validation tests. | fail→fail | 32,468 | 27,752 | -15% | 1 | 1 | 0% | 5,073 | 4,786 | -6% | 0 | 0 | — |
▸case-09 We have multiple trained PyTorch models stored in S3 buckets with ad-hoc filenames like 'model_v2_final_final.pt'. We want to establish a formal model lifecycle management process with stage transitions. Provide the implementation steps and verification checks. | fail→fail | 24,064 | 26,156 | +9% | 1 | 1 | 0% | 3,445 | 4,656 | +35% | 0 | 0 | — |
▸case-10 We need to provision infrastructure for model training clusters across both AWS and GCP using declarative code. Most teams manually create SageMaker and Vertex AI resources in the cloud consoles. Provide an infrastructure-as-code deployment guide and validation procedure. | fail→fail | 31,466 | 28,294 | -10% | 1 | 1 | 0% | 5,760 | 5,571 | -3% | 0 | 0 | — |
▸case-11 We want to serve a Large Language Model on a Kubernetes GPU cluster using a high-performance inference server. Team members suggested just running a plain Flask container in a single Pod. Outline the production container orchestration architecture and validation tests. | fail→fail | 20,643 | 28,462 | +38% | 1 | 1 | 0% | 3,658 | 4,862 | +33% | 0 | 0 | — |
▸case-12 Our real-time fraud detection service suffers from training-serving skew because historical features used in offline training are computed differently from online API features. Outline the steps to build a unified feature store and verify data consistency. | fail→fail | 19,172 | 15,747 | -18% | 1 | 1 | 0% | 3,249 | 3,392 | +4% | 0 | 0 | — |
▸case-13 When data scientists open pull requests modifying training scripts, we currently have no automated checks, resulting in broken training jobs in production. Outline an automated pull-request testing pipeline for ML code and its verification checklist. | fail→fail | 18,452 | 18,063 | -2% | 1 | 1 | 0% | 2,976 | 3,730 | +25% | 0 | 0 | — |
▸case-14 Our computer vision model deployed at the edge experiences accuracy degradation when lighting conditions change at customer sites. We need an automated monitoring architecture to detect distribution drift in input images and output confidence scores. Provide the deployment steps and validation process. | fail→fail | 30,260 | 24,423 | -19% | 1 | 1 | 0% | 4,535 | 5,167 | +14% | 0 | 0 | — |
▸case-15 A financial compliance audit flagged our MLOps platform because training data containing PII is accessible to all data science team members and model artifacts in S3 lack KMS encryption key policies. Outline a remediation plan and validation steps. | fail→fail | 25,082 | 27,490 | +10% | 1 | 1 | 0% | 3,532 | 4,808 | +36% | 0 | 0 | — |
▸case-16 Our BERT Transformer model for sentiment analysis takes 350ms per request, exceeding our production SLA of 50ms. Developers suggested buying bigger CPU cloud instances. Outline a model optimization and quantization strategy with verification tests. | fail→fail | 22,086 | 20,256 | -8% | 1 | 1 | 0% | 3,845 | 4,596 | +20% | 0 | 0 | — |
▸case-17 Our tabular recommendation model training job takes 18 hours on a single virtual machine. We need to scale out to a distributed multi-node GPU cluster to reduce training time to under one hour. Outline the cluster setup, workload orchestration workflow, and verification steps. | fail→fail | 30,586 | 28,845 | -6% | 1 | 1 | 0% | 4,426 | 4,722 | +7% | 0 | 0 | — |
▸case-18 We are updating our credit scoring XGBoost model from v1 to v2. We cannot risk routing production traffic directly to v2 without validating live prediction distribution against v1. Outline a shadow deployment and canary rollout workflow with validation checks. | fail→fail | 26,396 | 25,381 | -4% | 1 | 1 | 0% | 3,786 | 4,359 | +15% | 0 | 0 | — |
▸case-19 Our demand forecasting models are retrained manually once a month when someone remembers to run a Jupyter notebook. We want an automated Continuous Training (CT) pipeline triggered by data drift alerts or fresh batch data landing in Amazon S3. Outline the workflow setup and validation process. | fail→fail | 25,434 | 20,999 | -17% | 1 | 1 | 0% | 3,504 | 3,155 | -10% | 0 | 0 | — |
▸case-20 Our real-time prediction endpoint behind an AWS Application Load Balancer crashes during peak traffic spikes because it relies on static instance counts. Outline an autoscaling policy implementation for GPU model inference instances and its validation procedures. | fail→fail | 26,702 | 27,412 | +3% | 1 | 1 | 0% | 4,432 | 4,500 | +2% | 0 | 0 | — |
▸case-21 During a model audit, our team could not trace which exact dataset commit, hyperparameter file, and git commit were used to train our current production fraud model. Outline an ML metadata tracking architecture and verification checklist. | fail→fail | 29,594 | 24,121 | -18% | 1 | 1 | 0% | 4,210 | 4,973 | +18% | 0 | 0 | — |
▸case-22 Our monthly cloud bill for GPU training and inference instances increased by 300% last quarter because instances run 24/7 without active jobs. Outline an MLOps cost optimization framework with concrete actionable steps and verification steps. | fail→fail | 20,648 | 22,588 | +9% | 1 | 1 | 0% | 3,514 | 3,388 | -4% | 0 | 0 | — |