▸case-06 A new NLP project requires building custom transformer layers, fine-tuning pretrained vision-language checkpoints, and managing model weights via a public model hub. The junior engineer asks whether to write custom C++ neural net code from scratch or use standard python deep learning libraries. Recommend the framework stack. | pass→pass | 13,286 | 23,025 | +73% | 1 | 1 | 0% | 2,238 | 3,301 | +47% | 0 | 0 | — |
▸case-01 We are deploying a RoBERTa-based text classifier for live customer support chat routing. The current model takes around 280ms per request on CPU. The product manager suggests deploying it as-is because 280ms feels fast enough for chat. Provide an architectural plan and optimization pipeline to meet production deployment standards for real-time inference. | pass→pass | 31,801 | 43,344 | +36% | 1 | 1 | 0% | 4,227 | 7,060 | +67% | 0 | 0 | — |
▸case-02 Our team is training several deep learning models for sequence classification across multiple hyperparameter sweeps. The lead developer suggested logging run metrics manually into standard text log files or custom CSV spreadsheets. Recommend the standard production tool stack for tracking runs, loss curves, and artifact metadata. | pass→pass | 15,113 | 26,435 | +75% | 1 | 1 | 0% | 2,461 | 4,109 | +67% | 0 | 0 | — |
▸case-03 We trained a credit scoring neural network that achieves 94% test set accuracy. The team wants to push this directly to production based solely on the overall accuracy metric. What mandatory evaluation step should be performed prior to deployment? | pass→pass | 17,049 | 24,585 | +44% | 1 | 1 | 0% | 1,871 | 3,789 | +103% | 0 | 0 | — |
▸case-04 A deep learning medical triage model has achieved top-tier sensitivity scores. The clinical review board asks how healthcare providers will understand why a specific patient was assigned a high-risk category. The team proposed returning only the class probability score. Outline the required explainability framework. | pass→pass | 24,188 | 32,653 | +35% | 1 | 1 | 0% | 3,211 | 5,714 | +78% | 0 | 0 | — |
▸case-05 We are replacing our legacy recommendation model with a new transformer-based candidate generator. The devops team plans to execute a 100% immediate cutover during low-traffic hours. Detail the recommended production rollout strategy. | pass→pass | 15,371 | 21,698 | +41% | 1 | 1 | 0% | 2,582 | 4,042 | +57% | 0 | 0 | — |
▸case-07 Our computer vision research team needs an environment to quickly test augmentation pipelines, visualize intermediate feature maps, and document step-by-step experimentation before writing production modules. What primary interactive environment and language should be recommended? | pass→pass | 15,493 | 17,479 | +13% | 1 | 1 | 0% | 1,720 | 2,327 | +35% | 0 | 0 | — |
▸case-08 Our PostgreSQL production database is experiencing slow join performance on the user_orders table containing 50 million rows. Analyze the EXPLAIN ANALYZE execution plan showing sequential scans and recommend indexing strategies. | fail→fail | 24,023 | 23,861 | -1% | 1 | 1 | 0% | 3,350 | 3,475 | +4% | 0 | 0 | — |
▸case-09 We need a custom CSS grid layout and Tailwind color theme for a user settings panel in our web dashboard application. Provide the React component code with flexbox responsiveness. | fail→fail | 28,978 | 33,807 | +17% | 1 | 1 | 0% | 5,114 | 6,566 | +28% | 0 | 0 | — |
▸case-10 We need to set up an Apache Airflow cluster on Kubernetes with CeleryExecutors, configure PostgreSQL backends, and write DAGs for daily transactional database ETL syncs. Provide the Helm values file and DAG configuration. | fail→fail | 27,169 | 29,278 | +8% | 1 | 1 | 0% | 5,568 | 5,886 | +6% | 0 | 0 | — |
▸case-11 A BERT-large sentiment analysis model deployed on an API endpoint currently takes 350ms per request. The product requires responding in under 100ms without upgrading to expensive GPU instances. What model size reduction and acceleration steps should be executed? | pass→pass | 16,276 | 27,469 | +69% | 1 | 1 | 0% | 2,877 | 4,330 | +51% | 0 | 0 | — |
▸case-12 An automated resume screening model shows 91% overall precision. However, HR recruiters suspect the model penalizes candidates with gaps in employment or specific regional university names. Explain how to track and mitigate this risk before launch. | pass→pass | 23,156 | 21,258 | -8% | 1 | 1 | 0% | 3,023 | 4,086 | +35% | 0 | 0 | — |
▸case-13 A risk model built with a deep neural network flags suspicious transactions. Financial regulators require the system to explain why specific transactions were flagged as fraud. Recommend the XAI approach to implement. | pass→pass | 14,498 | 25,257 | +74% | 1 | 1 | 0% | 2,378 | 3,626 | +52% | 0 | 0 | — |
▸case-14 A team developed a new recommendation model v2.0. They want to swap out v1.0 completely in production and monitor error rates in log files post-launch. Propose an improved, lower-risk deployment methodology. | pass→pass | 12,135 | 22,888 | +89% | 1 | 1 | 0% | 2,068 | 3,475 | +68% | 0 | 0 | — |
▸case-15 We need to fine-tune a pre-trained LLM for domain-specific medical summaries and log all loss curves, GPU usage, and validation metrics automatically during training. What open-source framework and tracking library combination should be used? | fail→pass | 27,522 | 25,564 | -7% | 1 | 1 | 0% | 4,317 | 3,938 | -9% | 0 | 0 | — |
▸case-16 A Vision Transformer model for image classification has an average latency of 180ms on standard cloud CPU nodes. The business SLA mandates an end-to-end response time under 100ms. Detail the optimization pipeline. | pass→pass | 28,055 | 26,268 | -6% | 1 | 1 | 0% | 3,847 | 4,665 | +21% | 0 | 0 | — |
▸case-17 Before deploying an AI model that predicts loan default risks, the compliance officer requires proof that the model is both fair and interpretable. Outline the compliance validation checklist. | pass→pass | 22,179 | 27,633 | +25% | 1 | 1 | 0% | 2,955 | 4,285 | +45% | 0 | 0 | — |
▸case-18 Four ML engineers are working on different model architectures for image segmentation. Currently, each engineer stores model weights and metric plots in local directories on their laptops, making comparison difficult. Provide a centralized solution. | pass→pass | 16,882 | 19,835 | +17% | 1 | 1 | 0% | 2,672 | 3,854 | +44% | 0 | 0 | — |
▸case-19 A speech-to-text model runs in 400ms on server CPUs. The user experience requires real-time streaming audio transcription under 100ms. What strategy should be adopted? | pass→pass | 14,770 | 19,458 | +32% | 1 | 1 | 0% | 2,403 | 3,419 | +42% | 0 | 0 | — |
▸case-20 We are building a multi-modal search engine that embeds text and images. We need to load open-weight CLIP and BERT models, modify their projection layers in Python, and run training sweeps. Recommend the primary software libraries. | pass→pass | 26,184 | 15,898 | -39% | 1 | 1 | 0% | 3,220 | 3,215 | -0% | 0 | 0 | — |
▸case-21 A team presents a computer vision model for automated defect inspection with 98% top-1 accuracy. They claim no further testing is needed before production rollout. What critical model evaluation dimensions are missing? | fail→fail | 13,161 | 19,226 | +46% | 1 | 1 | 0% | 2,089 | 2,448 | +17% | 0 | 0 | — |
▸case-22 To deploy an updated sentiment analysis model safely, how should traffic routing and metrics collection be structured to compare old versus new model performance under live production load? | pass→pass | 19,082 | 27,707 | +45% | 1 | 1 | 0% | 3,148 | 4,037 | +28% | 0 | 0 | — |
▸case-23 An AI team is starting an exploratory data analysis and model architecture search project. They need a tool to run code interactively, plot feature distributions, and prototype PyTorch code snippets before committing production scripts. Which environment fits this requirement? | pass→pass | 12,532 | 11,474 | -8% | 1 | 1 | 0% | 1,161 | 1,245 | +7% | 0 | 0 | — |