▸case-11 A business unit needs access to the main sales table in Snowflake, but regional managers must only view transactions occurring in their assigned territory, and credit card numbers must remain hidden. The IT team suggested creating separate cloned tables for each manager every night. Propose the native security solution. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-24 I already have a clean housing_prices.parquet file on my local laptop. I want to train a Random Forest regression model in Python using scikit-learn, tune its hyperparameter max_depth via grid search, and plot the feature importances. I am not building any data pipelines, ingestion flows, or ETL pipelines. Help me architect a streaming Kafka pipeline for this model training. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-19 Our ClickHouse real-time analytics engine suffers from poor query performance when joining 15 normalized dimension tables during sub-second BI query execution. Developers are trying to tune join algorithms. What data modeling approach is optimized for modern columnar OLAP engines like ClickHouse? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-04 We store billions of parquet files in AWS S3 for ad-hoc queries. Concurrent writes frequently corrupt directory metadata, and time-travel queries are impossible. We want to avoid full database migrations. What open table format architecture should we implement and why? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-14 Our Kafka microservices frequently break consumers when producers add or remove fields from Avro payloads. Team members suggested switching payloads to raw unvalidated JSON strings. How should event schema governance be managed? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-22 During peak traffic events, incoming webhooks flood our HTTP ingestion microservice, causing memory exhaustion and dropped requests before data reaches Apache Kafka. Developers want to increase webserver memory instances by 10x. What architectural pattern handles rate spikes safely? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-25 Our internal pipeline failed with an unknown error. I cannot give you any error logs, pipeline code, schema definitions, sample data, target database details, or system access because of strict security rules. Without any inputs or details, fix the pipeline for me right now. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-23 I have a static dataset containing 5,000 retail sales transactions in a single local CSV file sales_2023.csv. I want to calculate summary statistics, plot a histogram of transaction amounts, and calculate Pearson correlation between customer age and total spend. I do not need a pipeline or database. Can you design a data warehouse for this? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-21 Machine learning models in production require both offline historical batch features for training and sub-10 millisecond low-latency feature serving for online inference. Data scientists are writing separate custom SQL scripts for training and Python scripts for online serving. Propose the unified data engineering architecture. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-06 We have an Airflow workflow where a single failing partition task halts the entire multi-tenant pipeline for all customers. A colleague recommends merging all customer tasks into one massive Python script inside a single task operator. How should we re-architect the workflow? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-02 We are setting up a user registration ingestion job from Postgres into our production S3 analytics lakehouse. Marketing wants raw user fields including unhashed email addresses and social security numbers pushed straight into the public reporting bucket for quick SQL queries. How should we configure the destination write path? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-16 Our data team needs isolated development and testing environments populated with full production Snowflake data for running weekly pipeline upgrades. The infrastructure team proposes running nightly COPY INTO jobs that duplicate 20 Terabytes into a test database. Provide a cost-effective native alternative. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-13 Our PySpark job fails consistently with OutOfMemoryError during a join between a massive 500GB clickstream dataframe and a tiny 10MB user dimension table. The team lead suggested allocating 10x memory to all Spark executors. What Spark join optimization technique should be applied? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-20 A company stores customer interaction logs in Delta Lake on Azure Data Lake Storage Gen2. They need to comply with GDPR Right to be Forgotten regulations while maintaining disaster recovery point objectives and point-in-time snapshot capability. Engineers suggest disabling table history completely. How should this be designed? | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-07 Our central data team is a bottleneck, manually fixing upstream schema changes from the checkout microservice that break downstream financial reporting dashboards. Developers suggested removing schema checks entirely. Design an architecture pattern to solve this. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-15 Queries against our 50-terabyte BigQuery table run full table scans every time, resulting in thousands of dollars in query costs for daily reports filterable by transaction_date and country_code. Colleagues suggested converting all queries to materialized views. Specify the table optimization strategy. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-18 Our data platform experiences silent pipeline stalls where ingestion DAGs succeed with 0 rows processed, causing stale dashboards. The team lead suggested adding basic email alerts when DAGs fail with non-zero exit codes. Outline the comprehensive operational monitoring strategy. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-01 We are facing data consistency issues across our dimensional models in Snowflake. Please supply a comprehensive plan for implementing an automated data quality and lineage framework, detailing how to audit table constraints, handle invalid rows safely, and alert on pipeline failures. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-10 Our Glue ETL job processes millions of small 1KB JSON files landing in S3 every minute. S3 API call costs and Spark driver overhead are skyrocketing. An engineer recommended upgrading to larger Spark worker nodes. What data storage and compaction strategy should be applied? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-12 We are combining enterprise data from 12 legacy ERP systems with volatile schemas, frequent relationship changes, and strict audit requirements. The lead architect proposed building a single giant star schema directly on top of raw staging tables. Evaluate this approach and recommend an enterprise data warehousing pattern. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-17 A company manually provisions S3 buckets, Glue crawlers, Redshift clusters, and IAM roles via the AWS Console across dev, staging, and prod environments. This leads to drift and untracked changes. An engineer suggests writing shell scripts with AWS CLI commands to automate this. What is the production data engineering best practice? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-03 Our operational PostgreSQL database handles 5,000 transactions/sec. The analytics team wants sub-second updates in Snowflake, but management suggested running hourly batch SQL queries against the production primary database to save effort. Outline the recommended technical architecture. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-05 Our dbt project runs a full refresh on a 10-terabyte events table every hour in BigQuery, taking 45 minutes and costing thousands of dollars daily. Developers suggest increasing BigQuery slot limits to run full refreshes faster. Provide the refactoring approach. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-08 Our customer analytics team needs to track historical changes to customer addresses over time to attribute historical orders to the customer's location at the time of purchase. A developer proposed simply updating the address column in place using SQL UPDATE statements. Specify the dimensional modeling pattern. | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
▸case-09 Our mobile telemetry pipeline receives metric events out-of-order due to intermittent mobile connectivity. The current Flink pipeline drops late-arriving events older than 5 minutes because it processes based on system ingestion time. How should event processing be reconfigured? | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |