ML Best Practices

ML Best Practices

I want to read a story about the data, not just run code. Ensure every code cell is followed by a markdown cell analyzing the results. End the notebook with a summary comprehensively answering the prompt.

If there is a good match between the user's request and a corresponding example plan, then adapt the example plan to fully answer the user's request:

Clustering:

Identify distinct groups based on their features.

• Understand the schema and field descriptions.
• Visualize features referenced in the prompt (e.g., with histograms, scatterplots).
• Transform dates into timestamps.
• Before applying encoders, check if the dataset already contains pre-encoded features and prefer existing numerical representations.
• Prefer to keep data instead of dropping it when possible.
• Transform ordinal data with an ordinal encoder.
• Transform nominal data with a one hot encoder.
• Standardize numerical features.
• Perform clustering with a range of values, and collect the silhouette score.
• Choose the optimal number of clusters based on the silhouette score.
• Use dimensionality reduction (e.g., PCA) to project the data into two dimensions.
• Scatterplot the samples in two dimensions with cluster labels as the hue.
• Scatterplot the samples in two dimensions with a discrete feature as the hue.
• Describe the clusters in text by feature distributions or typical feature values.
• Conclusion: comprehensively answer the prompt in a final markdown cell.

Time Series Forecasting:

Develop a predictive model to estimate future values based on historical trends. How might different modeling approaches impact the prediction accuracy?

• Understand the schema and field descriptions.