An Automate Guide to Detect and Handle Data Drift — Motivation Data drift occurs when the distribution of input features in the production environment differs from the training data, leading to potential inaccuracies and decreased model performance.

The simple logging solution for your data science project — Originally published at https://mathdatasimplified.com on July 17, 2023. Why Use Logging in a Data Science Project? Data scientists often use the print function to debug their code. However, as the number of print statements increases, it becomes difficult to identify where the output is from due to the lack of line numbers or function names.

Learn Git through Real-Life Scenarios — Originally published at https://mathdatasimplified.com on July 1, 2023. Why Choose Git for Your Data Science Project? Git is a version control system widely used in software development, but is it the right choice for your data science project? Absolutely. Here are some reasons why Git is invaluable for data science: Version Control Scenario: You replace the current data processing technique…

An in-depth comparison between Poetry, Pip, and Conda — Originally published at https://mathdatasimplified.com on June 13, 2023. Motivation As your data science project expands, the number of dependencies also increases. To keep the project’s environment reproducible and maintainable, it’s important to use an efficient dependency management tool. Thus, I decided to compare three popular tools for dependency management: Pip, Conda…

Interactively Build and Orchestrate Data Models — Originally published at https://mathdatasimplified.com on June 5, 2023. Motivation dbt (data build tool) is a powerful tool for data transformation within the data warehouse. However, it does come with some limitations, including the following: Lack of Output Preview: With dbt core, it is not possible to preview the output of a…

How to efficiently interact with config files in Python — Originally published at https://mathdatasimplified.com on May 26, 2023. Problem In your data science project, certain values tend to change frequently, such as file names, selected features, train-test split ratio, and hyperparameters for your model.

Discover the Hidden Benefits and Drawbacks of dbt — Motivation If your organization is looking to create a data-driven product, you should consider having efficient data pipelines to: Stay competitive: Quick access to data and its analysis through efficient data pipelines accelerates decision-making and keeps you ahead of your competition. Reduce costs: The time and effort required to collect…

Faster Time to Market and Increase Efficiency — Motivation Consider this scenario: A more accurate machine learning model is developed every month and added to the main branch. To deploy the model, you must download it to your machine, package it, and deploy it.

Speed Up Iteration and Cut Computation Cost — Motivation Many data teams begin by establishing an analytics practice on data warehouses such as BigQuery. However, solely relying on BigQuery for data science workloads may not be the best approach due to various reasons: Advanced needs beyond SQL: Use cases such as data validation, visualization, and machine learning forecasting may…