A Perfect Combo to Quickly Iterate on Your DS Project — Motivation Imagine your data pipeline looks similar to the graph below. The pink box represents a stage, which is an individual data process. Dependencies are the files that a stage depends on such as parameters, Python scripts, or input data. Now imagine Dependencies 2 changes. The standard approach is to rerun…

Make Your Pipelines Easier to Support and Maintain — Motivation In engineering, maintainability is the ease with which a product can be maintained to: Meet new requirements Cope with a changing environment Improve the performance of the product In a data science project, it is essential to build a maintainable pipeline because: The characteristics of data can change frequently Data…

How to Extract Datetime From Text and Data From Datetime — Motivation Datetime features are important for time series and forecasting. However, you don’t always have clean datetime features to work with. Wouldn’t it be nice if you can automatically extract datetime from text and data from datetime? In this article, you will learn 4 tools to do exactly that. datefinder: Automatically Find Dates and Time in a Python String

Create Clean and Robust Tests with Property-Based Testing — Motivation Imagine you are trying to figure out whether the function processing_fn is working properly. You use pytest to test the function with an example. The test passed, but you know that one example is not enough. …

Store Your Workflow in GitHub and Execute it in a Docker Container — Motivation Have you ever wanted to store and execute your workflow in two locations but found it difficult to do so? Wouldn’t it be nice if you easily store and execute your code in different locations? Prefect allows you to easily store and execute your workflows in all…

Process Your Text in One Line of Python Code — Motivation Before using text for an ML model or for visualization, you might want to clean your text by: replacing NA values with empty spaces lowercasing all text removing digits removing punctuation removing stopwords removing whitespace It can take a while for you to write all of the steps above to…

Gain Insights from a Comparison in Three Lines of Code — Motivation Imagine you are trying to determine if there is a significant difference in the median total payment between two cities that a taxi picks up. You decide to create a box plot to observe the total fare per pickup city. This plot gives you some ideas about…

Build a Robust Data Pipeline in Python with Deepchecks and Prefect — Motivation A data science project includes primary components such as getting data, processing data, training an ML model, then putting it into production. It is important to validate each component’s outputs to ensure each component works properly before feeding its outputs to the next component in the workflow. …