A Python package to run auto optimization of a Pandas DataFrame.

Reduce DataFrame Memory Consumption by ~70% and configure dtypes automatically for best precision with pd-helper.

I released a Python Package in Beta, this is the Announcement

The project is up on PyPi here: https://pypi.org/project/pd-helper/

Also on GitHub here: https://github.com/justinhchae/pd-helper

Edit 15 April 2021: Stable release 1.0.0 is now available.

Install:

pip install pd-helper

Basic Usage:

from pd_helper.helper import optimize

if __name__ == "__main__":
   # some DataFrame, df
   df = optimize(df)

Better Usage With Multiprocessing:

from pd_helper.helper import optimizeif __name__ == "__main__":
   # some DataFrame, df
   df = optimize(df, enable_mp=True)

Specify Special Mappings:

from pd_helper.helper import optimizeif __name__ == "__main__":
   # some DataFrame, df
   special_mappings…

How to approach program design with multiprocessing?

In a recent project, I stumbled across some clever ways to boost the speed of forecasting models such as ARIMA and Facebook Prophet and shared the results with Towards Data Science. However, while the performance results and insights are interesting, my prior article primarily focuses on the high-level concept of combining multiple types of forecasting with multiprocessing and does not focus on how to approach the design of such a program. …

I recently built and deployed a data application from my Mac using Python and did not write one line of HTML, CSS, or JavaScript — all made possible with a nifty package called Streamlit.

This article contains a few notes on what worked and what did not work when deploying code with Streamlit.

I recently deployed an early alpha version of a data dashboard with Steamlit. Although the app is in its infancy, I’m happy to have it in some kind of production state and I am excited to continue improving on it. Based on my experience, this article contains some of the biggest gotchas and tips on breaking through a few final issues that you may encounter with…

How to transform a Pandas DataFrame to JSON with flare-like hierarchy to produce D3 Sunburst visualizations.

Data is king but colors are cool!

If your data and insights are worth telling, the right colors and design can make or break the crucial connection to your audiences. As a result, whether you are trying to win a competition or just trying to turn in a class assignment, chances are that colors can help make your case.

As I learned from personal experience, tricky data transformations on the back-end often put the desired visualization out of reach. For example, despite various libraries in Pandas, I was surprised to discover…

This article may be for you if you are setting up a Mac Mini M1 (2020) out of the box or need to factory restore and reinstall the OS.

I have been experimenting a ton with my Mac Mini M1 (“M1”) which means I also need to restore it from scratch after breaking too many things. After about 5 cycles of configuring and restoring, I ended up with a decent checklist to get things back up and running. I use the M1 as a secondary workstation — mostly for programming — and the big things I usually configure are Homebrew, Xcode, PyCharm, and QGIS among other tools. My workstation revolves around a MacBook which I use to control the Mini via Screen Sharing at my home network.

Preliminaries

Before trying…

Making Sense of Big Data

When forecasting time series data, sometimes you need both speed and accuracy. When working from a laptop with Python, try multiprocessing to get the best of both worlds.

This Article May be For You If:

• You are trying to optimize a multiprocessing problem in Python on your local machine
• You are forecasting time series data with Statsmodels ARIMA, Facebook Prophet, or PyTorch LSTM
• You are trying to decide whether multiprocessing is an optimal configuration with PyTorch LSTM if you don’t have access to a GPU

Have Data, Need Time

Most of the time you have plenty of data. Some of the time, you have plenty of time to analyze the data…

Making Sense of Big Data

I built a simple app that takes a csv file and returns a memory-optimized pickle file for use as a Pandas DataFrame — this story shares my experience building with Streamlit, describes the problem to be solved, and promotes a prototype of the app.

This article may be for you if you are working with large CSV files, Pandas DataFrames, and Python.

Edit on 7 April 2021: Since writing this article, I have created and deployed a Python package called pd-helper that runs an optimized version of the code discussed in this story. You can search for it as “pd-helper”…

When something is nothing, and nothing is something…for boolean data in Pandas, there is crucial difference between NaN, Null, NA, and bools — a brief on when and how to use them.

Task: Clean a Pandas DataFrame comprising boolean (true/false) values to optimize memory. A constraint is to retain all null values as nulls, i.e. don’t turn null values to False because that is a meaningful change.

Action: Explicitly transform column dtypes, i.e. use float32 instead of float64 to conserve memory and bool instead of object.

Problem: When transforming selected columns to bool, all rows evaluate to either all True…