Flexible Rule-Based Decomposition and Metadata Independence in Modin: A Parallel Dataframe System

Download PDF

Abstract

Dataframes have become universally popular as a means to flexibly represent data in various stages of structure, and manipulate it using a rich set of operators—thereby becoming an essential tool in the data scientists’ toolbox. However, dataframe systems, such as pandas, scale poorly—and are non-interactive on moderate to large datasets. We discuss our experiences developing Modin, our first cut at a parallel dataframe system, which already has users across several industries, and considerable traction within the open source GitHub community with over 1M downloads. Modin translates pandas functions into a core set of operators that are individually parallelized via a set of columnar, row-wise, and cell-wise decomposition rules that we formalize in this paper. We also introduce the notion of metadata independence to allow metadata—such as order and type information—to be decoupled from the physical representation and maintained in a lazy fashion, computed when needed. Using rule-based decomposition and metadata independence, along with careful engineering, Modin is able to support pandas operations across both rows and columns on very large dataframes—unlike Koalas and Dask DataFrames that either breakdown or are unable to support such operations, while also being much faster than pandas.