Lux: Always-on Visualization Recommendations for Exploratory Dataframe Workflows

Download PDF

Abstract

Exploratory data science largely happens in computational notebooks with dataframe APIs, such as pandas, that support flexible means to transform, clean, and analyze data. Yet, visually explor- ing data in dataframes remains tedious, requiring substantial programming effort for visualization and mental effort to determine what analysis to perform next. We propose LUX, an always-on framework for accelerating visual insight discovery in dataframe workflows. When users print a dataframe in their notebooks, LUX recommends visualizations to provide a quick overview of the patterns and trends and suggests promising analysis directions. LUX features a high-level language for generating visualizations on de- mand to encourage rapid visual experimentation with data. We demonstrate that through the use of a careful design and three sys- tem optimizations, LUX adds no more than two seconds of overhead on top of pandas for over 98% of datasets in the UCI repository. We evaluate LUX in terms of usability via interviews with early adopters, finding that LUX helps fulfill the needs of data scientists for visualization support within their dataframe workflows. LUX has already been embraced by data science practitioners, with over 2.8k stars on Github.