go back
go back
Volume 14, No. 12
AutoExecutor: Predictive Parallelism for Spark SQL Queries
Authors:
Rathijit Sen (Microsoft), Abhishek Roy (Microsoft), Alekh Jindal (Keebo), Rui Fang (Microsoft), Jeff Zheng (Microsoft), Xiaolei Liu (Microsoft), Ruiping Li (Microsoft)
Abstract
Right-sizing resources for query execution is important for cost-efficient performance, but estimating how performance is affected by resource allocations, upfront, before query execution is difficult. We demonstrate AutoExecutor, a predictive system that uses machine learning models to predict query run times as a function of the number of allocated executors, that limits the maximum allowed parallelism, for Spark SQL queries running on Azure Synapse.
PVLDB is part of the VLDB Endowment Inc.
Privacy Policy