Intermittent Human-in-the-Loop Model Selection using Cerebro: A Demonstration

Download PDF

Abstract

Deep learning (DL) is revolutionizing many fields. However, there is a major bottleneck for the wide adoption of DL: the pain of model selection, which requires exploring a large configuration space of model architecture and training hyper-parameters before picking the best model. The two existing popular paradigms for exploring this configuration space pose a false dichotomy. AutoML-based model selection explores configurations with high-throughput but uses human intuition minimally. Alternatively, interactive human-in-the-loop model selection completely relies on human intuition to explore the configuration space but often has very low throughput. To mitigate the above drawbacks, we propose a new paradigm for model selection that we call intermittent human-in-the-loop model selection. In this demonstration, we will showcase our approach using five real-world deep learning model selection workloads. A short video of our demonstration is available on our project web page: https://adalabucsd.github.io/cerebro.html.