DICE: Data Discovery by Example

Download PDF

Abstract

In order to conduct analytical tasks, data scientists often need to find relevant data from an avalanche of sources (e.g., data lakes, large organizational databases). This effort is typically made in an ad-hoc, non-systematic manner, which makes it a daunting endeavour. Current data discovery systems typically require the user to find relevant tables manually, usually by issuing multiple queries (e.g., using SQL). However, expressing such queries is nontrivial, as it requires knowledge of the underlying structure (schema) of the data organization in advance. This issue is further exacerbated when data resides in data lakes, where there is no predefined schema that data must conform to. On the other hand, data scientists can often come up with a few example records of interest quickly. Motivated by this observation, we developed DICE--a human-in-the-loop system for Data dIsCovery by Example--that takes user-provided example records as input and returns more records that satisfy the user intent. DICE's key idea is to synthesize a SQL query that captures the user intent, specified via examples. To this end, DICE follows a three-step process: (1) DICE first discovers a few candidate queries by finding join paths across tables within the data lake. (2) Then DICE consults with the user for validation by presenting a few records to them, and, thus, eliminating spurious queries. (3) Based on the user feedback, DICE refines the search and repeats the process until the user is satisfied with the results. We will demonstrate how DICE can help in data discovery through an interactive, example-based interaction.