Abstract

In this paper we present techniques to incrementally harvest and query arbitrary metadata from machine learning pipelines, without disrupting agile practices. We center our approach on the developerfavored technique for generating metadata — log statements —leveraging the fact that logging creates context. We show how hindsight logging [8] allows such statements to be added and executed post-hoc, without requiring developer foresight. Relational views of incomplete metadata can be queried to dynamically materialize new metadata in bulk and on demand across multiple versions of work!ows. This is done in a “metadata later” style, o" the critical path of agile development. We realize these ideas in a system called FlorDB and demonstrate how the data context framework covers a range of both ad-hoc metadata as well as special cases treated today by bespoke feature stores and model repositories. Through a usage scenario—including both ML and human feedback—we illustrate how the component techniques come together to resolve classic software engineering trade-o"s between agility and discipline.