Meta's Next-generation Realtime Monitoring and Analytics Platform

Download PDF

Abstract

Unlike traditional database systems where data and system availability are tied together, a wide class of systems targeting realtime monitoring and analytics over structured logs exists where these properties can be decoupled. In these systems, availability and freshness of data is often more important than perfectly complete answers. One such system is Scuba. Historically, Scuba has favored system availability along with speed and freshness of results over data completeness and durability. While these choices allowed Scuba to grow from terabyte scale to petabyte scale and continue onboarding a variety of use cases, they also came at an operational cost of dealing with incomplete data and managing data loss. In this paper, we present the next generation of Scuba’s architecture, codenamed Kraken, which decouples the storage management from the query serving system and introduces a single, durable source of truth. We also describe the journey of how we deployed Kraken into full production as we gradually turned off the older system with no user-visible down time.