Goku: A Schemaless Time Series Database for Large Scale Monitoring at Pinterest

Download PDF

Abstract

Engineers rely heavily on observability tools to monitor their busi-Engineers rely heavily on observability tools to monitor their business and system metrics and set up alerting on it. A reliable and efficient monitoring system is very important for development ve-efficient monitoring system is very important for development velocity. In this paper, we introduce Goku, a time series database (TSDB) we built from the ground up at Pinterest. Over the years, we have studied user patterns and common requests to constantly evolve Goku to store and serve the use cases at Pinterest with high efficiency and reduced costs. At its core, Goku uses tiered storage to store new and frequently queried metrics data in memory while leveraging solid state drive (SSD) and hard disks (HDD) for older data. Goku aggregates metrics data at write time while also rolling up datapoints with lower time granularity to low latency to certain use cases. Goku also supports modifying configurations on metrics data like time to live (TTL), rollup granularity, backfilling capability, etc. Using multiple replicas and AWS S3 as backup, Goku is highly available and fault tolerant.