Magma: A high data density storage engine used in Couchbase

Download PDF

Abstract

We present Magma, a write-optimized high data density key-value storage engine used in the Couchbase NoSQL distributed document database. Today’s write-heavy data-intensive applications like ad-serving, internet-of-things, messaging, and online gaming generate massive amounts of data. As a result, the requirement for storing and retrieving large volumes of data has grown rapidly. Distributed databases that can scale out horizontally by adding more nodes can be used to serve the requirements of these internet-scale applications. To maintain a reasonable cost of ownership, we need to improve storage efficiency in handling large data volume per node, such that we don’t have to rely on adding more nodes. Our current generation storage engine, Couchstore is based on a log-structured append-only copy-on-write B+Tree architecture. To make substantial improvements to support higher data density and higher write throughput, we needed a storage engine architecture that lowers write amplification and avoids compaction operations that rewrite the whole DB files periodically. We introduce Magma, a hybrid key-value storage engine that combines LSM Trees and a segmented log approach from log-structured filesystems. We present a novel approach in performing garbage collection of stale document versions avoiding index lookup during log segment compaction. This is the key to achieving storage efficiency for magma and eliminates the need for random I/Os during compaction. Magma offers significantly lower write amplification, scalable incremental compaction, and lower space amplification while not regressing on the read amplification. Through the efficiency improvements, we are able to improve the single machine data density supported by the Couchbase Server by 3.3x, thereby reducing the cost of storing the same amount of data. We also reduced the memory to persistent data size ratio requirement by 10x.