Pinterest: How Optimizing Memory Management with LMDB Boosted Performance on Our API Service
Original post How Optimizing Memory Management with LMDB Boosted Performance on Our API Service
Article Summary
Pinterest's API team just freed up 4.5GB of memory per host with a clever database swap. The result? Fewer servers, happier users, and a masterclass in optimization.
The Pinterest engineering team tackled a memory bottleneck in their NGAPI service that handles all first-party client requests. Their multi-process architecture was duplicating configuration data across every process, creating massive memory pressure at scale.
Key Takeaways
- Reduced memory usage by 4.5% per host using LMDB instead of per-process copies
- Increased processes per host from 64 to 66, shrinking overall fleet size
- Evaluated three mmap solutions: Marisa Trie, Keyvi, and LMDB won
- Top 50 config files accounted for 90% of duplicate memory consumption
- Zero latency impact: LMDB reads match native Python lookup speeds
By switching to memory-mapped databases for shared configuration data, Pinterest reduced memory footprint and increased request capacity per host without any code refactoring or latency penalties.
Continue reading
What This Article Covers
Pinterest's NGAPI platform had a memory problem. Each process in their multi-process gevent architecture loaded configuration data independently, which meant the system couldn't scale to handle billions of requests efficiently.
Pinterest Engineering switched to LMDB, an embedded key-value store that uses memory-mapped files. Instead of each process loading its own JSON files, they set up a single shared database per host. A lightweight Python sidecar handles updates, and all processes read from that one instance.
After the migration, Pinterest could run 66 processes per host instead of 64 by cutting per-process memory overhead. The fleet handled higher request volumes without slowing down read latency, which stayed as fast as native Python lookups.
This page summarizes “How Optimizing Memory Management with LMDB Boosted Performance on Our API Service” by Pinterest, published Oct 11, 2022. All credit for the original work goes to the authors.
