How Optimizing Memory Management with LMDB Boosted Performance on Our API Service
Article Summary
Pinterest's API team just freed up 4.5GB of memory per host with a clever database swap. The result? Fewer servers, happier users, and a masterclass in optimization.
The Pinterest engineering team tackled a memory bottleneck in their NGAPI service that handles all first-party client requests. Their multi-process architecture was duplicating configuration data across every process, creating massive memory pressure at scale.
Key Takeaways
- Reduced memory usage by 4.5% per host using LMDB instead of per-process copies
- Increased processes per host from 64 to 66, shrinking overall fleet size
- Evaluated three mmap solutions: Marisa Trie, Keyvi, and LMDB won
- Top 50 config files accounted for 90% of duplicate memory consumption
- Zero latency impact: LMDB reads match native Python lookup speeds
By switching to memory-mapped databases for shared configuration data, Pinterest reduced memory footprint and increased request capacity per host without any code refactoring or latency penalties.
About This Article
Pinterest's NGAPI platform had a memory problem. Each process in their multi-process gevent architecture loaded configuration data independently, which meant the system couldn't scale to handle billions of requests efficiently.
Pinterest Engineering switched to LMDB, an embedded key-value store that uses memory-mapped files. Instead of each process loading its own JSON files, they set up a single shared database per host. A lightweight Python sidecar handles updates, and all processes read from that one instance.
After the migration, Pinterest could run 66 processes per host instead of 64 by cutting per-process memory overhead. The fleet handled higher request volumes without slowing down read latency, which stayed as fast as native Python lookups.