Pinterest Urvashi Reddy

How Pinterest Designed Its Continuous Integration System to Be More Than 50% Faster

Original post How We Designed Our Continuous Integration System to Be More Than 50% Faster

Article Summary

Pinterest's Engineering Productivity team slashed CI build times by over 50% in one week. Here's the architecture that made it possible.

Pinterest's team rebuilt their Continuous Integration system for Optimus, a monorepo with 120+ Java services serving 300 monthly contributors. They moved from running all targets to intelligently selecting only what changed, using Bazel queries and custom tooling.

Key Takeaways

Critical Insight

By running only changed code with Bazel queries and parallelizing releases, Pinterest cut CI times from 52 to 19 minutes at P50 and 69 to 49 minutes at P90.

Continue reading

What This Article Covers

Problem

Pinterest's CI pipeline for Optimus had 300 monthly contributors working across 120+ Java services. Every code change triggered all targets to run, which meant expensive network calls to fetch and load dependencies even when most of the code hadn't changed.

Solution

Pinterest Engineering built build-collector, a Golang CLI that leverages Bazel's rdeps and kind query functions. It identifies which targets actually need to run based on what files changed, then spreads the work across parallel workers.

Impact

After one week of migration, P90 build times dropped from 69 minutes to 49 minutes, a 1.4x improvement. The gains came from running only affected targets and better parallelization.

The team's custom Bazel release rules create an abstraction that most developers don't even see, but it powers their entire deployment workflow.

This page summarizes “How We Designed Our Continuous Integration System to Be More Than 50% Faster” by Pinterest, published Feb 18, 2021. All credit for the original work goes to the authors.

Read the original on medium.com (opens in new tab)

AI tools for iOS

Browse all AI tools →