How Square Supercharges Continuous Integration with Gradle
Original post Supercharging Continuous Integration with Gradle
Article Summary
Paul Hundal and Pablo Baxter from Square reveal how they slashed CI analysis time by 80% on a monorepo with 5,000+ modules. Their secret? Getting creative with Gradle's internals.
Square's Android team runs 1,200 CI shards over 200 times daily, with builds averaging 65 minutes. As their codebase exploded from 3,500 to 5,000+ modules, they needed radical optimization. This deep dive shows how they combined shard avoidance, Gradle Tooling API, and custom caching to transform their pipeline.
Key Takeaways
- Shard avoidance skips 600 of 1,200 shards per build, saving 1,800 to 13,200 minutes
- Custom Avoidance Cache hits 92% of the time by keying build file hashes
- Analysis time dropped from 11 minutes to 2 minutes (80% reduction)
- Only 8% of PRs actually modify build files, making caching highly effective
Square reduced their CI avoidance analysis from 11 minutes to 2 minutes by building custom tooling around Gradle's Tooling API and implementing intelligent caching across all repository branches.
Continue reading
What This Article Covers
Square's Android CI pipeline had a serious performance issue. The configuration phase took 3-5 minutes per shard avoidance analysis run across 1,200 shards, making it the slowest part of building the graph.
Paul Hundal and Pablo Baxter built Affected Paths, a tool that uses the Gradle Tooling API to talk directly with the Gradle daemon. It extracts custom models without writing metadata to disk, which cuts down on file I/O and memory overhead.
Affected Paths combined with an Avoidance Cache keyed on build file hashes stored in S3 gave Square a 92% cache hit rate. Avoidance analysis time dropped from 11 minutes to 2 minutes, an 80% improvement that gets developers feedback on code changes much faster.
This page summarizes “Supercharging Continuous Integration with Gradle” by Square, published Jul 27, 2023. All credit for the original work goes to the authors.
