How Microsoft Spots Latency Regressions Ahead of Time at Teams Mobile
Original post Spotting Latency Regressions Ahead of Time at Teams Mobile
Article Summary
Microsoft Teams Mobile merges 50+ commits daily from 350+ developers. How do they catch performance regressions before users feel the pain?
The Teams Mobile team built "Bouncer," an automated performance testing system that detects latency regressions as small as 15% on a commit-by-commit basis. This deep dive from Microsoft engineer Saumye Srivastava reveals the technical architecture behind catching slowdowns before they ship.
Key Takeaways
- Runs before/after performance tests on every mainline commit using physical Pixel 4A devices
- Achieved stable results by rooting devices, locking CPU frequencies, and mocking API responses locally
- Reduced false positives dramatically: performance tests now mostly green, red means real regression
- Uses coefficient of variation under 0.15 to ensure test stability across iterations
- Executes 100k+ test runs per quarter with median comparison across 9 iterations
Teams Mobile now catches performance regressions within hours of code merge instead of during production rollout, letting engineers focus on speed improvements rather than firefighting.
Continue reading
What This Article Covers
Teams Mobile's Android codebase gets 50+ commits daily from 350+ developers. When latency regressions show up in production dashboards, it's hard to figure out which code change caused them. This often delays releases and fixes.
Saumye Srivastava's team built Bouncer to run instrumentation tests before and after each mainline commit. They use Microsoft Hydralab on physical Pixel 4A devices, comparing median latency across 9 iterations. The tool flags regressions when they exceed 10-15%.
False positives dropped significantly. Performance tests went from consistently failing to mostly passing. When a red flag does appear, engineers can trust it's a real regression, so they don't have to manually adjust thresholds after every alert.
This page summarizes “Spotting Latency Regressions Ahead of Time at Teams Mobile” by Microsoft, published Jul 20, 2021. All credit for the original work goes to the authors.
