Spotting Latency Regressions Ahead of Time at Teams Mobile
Article Summary
Microsoft Teams Mobile merges 50+ commits daily from 350+ developers. How do they catch performance regressions before users feel the pain?
The Teams Mobile team built "Bouncer," an automated performance testing system that detects latency regressions as small as 15% on a commit-by-commit basis. This deep dive from Microsoft engineer Saumye Srivastava reveals the technical architecture behind catching slowdowns before they ship.
Key Takeaways
- Runs before/after performance tests on every mainline commit using physical Pixel 4A devices
- Achieved stable results by rooting devices, locking CPU frequencies, and mocking API responses locally
- Reduced false positives dramatically: performance tests now mostly green, red means real regression
- Uses coefficient of variation under 0.15 to ensure test stability across iterations
- Executes 100k+ test runs per quarter with median comparison across 9 iterations
Critical Insight
Teams Mobile now catches performance regressions within hours of code merge instead of during production rollout, letting engineers focus on speed improvements rather than firefighting.
