Twitch's A/B Testing using Google’s Staged Rollouts
Original post A/B Testing using Google’s Staged Rollouts
Article Summary
Ben Weber from Twitch discovered that Google Play's staged rollouts break standard A/B testing assumptions. Users who update early aren't random—they're systematically more engaged.
Twitch's Science Team wanted to use Google Play's staged rollout feature to A/B test major Android app changes without complex branching logic. But when they analyzed the data, they found significant biases: users in the test group were already more active before the experiment even started, violating the core assumption of randomized sampling.
Key Takeaways
- Test group users showed higher engagement BEFORE the experiment started
- Adoption took 2 weeks to reach 10%, only 8.3% updated within 1 week
- Used CausalImpact time-series and difference-in-differences to correct for bias
- Found +4-5% session increase despite biased sampling (95% confidence)
- Prior update frequency predicted group assignment, not country or device
Staged rollouts enable testing without complex branching code, but require time-series analysis and difference-in-differences methods instead of standard A/B testing to account for self-selection bias.
Continue reading
This page summarizes “A/B Testing using Google’s Staged Rollouts” by Twitch, published Jan 12, 2017. All credit for the original work goes to the authors.
