Measuring Product Impact Without A/B Testing: How Discord Used the Synthetic Control Method for Voice Messages
Article Summary
Discord wanted to measure Voice Messages impact, but A/B testing would break the feature. Network effects meant users in control groups couldn't receive voice messages from treatment users.
Discord's data science team shares how they used the Synthetic Control Method to evaluate Voice Messages when traditional experimentation wasn't possible. The article walks through their methodology, from problem identification to implementation and results.
Key Takeaways
- Synthetic controls compare one treated country to a weighted mix of untreated countries
- Brazil was treatment group, synthetic control was 50% Argentina, 30% Uruguay, 20% Chile
- Method controls for observable and unobservable differences better than geo tests
- Results showed clear engagement increase after Voice Messages launched in Brazil
- Approach works when randomization is impossible or sacrifices too much precision
Critical Insight
Discord successfully measured Voice Messages impact using synthetic controls, proving the method works for network effect features that break traditional A/B tests.
