App Health Through Metric-Aware Rollouts
Article Summary
DoorDash was shipping experiments that crashed their app and degraded performance for weeks before anyone could identify the cause. They built an automated system to catch issues before customers feel the pain.
The DoorDash experimentation platform team shares how they built metric-aware rollouts: an automated guardrail system that monitors app quality during feature releases. The system standardizes metrics, auto-pauses problematic rollouts, and gives teams real-time visibility into performance impacts.
Key Takeaways
- Composite metrics track bad page loads across iOS, Android, and web in one check
- Five sub-metrics (P95 latency, errors, crashes) pinpoint exact degradation sources
- Budget framework lets teams ship with temporary regressions plus mandatory clawback plans
- Auto-pause triggers when degradation exceeds tolerance thresholds set by metric owners
- Sequential testing and hypothesis correction control false positives at scale
Critical Insight
DoorDash reduced time from experiment start to rollback by automatically detecting and pausing releases that degrade app quality beyond configured tolerance levels.
