How we raised mobile end-to-end test stability to 98%
Article Summary
Michael Garfinkle from Shopify reveals how they went from pulling E2E tests from CI entirely to achieving 98% stability. The secret? Stop fighting flakiness and rebuild the framework that creates it.
Shopify's mobile app E2E tests had become so unreliable they were blocking more good PRs than bad ones, forcing the team to remove them from CI checks. Instead of patching their Appium-based setup, they built an opinionated wrapper with computer vision and a strict builder API that makes flaky tests hard to write.
Key Takeaways
- Test stability jumped from 50% to 98% after the rebuild
- Computer vision (PaddleOCR + OpenCV) finds elements like users do, not via Test IDs
- Builder API forces assertions at every step, preventing silent failures
- AI agents write correct tests first try due to predictable grammar
- Pre-promotion gate runs tests multiple times before allowing into blocking suite
By replacing a flexible but footgun-prone API with an opinionated framework that enforces good patterns, Shopify raised mobile E2E test stability from 50% to 98%.
About This Article
Shopify's mobile app E2E tests were rejecting more good pull requests than bad ones. Tests kept failing because screens took extra seconds to load, and developers had started relying on hardcoded pauses instead of proper waits.
Michael Garfinkle's team built a wrapper around Appium with a strict builder-style API. It requires assertions after every action. They added PaddleOCR for OCR and OpenCV for icon matching so elements could be found visually instead of through Test IDs.
Test stability jumped from 50% to 98% after the rebuild. The remaining failures were expected ones like network problems and simulator boot failures. E2E tests could finally go back to blocking CI checks.