Failing Fast with Reliable and Actionable System-Level Tests
Article Summary
Skyscanner's test suite had a 99% pass rate. Sounds great, right? Wrong. Those false failures were costing hours of productivity and masking real production issues.
Skyscanner's Web Publishing Squad tackled the industry-wide problem of flaky system-level tests. Test Engineer Ramsay Ashby shares how they transformed their Selenium test suite from 'crying wolf' to a reliable safety net.
Key Takeaways
- Improved test pass rate from 93% to near-perfect reliability in months
- Added test-level retry logic to embrace imperfections, not battle them
- All test builds now complete in under 5 minutes including mobile
- Achieved 7-day consecutive 100% pass rate streak without masking issues
Critical Insight
By strategically embracing imperfections with robust retry logic, Skyscanner eliminated false positives while maintaining confidence that every test failure signals a genuine product issue.
