Failing Fast with Reliable and Actionable System-Level Tests
Article Summary
Skyscanner's test suite had a 99% pass rate. Sounds great, right? Wrong. Those false failures were costing hours of productivity and masking real production issues.
Skyscanner's Web Publishing Squad tackled the industry-wide problem of flaky system-level tests. Test Engineer Ramsay Ashby shares how they transformed their Selenium test suite from 'crying wolf' to a reliable safety net.
Key Takeaways
- Improved test pass rate from 93% to near-perfect reliability in months
- Added test-level retry logic to embrace imperfections, not battle them
- All test builds now complete in under 5 minutes including mobile
- Achieved 7-day consecutive 100% pass rate streak without masking issues
By strategically embracing imperfections with robust retry logic, Skyscanner eliminated false positives while maintaining confidence that every test failure signals a genuine product issue.
About This Article
Skyscanner's Web Publishing Squad kept hitting test failures that weren't real problems. Network hiccups, flaky third-party services, and unstable dependencies were the culprits. These false alarms wasted time that should have gone toward finding actual bugs.
The team added retry logic at two levels: within individual test interactions and across the test framework itself. This approach filtered out transient failures without hiding real issues.
Test pass rates climbed from around 90% to a sustained 93.2% minimum over 60 days. Build execution stayed under 5 minutes across all environments, including mobile tests on Sauce Labs.