How we moved 1,500 Android screenshot tests to Roborazzi
Article Summary
Zachary Wander from Thumbtack just migrated 1,500 screenshot tests off Firebase Test Lab. The migration wasn't smooth, but the payoff in speed and developer experience made it worth every debugging session.
Thumbtack's Android team moved their entire screenshot testing suite from Firebase Test Lab to Roborazzi, a JVM-based framework that runs locally. The shift promised faster CI times and easier local testing, but came with unexpected challenges around test isolation, cross-platform rendering differences, and stale cache issues.
Key Takeaways
- Ported tests in batches of 20 to manage screenshot comparison workload
- Shadow rendering differed between macOS and Debian, breaking comparisons
- Test isolation issues caused font size bugs from parallel execution
- Undocumented robolectric.properties configs simplified test suite setup
- CI runtimes dropped significantly after migration completed
Moving 1,500 screenshot tests to Roborazzi required solving tricky cross-platform rendering and test isolation issues, but delivered faster CI and dramatically improved local testing workflows.
About This Article
Robolectric's caching system had a race condition. Deleted screenshots would reappear in the output folder after tests ran, making it hard to know which files needed to be committed to git.
Zachary Wander's team added a Gradle task that deletes Roborazzi's cache directory as a dependency to AbstractTestTask. This prevents old screenshots from being restored from the build cache.
The fix removed the need for manual cleanup and kept stale images out of git diffs when migrating 1,500 screenshot tests. Build times stayed about the same.