The LinkedIn Android Data Pipeline
Article Summary
LinkedIn rebuilt their Android data pipeline from scratch for their flagship app. The result? A system that handles everything from feed updates to messaging while keeping data consistent across screens.
Karthik Ramgopal, Distinguished Engineer at LinkedIn, breaks down how they designed a universal data pipeline that every feature team could use. The system needed to work flawlessly on low-end devices with flaky networks while maintaining speed and testability.
Key Takeaways
- Custom JSON parser using Jackson streaming avoids reflection, dramatically improving app launch time
- Fission cache stores individual sub-models instead of full responses, reducing duplication
- Auto-generated code from Pegasus schemas eliminates manual parsing and enforces backward compatibility
- Consistency Manager synchronizes model changes across screens with recursive change notifications
- Forked Volley to add streaming support, enabling parsing before download completes
LinkedIn's data pipeline combines code generation, intelligent caching, and streaming to deliver fast performance even on constrained Android devices.
About This Article
Android devices have limited memory, and reflection-based JSON parsing libraries like Gson create significant overhead when apps start up. This directly slows down launch times on low-end devices.
LinkedIn built a code generator that creates custom Jackson stream parsing code. It removes reflection overhead and cuts down intermediate memory allocations. The generated code also uses Android-specific optimizations like ArrayMap and public final fields to keep the method count low.
Apps start faster and use less memory on constrained devices. The approach also sidesteps the 65k method limit by generating fields instead of getter methods.