The LinkedIn Android Data Pipeline

LinkedIn rebuilt their Android data pipeline from scratch for their flagship app. The result? A system that handles everything from feed updates to messaging while keeping data consistent across screens.

Karthik Ramgopal, Distinguished Engineer at LinkedIn, breaks down how they designed a universal data pipeline that every feature team could use. The system needed to work flawlessly on low-end devices with flaky networks while maintaining speed and testability.

Key Takeaways

• Custom JSON parser using Jackson streaming avoids reflection, dramatically improving app launch time
• Fission cache stores individual sub-models instead of full responses, reducing duplication
• Auto-generated code from Pegasus schemas eliminates manual parsing and enforces backward compatibility
• Consistency Manager synchronizes model changes across screens with recursive change notifications
• Forked Volley to add streaming support, enabling parsing before download completes

The article reveals specific Android optimizations like using ArrayMap and generating fields instead of getters to avoid the 65k method limit.