A Massively Multi-User Datastore Synced with Mobile Clients
Article Summary
Tad Book from Square reveals how they built a datastore that handles massive merchant catalogs while keeping mobile devices in sync, even when offline for days. The secret? An append-only architecture that makes the impossible possible.
Square's merchant catalog system needed to handle huge datasets across offline mobile devices, support both granular queries and bulk rewrites, and provide flexibility without constant schema migrations. Their solution combines entity-attribute-value modeling with append-only storage on sharded MySQL to achieve database-like transactions without actual database transactions.
Key Takeaways
- Append-only model enables consistent paging even during concurrent writes
- Syncable constraints validate data on mobile clients before server submission
- Transaction rollbacks work by marking deletions, preserving full audit history
- Location-aware clients receive only relevant data subsets automatically
- User-defined types and attributes extend schema without migrations or deploys
Square achieved per-user transactionality and offline sync at scale by treating all writes as immutable append operations, eliminating traditional database transaction overhead.
