A Massively Multi-User Datastore Synced with Mobile Clients
Article Summary
Tad Book from Square reveals how they built a datastore that handles massive merchant catalogs while keeping mobile devices in sync, even when offline for days. The secret? An append-only architecture that makes the impossible possible.
Square's merchant catalog system needed to handle huge datasets across offline mobile devices, support both granular queries and bulk rewrites, and provide flexibility without constant schema migrations. Their solution combines entity-attribute-value modeling with append-only storage on sharded MySQL to achieve database-like transactions without actual database transactions.
Key Takeaways
- Append-only model enables consistent paging even during concurrent writes
- Syncable constraints validate data on mobile clients before server submission
- Transaction rollbacks work by marking deletions, preserving full audit history
- Location-aware clients receive only relevant data subsets automatically
- User-defined types and attributes extend schema without migrations or deploys
Square achieved per-user transactionality and offline sync at scale by treating all writes as immutable append operations, eliminating traditional database transaction overhead.
About This Article
Square needed to handle large bulk catalog operations through their API, including complete dataset rewrites. At the same time, they had to keep the web UI responsive for queries and support multiple concurrent users without slowdowns.
Tad Book's team built an entity-attribute-value model that modifies attributes individually. When you update a single attribute, it creates just one new entry instead of rewriting the whole object. This cut down on write overhead significantly.
The append-only design lets merchants see their catalog at any point in its history and check what changed after a specific moment. Reverting changes or restoring old catalog states becomes straightforward and doesn't cause side effects.