Pinterest: Build an end to end JSON logging system for clients apps
Original post Build an end to end JSON logging system for clients apps
Article Summary
Liang Ma and Wei Zhu from Pinterest built a JSON logging system that catches production issues hours before traditional metrics even notice. Here's how they gave mobile teams real-time visibility into 300 billion ideas.
After a critical iOS memory incident exposed blind spots in client monitoring, Pinterest's engineering team created an end-to-end logging pipeline. The system needed zero backend work from developers, supported schemaless payloads, and delivered real-time querying through OpenSearch integration.
Key Takeaways
- Schemaless JSON logs flow from apps through Kafka to OpenSearch and Hive
- Log-based metrics enable real-time alerting on production issues
- Developers query logs by app version, device model, and network type instantly
- Teams catch feature failures hours before daily metric aggregation runs
- Zero schema definition required: just log name and payload
Pinterest's self-service logging pipeline gives mobile developers production visibility and real-time alerting without touching backend infrastructure or defining schemas upfront.
Continue reading
What This Article Covers
In early 2020, a memory issue on Pinterest's iOS app exposed a real problem: there was no unified way to monitor how the app performed on users' devices. Developers had to pick between logging business events with context or saving miscellaneous files locally, but neither approach let them aggregate and analyze the data together.
Pinterest Engineering created a schemaless JSON logging pipeline that automatically adds metadata to every log entry, including app version, OS version, device model, and network type. The logs flow through Kafka into OpenSearch and Hive, where developers can query them with SQL without having to define schemas upfront.
Product teams can now spot feature failures as they happen through log-based metrics and alerts. They catch problems hours before the daily metric aggregation runs, which means faster incident response on both iOS and Android.
This page summarizes “Build an end to end JSON logging system for clients apps” by Pinterest, published Jan 10, 2023. All credit for the original work goes to the authors.
