Structured Logging: The Best Friend You'll Want When Things Go Wrong
Article Summary
Grab's logging bills were spiraling out of control. Their solution? A complete rethink of how they capture what happens in production.
Grab's engineering team shares how they built a structured logging framework from scratch to handle billions of rides across Southeast Asia. The article walks through their journey from expensive, unstructured logs to a JSON-based system that integrates seamlessly with their Elastic stack.
Key Takeaways
- Dynamic log levels let teams switch from WARN to DEBUG at runtime without redeploying
- Common schema ensures consistent JSON structure across all services and teams
- Nanosecond timestamps enable causal ordering to see exact event sequences
- Dockerized Kibana gives developers production-like logging in local environments
- Automatic trace ID correlation links all logs for a single request
Structured logging cut costs, improved debugging speed, and gave Grab's 100+ services a standardized way to understand what's happening in production.
About This Article
Grab's logging infrastructure relied on syslog-style key-value formats across multiple Golang services. This made logs hard to query and drove vendor costs up exponentially as the company scaled to 3 billion transport rides.
Aditya Praharaj's team built a custom Golang logging library that outputs JSON. They created the Common Grab Log Schema (CGLS) to ensure consistent structure and integrated it with Grab-Kit. This standardized logging across 100+ backend services without requiring any service rewrites.
The structured logging framework enabled nanosecond-precision causal ordering and reduced Mean Time To Resolve for outages. Developers can now toggle log levels at runtime through configuration management, which means they no longer need to redeploy services during debugging.