Performance Bottlenecks in Go Apps
Article Summary
Grab's engineering team discovered their Go apps were mysteriously throttling at 1.94 CPU cores but flying at 2 cores. The culprit? A sneaky interaction between Kubernetes VPA and GOMAXPROCS.
Grab's real-time data platform team (Coban) runs stream processing pipelines on Kubernetes with vertical pod autoscaling. While debugging consumer lag issues on their SinktoS3 pipeline, they uncovered a critical performance trap affecting Go applications.
Key Takeaways
- GOMAXPROCS rounds down CPU allocation: 1.94 cores becomes just 1 core of actual performance
- VPA scaled down from 2.5 to 1.94 cores, causing 50% CPU drop and massive consumer lag
- Bumping minimum to 2 cores restored 95% CPU utilization and eliminated backlog immediately
- VPA v0.13 now supports integer-only CPU recommendations for Kubernetes 1.25+
- Autoscaling can't solve everything: manual intervention needed for optimal Go performance
Critical Insight
A 0.06 core difference (1.94 vs 2.0) caused catastrophic performance degradation because Go's GOMAXPROCS only uses integer CPU values.
