Pushing ECS to Capacity

Migrating from ECS was never our plan A, even we always push it back in the list when we are having planning session of resources that need to be optimized. We calculated the ROI, and it looked like a bad trade deal: High Effort even though it has High Savings.

For multiple years, ECS was our reliable platform for orchestrating our microservices. We weren’t presented by plethora of configuration options and control plane management. It just works, covering all of our use cases.

As our infrastructure matured, changes of season and so did the challenges. We didn’t jump into another platform because it was cool, we jumped because we have less room to optimize.

The vision for optimization is not only about cost optimization, it’s about preparing a building blocks that could benefit us in the future. We knew even if eventually move to Kubernetes, all of our previous optimization still benefit us. So, we started with low hanging fruit.

To find out what it means to be Low Hanging Fruit we classify it as a task that has Low Effort — Low-to-medium Savings. Below classification helps us greatly for focusing on the least effort as it will give a faster impact and helping us reducing the cost bleeding faster.

Figure: Cloud cost optimization principles

Google Cloud cost optimization principles

We start by throwing all of the ideas in our head and after that we do brief analysis how much effort that we need to do it and potential cost savings as impact.

Figure: Optimization initiative matrix

Yay! we can pick RightSizing as our initiative now, looks like we are ready to make our hands dirty 👨‍💻, but wait before we do implementation we need to more in-depth measurement again as I remember some quotes “Cut Once, Measure twice” as our time is costly and making sure our effort being put on the right place.

Measure and Prioritize. We started with Measure each of our service on ECS Cluster, by utilizing Steampipe we calculate each service task count and task allocation after that we got total capacity reservation for each service.

Prioritize based on impact, where we do select service that will be optimized. By doing this we are making sure all of our effort has a significant impact.

Figure: Prioritizing a service

• The Action: We calculated total resource allocation per service to prioritize top service that we will do RightSizing. (Only select service that consume ≥ 5% Cluster CPU Capacity)
• The Result: In just one week, we shaved off 10% of our spend by doing RightSizing. It felt like an easy win.

Another opportunity we found during right-sizing was modifying autoscaling. Some services frequently hit their scaling limits; others had quirks such as memory leaks that kept memory-based scaling active, or incorrect scaling metrics. We moved this initiative up the backlog because a few days of configuration and targeted fixes could help. These changes reduced ECS spend by about 3%.

Conclusion

By the end of the first phase, we had implemented more aggressive autoscaling and reduced our task footprint. We moved to another initiative outside ECS, believing the cluster was already optimized, with 85–90% capacity reservation and recent cost savings.

As we work through the optimization backlog, easy wins become less common, leaving medium- and high-effort initiatives. To meet our target, we need to take on the harder work.

We then moved on to our next ECS initiative.

---

Originally published on Medium.