In episode six of the AWS Insiders podcast, AWS superfan and CTO of ESW Capital, Rahul Subramaniam, spoke with EVP of Cost Optimization at Trilogy, David Hessing, about AWS cost optimization. Listen to the full podcast episode here, or read on to see how Trilogy uses CloudFix and other best practices to reduce AWS costs, enhance performance, and improve their overall cloud hygiene.
The tiers of AWS cost optimization
For Trilogy, their pursuit of AWS cost savings begins with CloudFix. This is a product they sell but also use internally to realize 5-10% in savings by finding and fixing the safe, secure, zero downtime recommendations that come straight from AWS. Those easy fixes improve their cloud hygiene, and give them instant cost saving wins.
The second tier is what David called the resource-level optimization specs. This is where their team will write a specification and look at a set of AWS accounts, typically with up to $10 million in spend, and identify existing savings opportunities. These specifications involve steps like deleting or fine-tuning resources, and though they are a heavier lift than CloudFix they help Trilogy realize an additional 40-50% in AWS cost reductions.
The final tier is code-level cost reduction specs, where the Trilogy team will deep dive into a given product and find additional savings within the code itself. This is the hardest type of saving to achieve, as it requires specific coding and changes to the product, but allows them to reach up to 80% or 90% total AWS cost savings off the original cost of running the product.
Identifying patterns for AWS cost savings
When David first began the AWS cost optimization process, he was presented with the challenge of having tens of thousands of accounts with tens of millions of dollars in AWS spend, a lot of which was wasted. After initially struggling to find savings opportunities, his team began to see patterns. One of these was incorrect purchasing in AWS. Because of the extensive range of services that AWS offers, there are a lot of excellent options available for whatever goals you have. If you need storage, you pay for S3 or EBS. If you need compute, you pay for instances in EC2. What Trilogy discovered is that because some of their older software was running on certain services, they were paying for computing where what they really needed was storage. Adjusting to the correct type of resource was a huge savings opportunity.
Another pattern that emerged was database consolidation. When Trilogy buys software products, they each have their own database – one product acquisition actually had over 2000 databases. With so many separate databases running with their own dedicated compute and storage, you need to allow in the range of 70% headroom to account for periodic spikes on each individual database. But when you consolidate those databases onto a single, larger server, you only need to allow for about 20% headroom due to the increased size of the server coupled with the lower frequency of spikes. Because those spikes don’t happen at the same time, you need less space and can run all of those databases for a much lower cost – Trilogy has seend savings of 65-70% via this type of database consolidation.
Maintaining control and security with AWS Change Manager
Change Manager is a service embedded within the AWS Systems Manager, which provides a safe way for you to make control changes to your AWS environment by setting parameters. By setting up and installing those templates into your database account, you can create change requests against that template. This then allows you to see every single change, and make sure that it is carried out in a controlled, recorded, and secure manner.
Rahul pointed out that customers are afraid of changes being made to their AWS accounts, for two reasons:
- Customers often don’t know what is actually specified in the resources that they have launched. When someone then comes and talks about cleaning up an account of deployed resources, the idea of shutting off instances, resizing them, or even rebooting them at times can cause a lot of anxiety.
- Typically, to make any change in AWS you need to grant a third-party tool or third party people incredible amounts of access to make those changes. When you lack trust in either the tool or people, you are reluctant to hand over the keys to your accounts.
AWS Change Manager has dramatically altered how changes can be implemented within your account. The fact that you have a fully auditable, traceable mechanism of changes being made in your system is remarkable. CloudFix uses AWS Change Manager to make the savings-driven changes it finds, delivering simple, zero-risk, AWS-recommended cost savings without impacting anything in a customer’s AWS environment.
The shifting dimensions of AWS cost
Managing costs in AWS is an evolving, never ending process – there will always be scope for optimizing costs. From his 15-year experience working with AWS, Rahul has identified the following three AWS cost dimensions:
- AWS is constantly creating new services – with each service, there is a life cycle where you start using it, see its benefits, and it initially seems inexpensive. But as you continue to use it your costs begin to spike. You must then look at the causes of that spike, and identify the best practices and optimization strategies to then reduce your costs over time. This is the natural life cycle of every new AWS service.
- Development and staging environments – your developers are constantly launching new services and adding new resources, none of which are static. This requires ongoing monitoring and the introduction of policies to ensure that you control costs whenever your team launches these new services and resources.
- Keeping up with AWS – AWS’s intense pace of innovation means that you often feel like you’re at least six months to a year behind everything new that AWS is creating and releasing. You are always playing catch up, and part of that process includes the need to develop new runbooks so that your engineering team can understand, adopt, manage, and safely execute those new changes.
With all of these shifting dimensions and never ending changes, there will always be room for cost optimization – if you are spending any money on AWS, you will need a mechanism to to save money and reduce or eliminate wasted spend.
Additional insights from this AWS cost optimization discussion between between Rahul Subramaniam and David Hessing include:
- AWS cost optimization experiments
- AWS cost hygiene best practices
- How to build an AWS cost optimization team
- AWS cost optimization success metrics