Optimizing AWS CloudTrail Costs

Tips to reduce your CloudTrail costs by eliminating duplicate data logging.
AWS CloudTrail pricing varies depending on the trails selected. A CloudTrail free tier where you can deliver one copy of your management events to Amazon S3.

— by Beto Juárez III, Solution Architect -

AWS CloudTrail gives AWS customers the ability to enable governance, compliance, operational, and risk auditing of their AWS account. CloudTrail records activity from users and API usage across a customer’s AWS services. Without any monitoring or optimization, CloudTrail can incur unnecessary costs. In this article, we will dive into the source of the additional costs, the AWS recommended fix, and how CloudFix can automate it for you.

Duplicate logging may be responsible for surprise CloudWatch costs

To better understand the source of costs, we need to look at the type of events AWS CloudTrail logs. AWS CloudTrail can log three different types of events, with each type charged separately:

  1. Data events
  2. Insight events
  3. Management events

For this article, we are only considering management events, as they are usually responsible for the bulk of CloudTrail costs. Management events provide visibility into management operations performed on resources, including most API calls to AWS resources.

It’s this kind of request – for business intelligence – that can cause surprises. AWS does not have a way to prevent you from duplicating efforts, but they do charge a hefty fee for duplicate events sent via CloudWatch.

AWS CloudTrail pricing

The source of the unnecessary cost stems from the $2 per 100,000 event price. Two dollars per 100,000 events may sound like a small amount, but that cost can easily balloon with scale and demands from the business. These additional charges can accumulate with surprising results. At CloudFix, these duplicate management events were responsible for over 90% of our CloudWatch costs.

Management events and CloudTrail pricing: A closer look

AWS delivers the first copy of management events in the CloudTrail free tier. Additional copies are charged at $2 per 100,000 management events delivered. At $2 per copy, it’s easy to see how CloudTrail pricing can quickly become a big headache, especially after enjoying free tier in production.

Because there is no clear value in configuring two CloudTrail trails to deliver the exact copy of management events to different S3 buckets, we want to disable duplicate CloudTrail trails that deliver the same copy of management events.

By removing duplicate CloudWatch trails, all CloudTrail Management events will be logged using the free tier and avoid the $2 per 100,000 cost.

Identify and remove duplicate CloudWatch trails

Identify duplicate trails

Now that we know about the two types of duplicate CloudWatch trails that can incur costs, we can use the following methods to identify our duplicate trails:

  1. AWS Billing and Cost Management Console ⇒ choose Bills.
    1. Go to the Bill details by service tab.
    2. In AWS Services Charges, expand CloudTrail.
    3. Expand the AWS Region to view the event cost record details. Then, review the PaidEventsRecorded metric to identify duplicate event records.
  2. AWS Athena query
    1. Create a table in an S3 bucket for logging CloudWatch events
    2. Run a SQL-style query to the Athena table using the PaidEventsRecorded method to search records e.g.,
				
					SELECT eventName,count(eventName) AS NumberOfChanges,eventSource
FROM your_athena_tablename
WHERE eventtime >= '2019-01-01T00:00:00Z'and eventtime < '2019-01-31T00:00:00Z'
GROUP BY eventName, eventSource
ORDER BY NumberOfChanges DESC
				
			

In order to compare if two trails are duplicates, the following dimensions must be completely identical:

  • IsOrganizationTrail
  • IsMultiRegionTrail
  • ReadWriteType
  • ExcludeManagementEventSource
  • DataResources
  • HasInsightSelectors
  • InsightSelectors
  • IncludeGlobalServiceEvents
  • HasCustomEventSelectors
  • AdvancedEventSelectors

Once you verify the trails are identical, then you have a target list of duplicate trails. However, there is one more scenario that you should check for to maximize optimization: trail supersets.

Identifying trail supersets

To maximize this optimization, we must also check for supersets. For this scenario, we must identify where a CloudWatch superset trail will be tracking all the events of a subset trail.

Let’s review the two scenarios below:

  • Scenario 1: A = B
  • Scenario 2: A ⊃ B
# SCENARIO
1 Trail A & Trail B are identical. They track exactly the same events.
A = B
2 Trail A is a superset of Trail B. Trail A tracks all the events that Trail B tracks, plus others.
A ⊃ B

As you can see in the second scenario, customers are again charged for duplicate copies of management events. This can happen frequently when an organization has multiple projects that create a report with similar management metrics.

We can identify supersets by comparing the configuration settings for two trails along the following dimensions and criteria:

DIMENSIONCONDITION
IsOrganizationTrailBoth trails have the same configuration, or the superset trail has this set to “True”, while the subset trail has this set to “False”.
IsMultiRegionTrailBoth trails have the same configuration, or the superset trail has this set to “True”, while the subset trail has this set to “False”.
ReadWriteTypeBoth trails have the same configuration, or the superset trail has an “All” configuration, while the subset trail has a “ReadOnly” or “WriteOnly” configuration.
ExcludeManagementEventSourceBoth trails have the same configuration, or the superset trail excludes fewer events than the subset.
DataResourcesBoth trails have the same configuration, or the superset trail tracks all the data sources of the subset trail.
HasInsightSelectorsBoth trails have the same configuration, or the superset trail has this set to “True”, while the subset trail has this set to “False”.
InsightSelectorsBoth trails have the same configuration, or the superset trail contains all the types included in the subset trail.
IncludeGlobalServiceEvents

Both trails have the same configuration, or the superset trail has this set to “True”, while the subset trail has this set to “False”.

HasCustomEventSelectorsBoth trails have the same configuration, or the superset trail has this set to “False”, while the subset trail has this set to “True”.
AdvancedEventSelectorsBoth trails have the same configuration, or the superset trail has an “empty” list, while the subset trail has a non “empty” list.
  • IsOrganizationTrail
    Both trails have the same configuration, or the superset trail has this set to “True”, while the subset trail has this set to “False”.
  • IsMultiRegionTrail
    Both trails have the same configuration, or the superset trail has this set to “True”, while the subset trail has this set to “False”.
  • ReadWriteType
    Both trails have the same configuration, or the superset trail has an “All” configuration, while the subset trail has a “ReadOnly” or “WriteOnly” configuration.
  • ExcludeManagementEventSource
    Both trails have the same configuration, or the superset trail excludes fewer events than the subset.
  • DataResources
    Both trails have the same configuration, or the superset trail tracks all the data sources of the subset trail.
  • HasInsightSelectors
    Both trails have the same configuration, or the superset trail has this set to “True”, while the subset trail has this set to “False”.
  • InsightSelectors
    Both trails have the same configuration, or the superset trail contains all the types included in the subset trail.
  • IncludeGlobalServiceEvents
    Both trails have the same configuration, or the superset trail has this set to “True”, while the subset trail has this set to “False”.
  • HasCustomEventSelectors
    Both trails have the same configuration, or the superset trail has this set to “False”, while the subset trail has this set to “True”.
  • AdvancedEventSelectors
    Both trails have the same configuration, or the superset trail has an “empty” list, while the subset trail has a non “empty” list.

Which trail should I remove?

Now that we have identified the optimization opportunities, we can safely remove the duplicate trails. The following table summarizes the actions that should be taken to remove duplicate trails:
# SCENARIO ACTION
1 Trail A & Trail B are identical. They track exactly the same events.
A = B
Disable the most recent trail
2 Trail A is a superset of Trail B. Trail A tracks all the events that Trail B tracks, plus others.
A ⊃ B
Disable Trail B

How to remove duplicate CloudWatch trails

To manually remove a trail, you’ll need to use the updateTrail method in the AWS Management Console or in the AWS CLI. Below is a quick summary of how to disable these trails:

  1. The AWS Management Console and open the CloudTrail console. From there, select Trails, then navigate to CloudWatch Logs and deselect “Enabled” from the appropriate trail.
  1. Use the StopLogging method AWS CLI for any trail.

Create a billing alarm

One last thing to cover is the ability to be proactive and create billing alarms. Your AWS environment is constantly changing, and without a doubt, duplicates will occur again resulting in surprising costs. Consider adding a CloudWatch billing alarm to alert your team before the surprise bills arrive so that your team can take immediate action.

CloudFix automates this and other AWS Cost Optimizations

Now that you understand how to identify and disable duplicate CloudWatch trails, you can help your team write a script that can run a job to check for duplicate trails in your AWS account.

But what about next week? And the week after? Or the next billing alarm? Or the next optimization opportunity? As you are aware, AWS releases 50 advisories a week through blog posts full of information and optimizations. CloudFix can help you stay on top of not only your AWS CloudTrail unnecessary spend but also all the other AWS advisories released.

Introducing CloudFix

AWS releases 50 Advisories a week, in the form of blog posts, to help their customers take advantage of the latest AWS recommendations including cost and performance improvements. However staying on top of every advisory as well as creating and managing jobs for every recommendation is time consuming for cloud teams. That’s why we built CloudFix.

CloudFix removes the burden from you and your team by monitoring all the AWS Advisories and regularly scanning your AWS accounts, identifying cost savings opportunities, and automatically implementing safe, non-disruptive AWS approved fixes.

Unlike other tools, CloudFix automatically applies the AWS recommended Advisories by integrating with AWS Change Manager. That means you are in control of how and when changes are made in your AWS accounts.

CloudFix Dashboard
The CloudFix Dashboard

After connecting your AWS account, CloudFix continuously:

  1. Monitors and reviews advisories with AWS
  2. Scans your AWS accounts using our full library of optimizations
  3. Fixes problems simply & safely

The CloudFix Fixer

Our “Disable Duplicate Trails” Fixer has zero risk since CloudTrail events are rarely attached to the company product or main line of business.

The following table describes the changes CloudFix will make to your CloudWatch trails, given the two duplication scenarios we mentioned previously:

# SCENARIO ACTION DIFFICULTY RISK
1 Trail A & Trail B are identical. They track exactly the same events.
A = B
Disable the most recent trail Very-Low No-Risk
2 Trail A is a superset of Trail B. Trail A tracks all the events that Trail B tracks, plus others.
A ⊃ B
Disable Trail B Very-Low No-Risk

Remember, these are no-risk optimizations that are safe and AWS approved. All disabled CloudTrail trails can be easily re-enabled later if needed.

More CloudFix Fixers

Check out our other blog posts on cost savings or see how CloudFix integrates with AWS Systems Manager Change Manager to execute its Fixer optimizations.

More Updates

Reduce Costs

CloudFix now supports Financial Savings

Since the launch of CloudFix cost optimization platform from DevGraph we have seen huge interest and adoption. Today, we are excited to announce that you can save even more with our new financial savings capabilities in addition to resource related

Read More
Skip to content