The Price Is Wrong: How to Save on AWS

Transcript

Hilary Doyle: Rahul, I admit, I’m a little concerned about this episode. Your description was troubling.

Rahul Subramaniam: Is that because I told you that the episode would smell? I didn’t say it was going to stink.

Hilary Doyle: Okay. Well, I have an extremely sensitive nose.

Rahul Subramaniam: Well, that’s good because it can save you a ton of money and that’s what we’re going to talk about today.

Hilary Doyle: Sorry?

Rahul Subramaniam: Okay. So when we analyze AWS usage and can tell that something’s not quite right, we basically call it a FinOps smell.

Hilary Doyle: Right, okay. So when developers see patterns of problems and source code, they’ll sometimes call it code smell. This is, it’s the same thing?

Rahul Subramaniam: That’s absolutely right. And we’ve identified these smells from optimizing tens of thousands of AWS accounts. In this episode, we want to talk about some of these anti patterns and how to fix them.

Hilary Doyle: Okay, that sounds terrific. I admit I’m a little disappointed. I thought this episode might include more of your grandmother’s baking, but this actually, it’s a close second. So all right, the smells of AWS, I’ve got some ideas. Here we go.

This is AWS Insiders, an original podcast from Cloudfix, bringing you what you need to know about AWS through the people and the companies that know it best. Cloudfix is the nonstop automated way to find and fix AWS recommended savings opportunities, it never stops. I’m Hilary Doyle, I’m the co-founder of Wealthie Works Daily.

Rahul Subramaniam: And I’m Rahul Subramaniam, I’m the founder and CEO at Cloudfix. So Hilary, today we are really excited to share insights we’ve had at Cloudfix about optimizing AWS. 

Over the last 15 years, we’ve had a fascinating journey deciphering hundreds of thousands of AWS bills, and we’ve of course developed a few heuristics that suggest certain types of problems, stuff that we can actually sniff out in these AWS bills, and I want to share some of those with our listeners today.

Hilary Doyle: We will make sure that this episode smells like fresh cut grass and roses by the time you leave here today. But first, your AWS headlines. 

AWS has just announced expanded support of their fault injection simulator for Amazon Elastic Kubernetes service and Amazon Elastic Container Service, which adds additional fault injection simulator actions for EKS and ECS. Rahul, why on Earth would you want to purposefully inject fault into a running system?

Rahul Subramaniam: A fair question to ask. The fact that most folks don’t understand is that in the cloud, you have to design your systems in a manner that assumes that everything can fail, and you have to build resilience into your architecture. It’s the foundation of chaos engineering, and the fault injection system just helps you avoid any unexpected surprises. The story of how Netflix pioneered chaos engineering is a must read if you’re building apps in the cloud.

Hilary Doyle: Absolutely, and the weather has been great lately, but winter is coming because AWS just announced the general availability of a new AWS Snowball edge storage optimized device with higher storage capacity. It is a jump from 80 terabytes to 210 with high performance NVMe storage. It’ll enable customers to simplify multi petabyte data migrations from on-prem locations to AWS. Rahul, will this significantly increase or improve cloud migration in your estimation?

Rahul Subramaniam: It absolutely will. The world is moving to the cloud. In the early days, we used to ship hard drives when we had to decommission a data center because internet transfers with their limited bandwidth would just have taken us years. 

Here’s an interesting incident. Back in 2015, and I think that was around the time I was waiting for a meeting with one of the AWS product teams in the AWS office in Seattle, and we ran into Bill Vass who asked us to join him and wait in his office. We started a conversation and I complained about how many months it was taking to move one of our data centers into AWS and migrated into AWS.

He suddenly got excited and said, “Hey, you know what? We have this new device that we call Snowball and it can copy over all of the data in one shot and have everything migrated over a matter of days.” And he asked, “Can I just ship one to you?” 

We were lucky to get one of the earliest Snowball devices and while it took two to three attempts to get it right, we got the migration done in record time and it was amazing. Today’s Snowball has grown in capacity and efficiency by orders of magnitude.

Hilary Doyle: Great story. Bill Vass, VP of engineering at AWS plus a hallway run-in, it’s a lost art in the era of Zoom. That’s it for your AWS news headlines.

So we are going to dig into ways of optimizing people’s AWS usage and since we brought up code smell and Rahul, you and I grew up in the 1980s and love a good 1980s game show, we have come up with a high stakes game of smells with one special guest, Rahul’s moonlighting co-host Stephen Barr. The name of this game?

Roger: The Price is Wrong.

Hilary Doyle: First things first, let’s meet our contestants. Joining us from Reno, Nevada. He’s a collector of vintage potato peelers and expensive cheese. He recently appeared on The Masked Singer and season 75 of Survivor. Please welcome Stephen “Mad Dog” Barr.

Stephen Barr: None of those facts are true.

Hilary Doyle: Don’t get held up by the truth.

Stephen Barr: Well, I’m actually principal architect and technical evangelist here at Cloudfix.

Hilary Doyle: Can I get a witness? And joining you today – Mad Dog – from Chennai, India is the famed Lego architect who dreamt of becoming a nuclear physicist as a kid, Rahul “Super Maniac” Subramaniam.

Rahul Subramaniam: Well, mostly true in this case.

Hilary Doyle: So Stephen, let’s get started. The Price is Wrong is not for the faint of heart. We’re going to head over to the wheel of big spending. Walk with me here. Okay, Stephen, the wheel is covered in clues about what may be wrong with somebody’s AWS bill. So get a good grip on that wheel and give it a spin.

Stephen Barr: All right, here it goes.

Hilary Doyle: Oh, that was some heavy wheel spinning exertion. OMG, an audience favorite. You have landed on James and the Giant Account – smells like peaches. And based on that clue, Stephen, tell us what would be wrong on someone’s AWS bill if they landed on James and the Giant Account? Let’s go.

Stephen Barr: James and the Giant Account. James and the Giant Account. Oh, okay, I get it, yes. This is when the account is at the fundamental unit of division, and as organizations get bigger and bigger, you want to have different accounts like one for dev, one for prod, one for staging, and that way you don’t have resources that shouldn’t be associated with each other. 

You don’t want your production deployment pipeline to get to your development servers and vice versa. So if you have one giant account, then everything’s mixed together, it’s hard to figure out what’s driving cost. If you have one giant account, you probably have some really complicated tagging structure to account for all the resources you’re using. 

I think basically you end up with, well, I guess the title says, one giant account where you really can’t tell who’s doing what because it’s just everything is pooled together, right? It’s like if you go out to dinner with 40 people and you get one giant bill. I guess that’s what I could imagine that would look like.

Hilary Doyle: Been there and done that, it sounds like a nightmare, but delicious along the way. Over to you, Rahul. How did Stephen do with the James and the Giant Account bill?

Rahul Subramaniam: Okay, I think he was very close. So yes, this is all about customers having one single account where they just put everything in there and the entire team shares all the resources in there. Now, there are multiple problems with that beyond a certain scale. Sure, if you’re a two or three member dev team, no problem. But as your organization grows, having one account for everything is just a big no-no. 

And the reasons are the following: number one, your security footprint is just so large that you just cannot protect that account if something were to go wrong. And things do go wrong all the time. So by giving everyone access to one account, you’re just putting everything at risk all at the same time. 

The second problem is that AWS sets limits to all the different services that you use, okay? Which basically means that when you have one joint account, you are letting everybody use the maximum of all the limits across all the services and therefore the blast radius of something going wrong is literally everything in the account to the maximum extent possible. So you never want that scenario.

Hilary Doyle: I do not like us talking about blast radiuses on a show when we have code smells. All right, Stephen, let’s head back to the big wheel. Give her a spin.

Stephen Barr: Here goes. All right.

Hilary Doyle: Oh my goodness, no way. He has landed on a real trickster: Kubernetes the World. Not a world I want to live in, but Stephen, tell us about that AWS bill.

Stephen Barr: So I’m picturing a pie chart of all the services that you have in your bill, and typically there should be a couple of different big slices, a couple of small slices. But in this pie chart, it’s almost one giant slice and that’s going to be EKS. So if you’ve put everything into Kubernetes – what I see wrong with that is that you’re not using any of the higher order services. You wouldn’t be using ElastiCache, you wouldn’t be using Lambda. 

If everything is in EKS, then you’re probably not leveraging the cloud to the extent that all of the higher level services that AWS is providing for you. You are using Kubernetes for everything and then you have to rebuild and redeploy these higher level services on top of that and it seems really cumbersome.

Hilary Doyle: Rahul, how did he do?

Rahul Subramaniam: I think that was very good. I’ll add two things to this. The first is that I think there’s a higher problem that surfaces when you see a bill like that. I think you’ve got a management over here, or your technical team – I don’t know which one – which is very uncertain about their move to the cloud or move to a particular cloud that they’re trying out. 

It just tells me that they are trying to hedge their bets, and the safest bet over here is to do a Kubernetes cluster. And once they put everything in Kubernetes cluster, you feel like you’ve addressed that concern. Hey, what if I want to move to some other cloud? Or what if I just want to go back on premise? So the basis for that decision is the first red flag.

The second, as Stephen pointed out, is the fact that when you’re all in on Kubernetes, it means that you’re not leveraging all those amazing services that truly leverage cloud scale and the capabilities of the cloud at a fraction of the cost of running it inside a Kubernetes cluster. You’re not leveraging any of those amazing things and it’s likely going to be a massively unoptimized setup for you.

Hilary Doyle: Rahul, I cannot get this guy away from the big wheel. He is just ready to spin, check out the biceps on this man.

Stephen Barr: Come on. Big spending.

Hilary Doyle: Ooh, my absolute favorite. Stephen has landed on The Buffet. Do not touch the coleslaw, it’s been there for days.

Stephen Barr: I am hungry.

Hilary Doyle: You won’t be after you see this buffet. Stephen, give it to us straight. How’s the AWS Buffet bill looking?

Stephen Barr: So this is a really funny one and this is driven from a good place. These are enthusiasts and they want to try everything. You’ve got a bit of Aurora, you’ve got a bit of Redshift, you’ve got a bit of RDS, every storage solution they’ve got, you’ve got a bit of that on the plate. And people who are doing this, I think it can reflect a bit of that James and the Giant Account that we talked about earlier where there’s so many people in the account and the organization’s so big that overall they’re using every single service. But it could just reflect enthusiasts who are just trying every single one that comes out, and that’s not what you want to do. You want to prune the choices and pick a strategy that’s not using every single service.

Hilary Doyle: Something I’ve known for decades, enthusiasm is expensive. Rahul, how did Stephen do?

Rahul Subramaniam: You were spot on with this one, Stephen. I’m just going to add a couple of other bits to this. When we talked about James and the Giant Account, we were talking about one organizational unit which had everything in it. In this particular case, you have a dev team that’s either really enthusiastic or just has a terrible application design. 

With this one, you can’t even begin to talk about cost savings because there’s something more fundamentally wrong with how you are using the services. A very large footprint of different AWS services all running in one account is not only going to have the same side effects that we spoke of earlier, but in this case, you are probably in need of some major open heart surgery on your application. 

I really hate to say this, but you’ll probably need to have your team educated in what good cloud architecture looks like or even find a different team. When you have good architecture, you need just a handful of the core services in each account.

Hilary Doyle: Stephen is windmilling his arms, getting them ready, warmed up. Head on over to the big wheel and let’s give it a spin. I mean, this man’s strength knows no bounds.

Stephen Barr: No whammies.

Hilary Doyle: Just is the wheel ever going to stop spinning?

Rahul Subramaniam: I can only see a white board. All the colors are blending into white right now. 

Hilary Doyle: Okay, whoa. The wheel has come to a resting place, step away from the wheel because you don’t want to get a staph infection. This next clue, Swimming in a Pool of EC2. Sounds dangerous.

Stephen Barr: If you look at the bill and EC2 costs exactly the same. That’s what I’m guessing, because if a pool is at one level, maybe your EC2 is at the same level, and if your EC2 is the same week after week and month after month, then it means that you’re basically running a fixed amount of EC2. And at that point, it’s hard to make the argument, you’re using the cloud in the most expensive way possible because typically your loads are cyclical and your infrastructure should adapt. 

So if you’re in this pool of EC2 then, lots of imagery in my head…

Hilary Doyle: I know. I’m so sorry.

Stephen Barr: If you’re in this pool of EC2, you’re stuck with what you have and you’re treating it as if it was a data center. And so this is representing a mental shift that they haven’t done yet where they can adjust the amount of infrastructure they have very fluidly. They can have more, they can have less, and just like their loads are cyclical, their infrastructure should adapt to that pattern.

Hilary Doyle: Yeah, sounds like a lot of Band-Aids floating around in that pool. Rahul, how did the Mad Dog do?

Rahul Subramaniam: I’m still stuck on that visualization of someone paddling through in the middle of EC2 machines inside an actual physical pool. Anyway, Stephen did a pretty neat job where he talked about the on-premise mindset moving over to the cloud, and that’s absolutely right. I think there are still a lot of organizations that treat their AWS accounts like they are on-premise data centers, and I think they’ve got it all wrong when they do that.

The cloud is really good only when you use it for its elasticity and scale. What that means is that you really have to switch your mindset from, “Hey, I run X number of servers to a paper used by the second mindset.” If you treat a cloud like a data center, then I’m guessing nine out of 10 times you will end up spending more than you did in your data centers. 

I also want to call out one other symptom, not using any spot instances. It shows that your applications aren’t designed to be stateless, and you could be paying up to 10X more just due to that. The bottom line is that it makes no sense to be in the cloud if you haven’t understood and adopted the cloud mindset.

Hilary Doyle: Okay, I am going to just pause this game here for a second because if you are enjoying this Stephen and Rahul AWS tech love in, then let’s face it, you’re going to want to marry their other project. Stephen, tell us about your livestream.

Stephen Barr: So we have another live stream called AWS Made Easy, and then that one we review the latest AWS announcements. And then Rahul and I, we’ve developed this great system of rating each announcement or blog post on a number of clouds. So like one to five clouds, but then also does it simplify or add complexity to your life as a user of this particular service? So it’s been really, really fun to be able to have this live stream. We’re every Tuesday at 9:00 AM Pacific time.

Rahul Subramaniam: We try and answer the questions that come up from the audience and there are lots of AWS folks that are in the audience who chime in with their answers. And that’s the amazing part of the livestream for me.

Hilary Doyle: It’s called AWS Made Easy. You can find out more about it at cloudfix.com/livestream. Unsurprisingly, they stream live everywhere you can stream live.

Okay, Roger, let’s get back to. 

Roger: The Price is Wrong.

Hilary Doyle: Okay, Stephen. I hope you’ve kept your arms warm. So step back up to the wheel and give it a spin.

Stephen Barr: Let’s do it. Here goes.

Hilary Doyle: Ah, okay, one we all know well, Highway Robbery. Break it down for us. Stephen, what is the Highway Robbery AWS bill looking like?

Stephen Barr: This is too much EC2 relative to everything else. This is where I think about that pie chart where you should have different slices, you should have some compute, some storage, and there should be some distribution that looks normal. An EC2 is a huge portion of your spend. Then it suggests a low degree of cloud maturity and it’s the most expensive way to operate the cloud. 

So again, similar to the ‘swimming in a pool of EC2,’ you’re treating it like you just bought a bunch of servers, and you then have to reinvent the wheel on these servers and you’re not taking advantage of the high level services. You should see some ElastiCache pieces. You should see Lambda, you should see usage of spot. You should see a lot of things.

Hilary Doyle: Well, no reinventing the wheel on The Price is Wrong. Let’s just start there.

Stephen Barr: You’ve already got a great wheel.

Hilary Doyle: Thank you very much. I mean, you know it well. Listen, I mean as you point out, immaturity can be even more expensive than enthusiasm. I like to call this clue the “your lift and shift does not smell like roses.” Rahul, how did Stephen do?

Rahul Subramaniam: I think that is very close. Just a couple of things to add. One scenario that I see very often is customers buying bare metal instances. And that to me is a big red flag, it says to me that you’re looking to manage everything on those instances all by yourself. Why would you ever want to do that? I mean, you might as well buy these machines and put them in your data center. It’s probably going to be way cheaper anyway. 

There are two ways to get cost efficiencies. Okay, there’s financial engineering and there’s technical engineering. When I talk about financial engineering, I’m talking about instruments like savings plans, CRIs, and reserved instances. If you don’t have those in place, then you’re literally just bleeding money. 

And then the second scenario is technical engineering. If you have not optimized these EC2 instances usage with things like spot instances, auto-scaling groups, and spot fleets or S3 for your object store or SQS and EventBridge for your pub subsystem, those fundamental AWS services, then you are not leveraging the cloud for what it is really good at.

Hilary Doyle: Bleeding money sounds painful. So Stephen, step back up to the wheel. Two clues left. The wheel is looking skeletal and optimized.

Stephen Barr: All right, I’m going to do a very strategic spin here. There we go.

Hilary Doyle: All right. So we have landed on, it’s a difficult clue to parse Stephen. It just says what every mattress manufacturer warns you about. I’m assuming this has something to do with tags.

Stephen Barr: Tags. They always say don’t cut off this tag. And there’s this big warning and I have no idea why it’s so important that you don’t cut this particular tag. So if there’s no tags on your mattress, I guess that’s horribly catastrophically bad. But it’s probably even worse if there’s no tags on your resources. 

So tags are what allow you to label things and group things. It’s the giant label maker for all the things in your account and you want to keep organized and you want to be able to know what your resources are doing. And maybe you have certain tags that are saying, “This is all associated with this application,” or this part of the application or this service or who launched it, or is it a testing one if it’s going to be shut down at a certain date. You want to be able to label your resources and tags are the mechanism of doing that. And so if you don’t see any tags, it’s just complete disorganization.

Hilary Doyle: Seven years of bad sleeps without tags. Rahul, how has Stephen done with the mattress AWS bill?

Rahul Subramaniam: It was 100% on the money with that one.

Hilary Doyle: Yay.

Rahul Subramaniam: I’ll add one more analogy. Hilary, I’m going to ask you this question. Since your entire startup in entrepreneurship is around money, what’s your impression of evaluating or looking at your credit card bills at the end of every month?

Hilary Doyle: That’s a painful question to ask any entrepreneur because I think that most entrepreneurs, even when they happen to be starting a startup that is centered around financial literacy, they are hesitant to look at their credit card bills at the end of any month.

Rahul Subramaniam: But you have to, right?

Hilary Doyle: Yeah.

Rahul Subramaniam: But you have to if you want to understand where you spent your money.

Hilary Doyle: Yes.

Rahul Subramaniam: Now imagine you got your credit card bill with absolutely no line items that said.

Hilary Doyle: Right.

Rahul Subramaniam: Where you spent the money. It just gave you one number and in fact you just had the line numbers with amounts. How would you know where you spent your money?

Hilary Doyle: Yeah, you raise a very fine point.

Rahul Subramaniam: I can’t remember what I spent last week on my bill. So tags are literally that important. In fact, I would say even more important. It might be okay for you to not look at your credit card bills, but not looking at your AWS accounts and doing so without tags is literally shooting yourself in the foot. So I would recommend that the first thing you do when you have an AWS account is make sure that you have good tagging policy, and make sure that every resource that is launched has appropriate tags in place.

Hilary Doyle: All right, Stephen, I have some sad news, which is there is only one clue left on this wheel, so let’s hope you can spin and reach it.

Rahul Subramaniam: It’s literally the one little slice that’s holding on for dear life on that wheel right now.

Stephen Barr: All right, you guys.

Hilary Doyle: We have landed on Money Doesn’t Grow on Decision Trees. Stephen, tell us about that AWS bill.

Stephen Barr: So I’ll just start with the story. I had a friend message me and he’s a data scientist and he said, “I’m in trouble. I left a bunch of P3 instances on for a week.”

Hilary Doyle: Oh, God.

Stephen Barr: That I forgot about. And P3 instances, Rahul can tell you, they’re expensive. They are $12 to almost $25 an hour in some cases depending on how you resource them. And if you leave a bunch of them on over the weekend, that adds up. So data scientists tend to do this sometimes, that’s where data decision trees is data science one of the predictive model. And so data scientists tend to be a bit loose with how they can leave on their resources and the stuff they’re doing is very computationally intensive.

And so they’re not going to start up a T2 micro, they’re going to start up a P3.8xlarge or P3.16xlarge to get those big GPU instances which are really expensive. So we want to make sure, especially for the data science practice, that you want to limit that blast radius and figure out how to reduce that spend. I guess it’s all about the SageMaker cost and also the data science related instance costs that are the warning in this Money Doesn’t Grow on Decision Trees clue.

Hilary Doyle: Just a quick question in the meantime, this seems like such an honest mistake and everyone is going to make it at some point. Does AWS ever just acknowledge the mistake and say, “Listen, we don’t need to charge you for that time?”

Stephen Barr: Yeah, there is this one time I accidentally checked in a private key on a GitHub repository – and this is before they had automation and immediately my account turned into a Bitcoin mining account. And luckily they gave me a break on that one, but it was a one time thing, it will not do that again. Yeah, it’s frightening when that happens.

Hilary Doyle: Rahul, how Stephen done on this final clue?

Rahul Subramaniam: Absolutely spot on.

Hilary Doyle: Ending on a high, yes.

Rahul Subramaniam: I’m just going to add one tip that literally comes from the mouth of the GM of AWS AI. About six to eight months ago. He and I were having a conversation about what is the one mistake that folks make when it comes to services like SageMaker on AWS? And he was like, “Rahul, if only they would shut off their instances, they could cut their bills by up to 75%.” The number of instances that are just literally left on costing hundreds of thousands of dollars is non-trivial, so if only they could shut it off. 

And unfortunately, AWS has no way of knowing whether you deliberately want to leave the machine on because they can’t look inside the machine. It’s not part of the security protocol so they have no way of knowing. So unless you explicitly shut it off, it’s not going to shut off. So you’re going to be billed for every second that those machines are going to be running.

Hilary Doyle: Between the two of you, what do you think is the most efficient or effective way of ensuring your development teams are turning off all of their instances? Prizes, money, vacation time?

Rahul Subramaniam: Automation, as Stephen just said.

Hilary Doyle: Okay.

Rahul Subramaniam: Unless you have automation.

Hilary Doyle: Automation.

Rahul Subramaniam: There is absolutely no way to ensure behavioral change as well as organizational change. It just doesn’t work. Automation is the only way to ensure that this happens.

Hilary Doyle: Wow, that’s such a valuable tip, but we’ve got a show to complete here. Stephen, it’s time to move to our final game. The showcase showdown. There’s no wheel here as you can see, but there are three curtains and behind one curtain is your final clue behind the other two curtains, Stephen, not a dinette set, but an Oracle T-shirt. So choose wisely. Stephen, would you like door number one, door number two, or door number three?

Stephen Barr: I’m going to go with door number three.

Hilary Doyle: Door number three, let’s pull the curtain back. When you know your final clue and the clue set, it’s not far off from a dinette set. Table stakes.

Stephen Barr: Table stake. Okay. When you look at your DynamoDB tables, you might see evidence that maybe, depending on the name of the tables, that each application is using a lot of tables. And this suggests that they’re not using DynamoDB the right way. DynamoDB, and there’s been a lot of great re:Invent talks, and you’ve had Alex DeBrie on this show talking about single table design or few table design. And you can structure your DynamoDB tables in a way that you can query all the customers and all the customers who have made a certain order and the cost of those orders and the inventory of the items in the order. You can structure Dynamo to do that with one table, but if you see a lot of DynamoDB tables all for one application, well they have to be joining that data somewhere and that means it’s going to be joined in the application layer. And that’s another sign of they’re not using the tool the right way.

Hilary Doyle: Give this guy some applause, Rahul, how did Stephen do?

Rahul Subramaniam: I think he nailed it when he said this was all about DynamoDB tables. The nuance here is the problem when you look at DynamoDB tables and the bills associated with them, when you see lots of tables, invariably it’s a sign that someone is bringing either a relational database mindset to DynamoDB, which is actually a no SQL database or a document store. And that can be phenomenally inefficient for something like DynamoDB. And you could increase your costs dramatically because when you’re doing a relational data store structure in DynamoDB, you could have scans that require you to go through all the data, all the time. And that could literally cost you a bomb very, very quickly before you even realize it.

And one of the symptoms – that is very easy to recognize that mindset problem – is when you look at your table names and they are literally the objects that you would have in a relational database. The other scenario might be that you have not thought through the data access use cases really well. And for every user scenario, you are creating a separate table where data is stored independently and that has its other overhead. 

So for anyone that sees these two symptoms, you should seriously take a look at single table design, really understand how that works. It’s a different paradigm, it’s a different concept, but if you understand it and you start using it, then it is a complete game changer in terms of a transactional data store.

Hilary Doyle: Game changer. Well, that was our game ender. Stephen, thank you so much for joining us on The Price is Wrong. Today’s winner is our listener, but don’t worry, Stephen, you are not leaving empty-handed. Roger, what is Stephen taking home with him today?

Roger: An all expense paid trip to Seattle, Washington.

Stephen Barr: But I live there.

Hilary Doyle: If you smell anything fishy and want to tell us about it, we’re talking about your AWS bill people. You can reach us at podcast@cloudfixx.com

Rahul Subramaniam: And please leave us a review and don’t forget to follow the show to get the new episodes as soon as they’re released.

Hilary Doyle: AWS Insiders is brought to you by Cloudfix, an AWS cost optimization tool. You can learn more about them and you should at cloudfix.com.

Rahul Subramaniam: Thanks for listening. Bye-bye.

Hilary Doyle: Give the wheel one last spin, Stephen, before you leave today.

Stephen Barr: King of the wheel.

Hilary Doyle: Oh my God. It’s like watching a time jump.