Amazon, Developer, Bill Traction discussed on Arrested DevOps
Is that something that I've noticed for a while? Is that what customers and what customers actual pain points are historically aligned super. Well what happens? Is You wind up with someone in finance getting Amazon Bill and they see that looks like phone number. That isn't there so they start to worry and they wonder how many books engineering is buying you. Then have that moved through about five different levels of corporate telephone to the person who spun up a cluster but it's magically never allowed to see the bill and it turns out the pain is not that it's too expensive but rather that it's far too difficult to figure out what the cost drivers are and allocate that so it comes down to understanding optimizing and predicting it now that we're suddenly seeing a recession style. Pandemic event customers are a lot more accurate when they say so. What are you here to do? We're here to save money now. Suddenly that's true rather than just understanding it. So it's it's simplifying aspects of the consulting but it also means that they're that now there's a little bit less nuanced have those conversations and used as we used to say it. Looks like a dumb question but why do you care about the WS bell? Well now it actually is a dumb question because most of our customers are inbound folks. We're talking to. There's a very clear reason why suddenly all their projections magically shift. You mentioned that. The engineers turning on the instances do not get the feedback of how much thoughts costing like. We did devops in order to give people the feedback loops of the consequences of their decisions and yet with cloud. We just took that away to some extent. Yeah so you've always had this problem with traditional data centers to the just was buried in year. Multi your cycles. So that's kind of the beauty of cloud for me. Anyway is that there is any hope of ever measuring how much it actually costs as opposed to the data center. People burying it in the cost of new projects because they're only allowed to get budget for new projects so they roll all that maintenance into it and you wonder why your data center projects are still ridiculously more expensive than they really are. Don't worry you can still get away with financial hijinks in the cloud to just tends to look slightly different. Exactly I mean. That's that's the interesting part with a lot of these projects like you might have people that are doing the devops and they you build it. You run it world and and it's a great way to move fast. You give each team the authority to do what they need to do. But in many cases like it's super nuanced I mean there. There are bugs that Corean and crew have found with Amazon network data transfer. I think my favorite being that it's cheaper to transfer data between one and two than it has in between availability zones. Something of which. I don't think anyone at Amazon realized or knew it was shocking right. But that's just one of many examples. I mean for for people who have legitimate spend on Amazon or probably any cloud provider Amazon. Being obviously the biggest the number of line items in their bill could be tens of thousands or millions. I mean the number of pages in their bill could be measured in hundreds of pages depending on how they're spending their money and what they're doing with it don't print it out and again you might wonder what a non-legitimate cloud spend is well. That's my ret. Conned origin story were ten years ago. I spun something up as a test that I've been paying twenty two cents a month ever since and so three years ago I snapped swore revenge. And here we are. I feel like that was the other reason I knew that this is something that I enjoyed was similar story. A few years ago I was cleaning out my Amazon account and I I went through and looked and I had this to sent glacier charge. Glacier in my Amazon account did not understand. It was two pennies. I just ignored it like I had been ignorant for many years but house like no. I'm GONNA solve this and I went through the most convoluted series of documents on the Amazon site. In order to find some legacy vault that had been sitting around since glacier was first announced Many years ago to then run a series of commands and I finally was able to delete it through like a bunch of Lupine and a bunch of really terrible code authored. The Amazon Command Line. It probably took me five hours to do that. Hassle we now know leading it. There was there was there was there was. I definitely got hit on the. Api calls. Because I had to do so many calls in order to find like every file and the vaulter whatever the scenario was but the end result was yeah. I spent like five hours to save you cents a month. But it's not on my bill anymore and you really can't put a price on it through manners. Well it's very much I've always felt like because I have that twenty two cent charge too and it's like the friends episode with Ross Cheddar like I want to quit the gym right like it's easier to quit Jim than it is to get rid of that last little bit of of your belt but if you have more than twenty two cent bill. So what are kind of the misconceptions? I don't WanNa say mistakes. But we're people missing when they're thinking about managing cloud spend and this can be a misconception. That's happening at a practitioner level or at a strategic level. Where people missing? You WanNa take this should I? You know I'll I'll start with one that is I think is where people miss out on. And it's not necessarily like you should do this and you'll save money. It's you should do this because a couple years from now you're going to look back and think yourself and that is having a clear concise way of tagging your usage of things in the cloud. But when you think about tagging you wanNA think about Basically how your company makes money by using the cloud and maybe I'm more biased. Because I worked for a lot of SAS companies but the SAS companies are the ones who are selling some sort of service and they're trying to figure out essentially. How much does it cost per user? What DOES OUR COST OF GOODS SOLD? And the reason why you know two year old you. It's smart for adding those tags is because current you is going to have the CFO role over to you one day and say what is our cost of good sold and you have to have a way of mutual. Ca slicing days that data. I think the more challenging part of that is and there's ways to do it in Amazon is to not only have structure policy. Assume your users are not gonNA listen to you or follow it at all and find a way to like enforce that policy so that if they don't create the proper tagging you just start deleting resources right under underneath them because if you start having stuff spin up then you start running reports and it's like oh the state is not tagged or all this all. These servers are not tagged. You're going to spend way too much time walking around and saying is this yours is this year's and everyone's going to say it's not mine and you say okay cool down and then he kind of approaches to block it off with security rules or tightening up permissions but no one understands. I am so yeah. Just turn it off. It's just turn it off. Yeah I take the scorched earth approach. We're in this case. It's not that hard to find another job. Come on another another more common approach. I've seen to across. The board is all the same month. I had two or three customers asked me about Alexa for business. Which is why. Why are you asking? It's a three dollar charge. Why are you? Oh that's right. The bills are alphabetical. And this is a microcosm of what we see across the board. Which is that. We still have all of these customers who are focusing on their internal narratives and the first thing that they see. I had a very early customer. That was focused on cutting the developer environment. Where they wanted to have something technical that would spend things obscene things down have a slack dot weighing in or just getting China's customers and they okay great. Let's go ahead and build that for you. But I was doing analysis and oh development spend is three percent of your bill now originally when you were building this out. It was your entire bill but then your product caught fire. Bill Traction wound up spending up something significant and now the development build. Yeah you could save some money but there is a lot of other things you can do. I don't require disrupting developer. Workflows and have a dramatically outsized impact relative to the entirety of the development environment spent so as focused on the right part of the story and sometimes at one of the most valuable things we can provide. Is that unbiased third party perspective where we haven't been exposed to the internal narratives around what the bill contains. Yeah I think that's I think that you kind of mentioned it before a Jessica. You're talking about the we. We gave everyone the feedback loops around when you deploy thing and there are some plug ins. I've seen for things like Tara Forum and other stuff that you can know what the cost of what you're about to do is gonNa cost you but I I really fear for the future in the Kuban any space because if we already have obstruction that people don't understand what it costs when you suddenly start shoving containers into thing where you don't even know what the need like the underlying servers. Yeah like how do you tag that you know? That's a interesting challenge. I also wonder even if you expose the cost that hey if you press this button developer. It's going to cost X. DOLLARS. Who The hell knows the context of what that amount means right so like to me so that could go both ways right. It could be considered. It could be abundances if you do this. It's going to cost five hundred dollars a month and I'm just sitting there going. Why don't WanNa pay five hundred dollars? That's a lot of money for me but for number one. Hopefully my organization has slightly deeper pockets than you know my my debit card. But but there's the context of what's expensive to me versus the corporation. Also the value right again the context in the nuance of this amounts in a vacuum. Just a dollar amount doesn't mean anything and when you tie it individual teams or users. It doesn't work out. Well then you have counting coming down asking who the hell is Jenkins. They're costing all the money and you know that's that's our continuous integration and build system. It's fine butler. Oh Okay and they leave and then Jordan. Hankins crawls out from under the desk crew. Thanks you really saved my bacon on that one but without nuance or context. It's impossible to look at a bill attributed the users and say oh that's super expensive versus that's normal Y is that person costing a king's ransom every month In aws charges. Oh because they work in data science. And that's what they do you can search through bass quantities of data to unlock the hidden..