Jeff, Debord, Amazon discussed on Software Engineering Daily
You know, you you want the absolutely correct not somebody poking around on part of the data version of the truth. You want the absolutely correct version of the truth that was blessed and built by the data analysis team. And so you still have those old systems, but people have these newer systems, and everything seems great until you from finance get in a room with somebody from sales and somebody for marketing, and they also have gotten data extracts and sliced and diced them themselves, and you go guys what's going on sales. A really down this quarter. Sales goes, no, they're not they're up and marketing goes, no, they're not they're flat. And now you have data chaos. Right. You have different extracts that were pulled from the data warehouse at different times. Maybe those came from different sources, and then people analyze them differently. And that conversion from raw data into sort of business metrics could go really sideways because people had completely different ways of interpreting that data I think the easiest way for most people to think about as XL where in your workbook in your spreadsheet is both the data itself. And then all of the sort of logical transformations that you've made of that data. And you know, if I am working on an excel workbook and have done a bunch of stuff to it. And then send it over you, Jeff, and and you start working on. And then I go. Oh, no, I sent you the the wrong version hold on. Let me send you the new one. And then trying to walk back. Those like, you can see and probably have experienced how quickly that can get messy. And so that's really the world that people are living in. Is where they have to either choose to lock everything down or they have to choose to let people sort of have a free for all where they can interpret the data themselves. And neither of those is great. But in both cases, the core reason the core constraint that drove those decisions was the same which is that databases and data warehouses are really expensive the really hard to maintain and the really slow and so all of your decisions or about how do I protect that data warehouse from my users, and while I would say that the big data revolution has not necessarily fully delivered on its promise. I think one place where absolutely has delivered is on the on the the speed and power of the underlying technology. So you know, I would say starting with the hoop ecosystem and the two thousands. And then certainly continuing with these cloud data warehouses like Redshift from Amazon and Google big query and snowflake and a bunch of others. They are incredibly powerful there. Credibly fast, they cost pennies compared to what the old data warehouses cost, and you can spin them up in seconds. And so you know, that opens up all new opportunities because all of a sudden that core constraint that you were bound by has gone away. So this changing nature of data engineering, and the the cost changes. And also, the, you know, the problem statement that you gave of ending up with different departments having different perspectives on what the quarterly revenue is, for example, that has impacted the evolution of the BI layer. That's the set of problems the set of constraints is in part the inspiration for looker. So you've discussed in large part. The the changing nature of data engineering told me a bit about the Evelyn of the BI layer the the visualization the business intelligence layer from. Excel to modern things like looker. Sure. So I mean, I do think that it follows those waves so in that first wave where you had the data data appliance. You had a very locked down tool that could make those couple of dashboards and Debord it from the same vendor that you bought the data plans from because it all to work seamlessly together. Otherwise, it wasn't gonna work at all in that second wave as people start to get desktop computers that are reasonably fast and departmental servers. You see things like click and Tableau start to come out, which are really powerful tools for sort of doing self service..