Tobias Macy, Cough discussed on Software Engineering Daily

Automatic TRANSCRIPT

Tobias macy welcomed software engineering daily things to thanks for having me your the host of the data engineering podcast as well as the podcast about python which is called podcast dot init so i wanna focus most of our conversation around data engineering i've done some shows recently about it and data engineering started with the duke ecosystem arguably unless you go back to pre hoop days but i think most people started talking about this term around the hoodoo times and then it was followed by these open source tools like storm and cough and then more recently things like spark and flank and then there's a ton of proprietary tools described the time line for data engineering and how it got to its present day state from your point of view yeah so the sort of origination of data engineering i guess if you wanna go far enough back was really with the database administrators because they are the ones who are responsible for making sure that your data was available and backed up and had an appropriate shape and schema for the application to be able to access a but in terms of the sort of modern view of what data engineering happens to be the next duration was in terms of the business intelligence suite of people being able to pull data out of application systems and then modify it and analyze it for being able to create usable reports for the cease suite or people higher up in the business to be able to understand how things were operating and then as you mentioned who do came in and allowed for much more massive scale of data processing and then when that started to become too slow a lot of the streaming systems came in and throughout all of this there is bed and the socalled etl or extract transform and load sort of paradigm with a number of different tools available for them.

Coming up next