35 Burst results for "Kafka"

William Jacobson Argues Democrats Employ the Logical Fallacy of 'Kafka-Trapping'

The Dan Bongino Show

01:55 min | 6 d ago

William Jacobson Argues Democrats Employ the Logical Fallacy of 'Kafka-Trapping'

"They use. I read an interesting piece. It will be in my newsletter today upon you know that calm You want to check it out by William Jacobson is really terrific. It's a piece in the Washington Examiner, where he talks about how these proponents trying to teach your kids to be racist through critical racism theory. How they use the Kafka trap against their opponents. What the Kafka trap is love. These fancy terms, right? Makes you sound super smart, right cocktail parties. Got a little cocktail weenies going around. Hey, can I have one of those? And by the way, did you do a Kafka trap? Lesa makes you sound super intelligent. We're just going to throw it out there because it's true. What's the Kafka trap? The coffin trap is where you use the Nile's of something. To prove that the person is part of something they're denied. In other words, Deny Europe you go up and say Listen, I'm not gonna have this critical racism theory. Talk to my talk to my kids were not racist. In our household. We teach our kids to treat everyone fairly and responsibly to love our neighbor. And what did the leftist come back with the Kafka trap? You're denying it. That's white fragility. You know what that means? You're a double, super racist. And you're like, Wait, What? I just said I'm not a racist. Now you're telling me I'm extra super double, like mutant powered racist like how does that work? You are denied the stronger You deny it. Stronger. You're a part of it, and you're looking at them. You scratching your head like, uh, My living in like stupid ville. Am I living in a vacuum of dumb on the event horizon of an intellectual vacuum? Is this happening? That's the Kafka trap. Here. You want a definition caught this online. Kafka trap Replace X by the way with racist here and you'll see what I mean. A Kafka trap is a fallacy where if someone denies being X, it has taken as evidence that the person is X. Since someone who is X would deny being X. It's the ride from the novel The trial by the Czech writer Franz

William Jacobson Washington Examiner Europe Kafka Franz
Ben Okri Reads Franz Kafka

The New Yorker: Fiction

01:37 min | 3 weeks ago

Ben Okri Reads Franz Kafka

"Hi ben. Welcome so when we started talking about doing the podcast. one thing. That was very clear to me. Was that you feel an affinity with kafka's work. What is it that makes what he does important to. You is hard to say the more you read kafka the more confusing it is actually He's someone who gets more mysterious with more. Acquaintance is very strange. And i think it's the deceptive quality. Has i think. Actually i think is the way. His mind probes reality. It's a universalising quantity that his mind has. He's trying to himself either because of some deep trauma in his life because of what he felt about life at trained himself as a storyteller to constantly allegra allies while at the same time being deeply particular and nobody no other writer developed it to the degree That he had and also like his voice is voices. Most peculiar of course. I don't read the german and the same german. His voices peculiarly plain sometimes bureaucratically plane but all translations into english and michael's we saw most beautiful while the best he. Has this voice that. Tom manages to bypass your your brain and it goes straight into your psyche. He's found this tone this very normal tone that just slowly shifts Into completely unexpected places without any without any striving

Kafka Allegra Michael TOM
Apple and Facebook Are All Ears

Reset

01:41 min | 2 months ago

Apple and Facebook Are All Ears

"Okay fair warning. We are about to get a bit meta this. Podcast episode is about the future of podcasts and why to tech companies apple and facebook are plunging feet first into the audio pool recruits. Peter kafka's. you're talking about. Hey peter so over the last couple of weeks there's been a bunch of talk about how big tech companies are getting into audio. There's apple and facebook. Let's start with apple which has more experience in this space. Tell me what apple is doing in the world. Podcasts apple mostly created the market for podcasts. And i say that knowing that people in the madame because apple did not technically invent podcasts. But they're the ones who sort of brought it to the mainstream back in two thousand five and then they pretty much left it alone. They have not tried to do much with podcasting. They have not assigned many people They haven't put much resources toward it and they haven't tried to make any money and now that's changing a little bit. They are going to allow individual podcast creators and publishers to sell subscriptions to podcast within apple. And so there might be a sm- they're gonna say smallish business. I think it's gonna be a huge business but it's gonna be something for apple so they've gone for making no money on it to making potentially some money so apple's going basically take a cut of podcast revenue that's made by a show host or pretty sure is that is that right. That's exactly right. It's the same same model. They have for apps and services like hp max etcetera. The publisher over makes the thing keeps seventy percent. Apple keeps thirty percent if you keep doing it for more than a year that cut goes to eighty five and fifteen

Apple Peter Kafka Facebook Peter HP
Netflix might be cracking down on password sharing

Here & Now

03:42 min | 3 months ago

Netflix might be cracking down on password sharing

"Netflix users. beware. The company is considering a crackdown on customers. Who share their passwords with friends or family members. Netflix knows many of its two hundred million subscribers or piggybacking on someone else's account and now the company is testing a new papa message asking some users to verify that they own the account. They're trying to log into. Peter kafka is senior correspondent at recode and host of recode media peter either. So what is netflix up to here. Well that's a good question Their official line is just a test. Who knows what the who's who knows what we're gonna do but in theory. It looks like they might be reconsidering a longtime stands. They've had which is basically you. We're not telling you this but we're not gonna complain if you share your net password with a lot of other people They're doing a test. That says if for instance you've ever borrowed a netflix password You might see a pop up on your tv saying hey. You can only do this if you have a netflix account with us and if you do let us know and we'll send you a log in and you can get on yourself. It seems appears to be nudging you towards going from free loader to paying netflix's subscribe well. It does seem to me that it could be like the end of an era because people have been sharing passwords for as long as netflix has been around. You know just a few years ago. Ceo reed hastings said password sharing something that you have to learn to live with. So why crackdown now. Well we don't again. We don't know they're cracking down. You know the the obvious conclusion you can draw is. Netflix used to sort of own streaming. That didn't have any competition and now there's a lot of competition from disney discovery plus and paramount plus. Hbo max and the suggestion would be all right. We people paying for us instead of paying for hbo max. Paramount plus people make a decision I'm not sure that's the case There's also a theory that h netflix's just taste testing out a count security you know. Netflix doesn't give free trials anymore. They used to do that. They've they've moved away from that And so if you wanna tease this all the way up and saying look you know what net flicks is you like it. It's time for you to star paying It would also suggest that theory holds up. That netflix is thinking we've got two hundred million subscribers worldwide. Seventy million in the us. How many more are we going to acquire With getting the freeloaders start paying but again redoing speculation here sure and do you have any idea how common it is for people to share passwords. I mean i. I i i personally. What happens a lot. I've shared my password for my entire family at various points. And i think they've all ended up paying for their own and in one point. I think that for a long time for for a lot of these dreamers was look. We'd rather have you paying but in the but you're also sort of giving us remarketing There's a there's a consulting firm magazine associates that that estimates that may be thirty percent of of netflix's users are are are are sharing. I guess passwords anecdotally. It's it's a lot of people because there really hasn't been any reason not to You did here During the pandemic people who'd been used to sharing their passwords. And you know In theory netflix. Lets you stream multiple accounts at the same time during the pandemic when everyone is at home streaming netflix. You've heard the people were running into that limit already. So this is something that could have been stirred on stirred up by the pandemic business has been pretty good. I think for netflix gained an extra thirty seven million customers during the pandemic because of all those people staying home watching the

Netflix Ceo Reed Hastings Peter Kafka HBO Paramount Disney United States
Lew Cirne on founding Wily Technology and New Relic

Software Engineering Daily

03:50 min | 5 months ago

Lew Cirne on founding Wily Technology and New Relic

"Lou. Welcome to the show scrapes video. Thank you you started new relic awhile ago and before that you started a different company wiley. Both of these companies were focused on what we now call observability hauer software applications today different than from when you started new relic. Well yeah you know. I'll talk about what's different today but also talk about what the same and i guess. It's what's the same as so long as our software there is going to be bugs and they're going to be problems that happen only in production and that's that'll be true forever. I think so as long as humans create software and just like in the medical field are so long as people get sick. Then there's going to be a need for doctors. And so i think so. Long as there are software. There's going to be need for tooling and visibility capabilities to help understand. Soffer behaves when it's running under load in particular and take that understanding to improve the performance availability stability and the customer experience of that software. So when i started wiley twenty-three years ago the idea was this brand. New thing. At the time in ninety eight was java and the idea was let's see inside. Jvm without asking our customers to change any of their source code and put that visibility to production low overhead and captures much data as possible and presented an easiest way possible to help customers debug their jvm's and fast forward to two thousand eight. When i found new relic the thought was well. It's a multi language world now and applications aren't running on two or three physical servers are running on twenty or thirty or so back in that time. Virtual hosts and it was very early in the cloud but people those hosts for running increasingly in new environments like amazon web services. So the idea was. How do you put visibility into that are composed of say a half dozen services running in a virtual environment where. There's multi-language in that. That really was the sweet spot. If new relic when it was founded through the first several years and now here we are in two thousand twenty one what an application looks like today is often hundreds of services thousands of containers more and more in coober netease incredibly complex. A lot of a synchronous work a lot of stuff going on systems like kafka and so trying to make sense of a really complex system is more challenging than ever and it seems like what's behind all of this complexity is imperative to help developers be more productive to first of all have smaller more independent teams who can deploy with pretty good isolation and rely on good. Api's and things like that to allow lots of those teams to collaborate on a large effort at high velocity but so that that increases velocity but it comes at the cost of increased complexity on how that whole integrated system works and the solution to that in our opinion is complete visibility into all the application micro services all of the infrastructure and the end user experience. All into a common platform that operates at massive scale and really the guts of observability. If you understand the difference between observability monitoring i'd say monitoring is about telling you when something's wrong but observability is having access to all the telemetry need to answer. Why is something wrong which you don't even know what question you need to ask next to get to the understanding of what's wrong in today's world it just like collecting a massive amount of data and trying to make sense of it. Is you know as rapidly as possible.

Wiley Soffer LOU Kafka Amazon
HBO Max activations double to 17.2 million in fourth quarter

Techmeme Ride Home

03:08 min | 5 months ago

HBO Max activations double to 17.2 million in fourth quarter

"At and t. also reported earnings and we don't normally care about their quarterly numbers and actually don't care today what we do care about is they're streaming numbers. Hbo max went from being sort of a punchline sort of an also ran in the streaming worst to suddenly. All of their movies will be available to stream day of release at no extra cost. Maybe we need hbo. Max our lives after also did wonder woman nineteen eighty-four moves the needle for them. Early signs are pointing to. Yes apparently if you take. Hbo and hbo max subscribers and put them in one bucket. That number grew twenty percent year over year to forty one and a half million subscribers reaching number that. Hbo was hoping for two years ahead of their own forecast. Hbo max activation doubled to seventeen point two million since the end of q. Three quoting variety. The company said it invested about eight hundred million dollars in. Hbo max in the fourth quarter and more than two billion dollars for the year on the streaming service these subscriber gains in q four were undoubtedly boosted by warnermedia's in four to distribute. Hbo max on roku and amazon's fire tv on the q for call with analysts. At and t. ceo. John stinky said were media is aiming for a cue to launch of a price reduced ad supported version of. Hbo max but he didn't provide other details on the avio de product. The company plans to launch hbo. Max internationally this year. Starting with latin america in the second quarter stinky said. At and t. will host an investor event in the second. Half of q one to outline this and quote so again as i said before ad-supported. Hbo is not hbo. That's tv sorry. That's an obvious joke. But i did want to mention that little bit about the money spent invested in. Hbo max and also where the subscribers to hbo. Max are coming from because when disney plus adds a new subscriber it is completely new revenue with hbo. There's this weird situation where did eight and a half million people start paying fifteen dollars a month for. Hbo max out of the blue. What number of them were just people moving from. Hbo regular to hbo. Max like if five million people joined. Hbo max for the very first time. 'cause of wonder woman. And they're totally new subs and they stick around for a full year. Then yeah warner definitely made back the potentially one billion dollars in wonder woman nineteen eighty-four box office. They should have made if everything was normal. But if not then the math is not so good and also. That's only one movie. They have a slate of eighteen. More still coming this year. Remember that conversation. We had with peter kafka on a weekend bonus episode not to long ago. No one is sure at least not yet that the math actually works out on this whole streaming business. At least for anyone not named net flicks.

HBO MAX Warnermedia John Stinky Roku Stinky Amazon Latin America Disney Warner Peter Kafka
Redpanda is a Kafka Alternative

Software Engineering Daily

00:26 sec | 5 months ago

Redpanda is a Kafka Alternative

"Africa has achieved widespread popularity as distributed queue and event streaming platform with enterprise adoption and a billion dollar company. Confluence built around it but could there be value and building a new platform from scratch. Red panda is a streaming platform built to be compatible with kafka and it does not require the jvm nor zookeeper both of which are dependencies. That made cough harder to work with and perhaps necessary

Africa Cough
Redpanda is a Kafka Alternative

Software Engineering Daily

00:26 sec | 5 months ago

Redpanda is a Kafka Alternative

"Has achieved widespread popularity as distributed queue and event streaming platform with enterprises option and a billion dollar company. Confluence built around it. But could there be value and building a new platform from scratch. Red panda is a streaming platform built to be compatible with kafka and it does not require the jvm nor zookeeper both of which are dependencies that made kafka harder to work with and perhaps necessary

Philadelphia Eagles head coaching search tracker

Bleeding Green Nation

03:13 min | 5 months ago

Philadelphia Eagles head coaching search tracker

"Cody. Thanks for coming back on. I in the enemy man. How are you. I'm doing great great to be with you. I'm excited to talk some football absolutely and let's jump right in to the eagles. Coaching search here. Because more names get added to this list everyday to say that the eagles have cast a wide net would be understating it a little bit. You tweeted out. And i think it was either yesterday or today but as a recording. This on the fourteenth is was on the thirteenth. You tweeted out a whole list of names. We've got carolina offensive coordinator. Joe brady tennis offensive coordinator arthur smith san francisco defensive coordinator robert soleil tampa bay defensive coordinator. Todd bowles new england linebackers coach. John mayo kansas city quarterbacks coach mike kafka philadelphia assistant head coach do staley oklahoma head coach lincoln riley ohio state head coach ryan day. Cincinnati head. Coach luke fickle and more are coming each and every day and i know among eagles fans. You know we see this as a pretty dysfunctional team right now. We see this as an organization. That doesn't seem to know exactly what they're doing and we're skeptical as to whether or not a a really high profile coach is going to want to come here given the list of names here. Where are you at on this eagles. Coaching search as there really are just getting started right. Now yeah and we saw it just kellen more added today and obviously you know they're going to continue to cast the net and i think just in general. I think there's a little merit to the idea that you know when you compare openings. The eagles aren't A bursting with attraction just because of the situation. I mean Doug peterson clearly. If the reports are to be believed is is kind of worn down from from this run with eagles. And but i honestly don't put too much stock into this idea that they're undesirable because at the end of the day there's thirty two jobs. There's thirty two head. Coaching jobs in the nfl. We see guys in view with with teams like the jets and the jaguars every coaching cycle. Yes they sometimes have high picks so they sometimes have a lot of cap space but at the end of the day. It's nfl fell head coaching job. You can't tell me you know if you're joe brady and you're getting interest from all these teams. I mean you're not going to just not going to turn down the opportunity and so And eventually some of the spots are gonna get filled For me i mean i. I start for the eagles with looking at the offensive side of the ball. I know that's it's a big talking point. Jeffrey lurie with with you know really any eagles fan. That's been under or watching. Jeffey larry hires run. The team I i think really it comes down to are you going for You're gonna have to sacrifice somewhere. Because with somebody like joe brady even kellen moore. You're really banking on them. Being kind of an offensive mastermind and up and coming mastermind as opposed to this Person who's been around the league for years and respected and you know doug peterson even though he didn't have head coaching experience. Jeffrey loria himself talked about that emotional intelligence he talked about. Doug peterson former quarterback. In this league he had time under andy reid and so i. I think it's just a matter of where are they gonna sacrifice. Because if they go with robert. Sal the forty niners you're sacrificing. You might bring in a nice offensive coordinator with them but he may leave here for another head coaching job. And so what do they want. We're gonna find out.

Eagles Joe Brady Robert Soleil Todd Bowles John Mayo Doug Peterson Mike Kafka Lincoln Riley Ryan Day Coach Luke Fickle Arthur Smith Cody Tampa Bay Kellen Kansas City New England Cincinnati Football NFL Tennis
'The Great Gatsby,' 'Mrs. Dalloway' And Other 1925 Works Enter The Public Domain

Houston Public Media Local Newscasts

00:57 sec | 6 months ago

'The Great Gatsby,' 'Mrs. Dalloway' And Other 1925 Works Enter The Public Domain

"Today is public domain day. As of january first thousands of books movies songs and other material from nineteen twenty five are no longer under copyright protection including the great gatsby. Npr's neda ulaby has more besides the f. scott fitzgerald masterpiece books entering the public domain now. Include mrs dalloway by virginia woolf and classics by sinclair lewis franz kafka ernest hemingway and agatha christie so are other works from nineteen twenty five like buster. Keaton silent film go west and the songs week toward brown now community. Orchestras can play music in the public domain for free scholars will not have to get permission to study. This material and books on the public domain can appear online without charge all part of living cultural conversation that anyone can join netto lippi. Npr news both

Neda Ulaby Mrs Dalloway Scott Fitzgerald Sinclair Lewis NPR Virginia Woolf Franz Kafka Ernest Hemingway Agatha Christie Keaton Netto Lippi Npr News
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

06:06 min | 6 months ago

"kafka" Discussed on Software Engineering Daily

"Observers you get your profile back. That part of the request thing doesn't go through governor so that all goes through like you said. Heb we use so wrestling is our free mark to do like rpc's and is what we used for. The real time do need fast response things. All that produces kafka traffic one way or another sometimes hidden from the actual product application. Like we move. Like i mentioned quickly or even the doing eventually a lot of that. Etl deal doubt. It can be processed in and academics will be served and again move through kafka so-so kafka actually exited the middle system between online and offline. And all that we'll go through one way or another from the perspective you could be in the profile team doing something and you would not directly. You would use some system that will do kafka for you one way or another. Even if you're doing things. I guess but would be. You're looking at people. You may know right so people you know. Is we compute so we can suggest. Hey you may know this person and that will go through in many directions. But at the end of the day you're calling our vw's deters you data after it was computer in offline and you would not call kafka directly so let's say user loads a page that includes the people you may know module. Does that mean that when the user requests the people you may know there's some service that's going to get the data from the database and serve that to the user but a synchronous -ly there is maybe an event that's written to kafka humans many events okay and they'd be populated cover pop. Okay right got it so it was kafka. Is it like an asynchronous notification system that allows anybody to create and read events that describe the traffic that's happening across lincoln's infrastructure. You could think about it that way. Versus is a little bit more structured so in our case all the click data will do all that generates eight other kind of inches. Etl for analysis right and anybody can subscribe as you have permission so we have apples and all the stuff but assuming you have permissions you can subscribe to that and then just start processing that data for whatever reason. There are many reasons why you might use the cedar things that happened. That transactional like you go into edit your profile that communication that your profile goes to our database right are service internally. That ends up replica hidden from you. There's a leader that source it from the movies. We put obviously change events. And that goes into cocker. And then now you can take action on those events like something change like you change your title from we built many systems for example will take actions on these and an augment your profile in one way or another good example is actually the way we do sensation of titles rain so example. I'd like to use. Is you call yourself java guru and that's what you put because we allow you free form entry and you say oh i'm a java guru at the end when we're suggesting jobs to you we're probably not gonna suggest meditation or probably not gonna suggest coffee. We will suggest software engineering jobs and because some systems saw the title decided that process. This said you know what that actually means. Software engineer job expertise and then we augment your thing to be able to do that. And that is because an have generated out of the data that could change in our database. There's a quote pattern that is sometimes mentioned in this kind of discussion called event. Sourcing i don't know if you think this is a useful term but maybe could you give a definition of event sourcing and describe whether it applies to the situation. Yes i have to admit. I am not clear. What invent sourcing encompasses whether it is the fact of having multiple systems use the same event to deal different things or whether it is a series of events that you can use to reconstruct something from the start and you have all the events he can now like. Whatever backtrack right look into play. I am not certain which of those determine means we do. Both of those are different. Systems will deal with this in one way or another right. Obviously we have systems that can replay stuff and like construct state and goten necessary and we have things where some changing system is replicated to multiple systems downstream to take different actions. So both happen. Okay so let's zoom in on just the subject of the word event so event is a data type that describes a change that has occurred in the system and if you took all the events that happened across linked in i think to some degree to describe the updates to the databases that are happening across lincoln's infrastructure. All these events are getting stored in kafka. To what extent do you need to keep those events like. How long do you need to keep them for. Because if this event data's describing everything that's going on across all of lincoln do you want to save all of those events. Do you need to keep them in memory. do you want to flush them to disk some interval. Could you just discuss the topic of event data. Yeah i don't think you're podcasts. As long enough to have discussion. But let's talk about it for a little bit chair. Yeah if it was my choice. I would keep all events forever of all type and have the latest snapshots like do any computation. Any point in time. That not always possible in the case of kafka kafka has retention and durable so we stored to disk so a cocktail. Sorta this dnc central kafka after sometime. It'll flush him out and delete by by default. Currently we have a four day retention on like new topics that created with no specific settings multiple pipelines. But let's let's use that. As a general term it could be less it could be more. We can configure it at the end of the day though. There are multiple reasons why you don't want to keep everything and took off. There might be changes happen. It costs you money to keep all the data which is not considered source of truth only temporary buffer. But apparently for those that haven't worked in this area it also has legal ramifications right. If you're a system that stores data you have to comply with a bunch of regulations and you try not to have all your systems..

kafka lincoln wrestling kafka kafka
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

03:52 min | 6 months ago

"kafka" Discussed on Software Engineering Daily

"You want to Streams doesn't have any opinions about that at all kind of stays out of your way with link it's this special thing that goes to the other cluster that the you also have to maintain got it to take a step back. There are organizations to this point that have pretty mature kafka installations. Can you tell me about what are the struggles that late stage kafka users hat. Is there anything in in terms of cost management or infrastructure sprawl. What kinds of issues do large kafka users half a schema is always a thing right because I'm talking about this wonderful world. In which you just produce messages into topics and and thousand services sprout out of them and each one is unique and beautiful flower. You know which is is all true but like hey you know what's in that topic We kind of need to know that so. Scheming management is a thing that becomes more important at scale and because at scale. There's a topic. And i don't know the folks who work on that topic. I don't know what's in it. I don't. I can't go ask them easily. The wikipedia is out of date blah blah blah. You know if. I want this evolutionary architecture to happen at scale Which is even better than at a smaller scale. I need to have a reliable way of knowing what's in that topic and complement has a component the conference game a registry and it's available as part conflict cloud. It's fredin downloaded use it. It's just. It's it's not a part of apache kafka. I mentioned conflict. Because it's it's our thing and it's it's not a part of the apache project but it it helps with all those things so schema is a big deal and i always see big organizations struggle with standardizing on. Api's and all right or wrapper around things which tells me that there's probably a good reason for that You know i. I have not managed engineering team at the scale of one thousand engineers in a multibillion dollar business. So i don't. I don't know what those things are. I can say i never like it right. I never liked being company. X. developed the company x. wrapper around the core coffee. Api's but they all do it and so kind of getting that right and building building standards building compliance into those things you know you dissolve the stuff that you just wanna make sure everybody. Even if they're not a kafka expert does the right thing and everybody does that by building a rapper and seems like those rappers are usually a big investment. Like i said. I personally have a negative reaction to them but it's hard for me to make the case that there amal investment because a lot of people do them and they do have a lot of hard problems to solve at scale to wind down the conversation. You work at confluence. Conflict is a company that builds application systems and do consulting style. Work around kafka and conflict has grown tremendously the company's doing really really. Well tell me about your work at conflict and what it's been like over the past two and a half years. I think it's been like what's been your experience at the rapidly growing kafka company. Yeah so. I always wish i could say that. I came to confluence. Because i'm really good at being an industry analyst and picking on said hey this caucus thing is going to be good and it just wasn't that at all took me about six months after getting here to to realize oh wait. Hey seriously this.

kafka
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

03:52 min | 6 months ago

"kafka" Discussed on Software Engineering Daily

"Heck you want to Streams doesn't have any opinions about that at all kind of stays out of your way whereas with link it's this special thing that goes to the other cluster that the you also have to maintain got it to take a step back. There are organizations to this point that have pretty mature kafka instiutions. Can you tell me about what are the struggles that late stage kafka users hat. Is there anything in in terms of cost management or infrastructure sprawl. What kinds of issues do large kafka users half a schema is always a thing right because I'm talking about this wonderful world in which you just produce messages into topics and and thousand services out of them and each one is unique and beautiful flower. You know which is is all true but like hey you know what's in that topic We kind of need to know that so scheme. A management is a thing that becomes more important at scale and because at scale. There's a topic. And i don't know the folks who work on that topic. I don't know what's in it. I don't. I can't go ask them easily. The wikipedia is out of date blah blah blah. You know if. I want this evolutionary architecture to happen at scale Which is even better than at a smaller scale. I need to have a reliable way of knowing what's in that topic and complement has a component the conference game a registry and it's available as part conflict cloud. It's free it and use it. It's just. It's it's not a part of apache kafka i mentioned consulate because it's it's our thing and it's it's not a part of the apache project but it helps with all those things so schema is a big deal and i always see big organizations struggle with standardizing on api's and all right or wrapper around things which tells me that. There's probably a good reason for that You know i. I have not managed engineering team at the scale of one thousand engineers in multibillion dollar business. So i don't i don't know what those things are i can say i never like it right. I never liked being company. X. developed the company x. wrapper around the core coffee. Api's but they all do it and so kind of getting that right and building building standards building compliance into those things you know you dissolve the stuff that you just wanna make sure everybody. Even if they're not a kafka expert does the right thing and everybody does that by building a rapper and seems like those rappers are usually a big investment. Like i said. I personally have a negative reaction to them but it's hard for me to make the case that there mel investment because a lot of people do them and they do have a lot of problems to solve at scale to wind down the conversation. You work at confluence. Conflict is a company that builds application systems and do consulting style. Work around kafka and conflict has grown tremendously the company's doing really really. Well tell me about your work at conflict and what it's been like over the past two and a half years. I think it's been like what's been your experience at the rapidly growing kafka company. Yeah so. I always wish i could say that. I came to confluence. Because i'm really good at being an industry analyst and picking on said hey this caucus thing is going to be good and it just wasn't that at all took me about six months after getting here to to realize. Oh wait hey no seriously this.

mel kafka
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

09:21 min | 6 months ago

"kafka" Discussed on Software Engineering Daily

"Roll ups or or of the events as the events hap now. There's an abstraction called a data lake and most people who are still listening to this point probably have familiarity with the data lake. It's a thing that we're dumping a ton of data into kafka can look like a data lake in in in some ways because if you're dumping raw all of these the change data capture all of the changes to your infrastructure through cough khou and maybe it's getting flushed out of there some kind of garbage collection policy to it but in any case there's a lot of data there. How does the usage of kafka compared to a data lake that is. I don't think i've reflected on that question before. That's a great question. It's not a lake. It's a stream jeff. So i don't know but let us plan that's cheesy but let's let's explore that analogy just a minute. The idea with the data lake is that you do put things in it and you'll opportunistically go back and kinda fish through the lake later on you'll right spark jobs. Let's say now is probably the thing you're gonna do to do those batch analysis of that data and i'll be frank. I don't think kafka makes that a bad idea. I think having something like a data lake for offline experimental data science development is probably recommended in systems of even medium-scale. What's different about kafka is. I think ultimately operational expectations. That when you do a thing in a data lake latency is not too worried. You know everybody wants everything to be fast. But if a spark job takes a few minutes to run on a large set of data that feels pretty good and a latency of a few minutes through a kafka cluster for streaming pipeline would be pathological in the extreme latency of five seconds is something you want to stop and look at and figure out what's broken so i feel i feel like i'm missing something subtler. I'm just going to go with that for right. Now that you know there are similar in that they both have water in them but hey water flows one of them. And if you're trying to build a real time etl pipeline. You will not do that with a data lake. You definitely need a streaming technology for that. Well let's talk more about the streaming technologies to stream processing is obviously a tool that is widely used with kafka there are cough streams than there are other streaming frameworks like flink suggested patterns for which streaming tools to use in which context. Yes so. I spend my time as an advocate for things in the broader kafka ecosystem so unsurprisingly. I normally come out with a fairly costco friendly set of recommendations worth talking about flink though. Because it's a super powerful framework lot of smart people behind it a lot of interesting adoption. I think we are seeing a pattern. Emerging where flink gets adopted where there is a big stream of data to process like. it's the feeling of. Oh wow i have this giant stream. I have to do things with it. And almost as if there's one big stream and i need to manage one extreme. And that's true because operationally flink is its own. Cluster in kafka you have cluster of the brokers doing storage and sub and in flink that kafka cluster is going to feed the flink cluster on which distributed stream processing computation takes place which means my stream processing programs are kind of these special purpose things that i deployed to the cluster. Using that clusters opinion of how applications are built in deployed. There's like one way to do it. You run that application and super simplifying the account of flink. I know you've had great discussions out on the show and there are a lot of interesting things to say about it. But because of its deployment decisions it seems to be gravitating towards this giant stream process. It kafka streams takes a very different approach in that it's a java. Api literally dependency that you put in your build file. And it's an api you code against and so you have some application. You know one of these micro services that's doing whatever it does like maybe it's a spring boot app with a web front end or some restful thing that it's exposing to the world you know whatever it's it's a program does things plus also it has to do some processing of some topics to get some key value stores arranged inside the service to be available to that spring boot app to do things well kafka streams gives you essentially like conceptual parody with link. Were all doing the same. Things were joining and we're filtering aggregating and we have lower level. Api's that we can dig down and do more detailed things so you're doing all those stream processing things but you're doing them in the context of your application. There's a scale story for that. By the way for how you horizontally scale that application since you're adding all the stream processing application your server getting hotter you're going to scale it and that that is simple to do to scale the application normally caucus terms but that's the idea right so when that stream processing is married to some other application functionality. Like i'm a micro service and i wanna process. Streams feels incredibly difficult now to go do that with flink and very natural to do that with kafka streams. So it there. There's more to that story but just as a very first blush way of differentiating them. That's how i do it so this is to say go ahead. I want to get case equal in there to go for whist on this all right so this gets back early on you asked. is it. Okay for service. Just to consume from a topic to have to do kafka streams and stream processing and i Yes and we'll we'll talk about it more later. It's totally fine. So if you're a service and there's a topic that has what it what you need in it for you to do your work. You just consume that thing and go on your merry way and you start to look a lot like server. La's function at that point. There's this event you use stateless. -ly do your work and go on your way and that's a beautiful way to live. It's unlikely that that topic is just there for you lying around you know when you're writing new service. Acts and that micro service is sort of starting to feel more like Service function a lambda or something like that. Because it's a really simple stateless micro service. How do you get that topic. Ready with that stuff in it. Well i could go right another caucus dreams application and deploy another service. That does the joining an grouping and whatever computation. I need to on the various streams to get the work ready for that other thing or i could use k. Sequel kissy will does more than this by the way. I'm giving you one use case it's like it's a sequel like language to do the same kind of stream processing kafka streams does so you can write a align or a few lines of sequel to kind of prepare that work for that service to digest and it may be clear also that in the streaming etl pipeline case that we were talking about a minute ago. Casey sequel tends to pop up as the way people prefer to do the analysis of things once. You've got the data in a topic you can just kind of case sequel it into the form that you want and then dump it out into whatever system you're using to visualize so i think one way of interpreting wave said is that if you're looking at different streaming frameworks one trade off you're going to be faced with his. How close is the streaming framework. Going to be sitting to your data and from what i heard this. Streams library is going to be sitting on the same application. That is that is running kafka or the same server. That's running kafka the same abstraction that's running kafka so the data is right there as opposed to flink not on the broker it's not a broker. Okay not on the broker. Yeah it's it's in a consumer basically so it's still your application which is running outside the consumer. Okay so why is it that that's able to get you to lower latency than something like flank. I don't oh. I don't want to say on the air that it does because i actually i could not produce for you. Benchmarks that validate that. That's true so. I don't know that it's lower latency. I do know that it's lower deployment complex oxo okay. Yeah 'cause in flink you like. You're writing your quote unquote flink program. And deploying it to the cluster and kafka streams you're writing a micro service and you're using the stream processing api and you deploy your micro service however the heck you want to Streams doesn't have any opinions about that at all kind of stays out of your way with link it's this special thing that goes to the other cluster that.

flink kafka costco jeff frank La Casey
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

07:11 min | 6 months ago

"kafka" Discussed on Software Engineering Daily

"To be here. It's been too long for which i accept full ability. It has been too long and today. We're gonna talk about applications of apache kafka you and i have talked about kafka in other contexts. In the past and more broadly on this show. We've done plenty of previous episodes about the basics of kafka kafka's distributed replicated event queue. We'll get into the more advanced topics but speaking from a industry perspective. Why is it distributed. Event queue an abstraction that is so widely applicable. Why are people using this thing and many different ways. Yeah the will. It's how we built databases right. You have to remember when you think about a database as a developer. You think about the api. And maybe if you think about a database of course that api sequel and you know you've got underneath that you've got some sort of tabular data model or whatever data model and maybe if you think of a database as da as a person operating it you think of what's going on in the file system and maybe indexes and and concerns like that but inside kind of at the center of the tootsie pop inside of every database is an event log a commit lock right. That's things start and this is a little bit. I think sometimes little bit cutesy of of me to give this explanation but if you think of a database it's sort of like there's this log of events and you build up. These materialized views on top of that log. Those are tables indexes and things like that because reading the log would be a terrible way to live so you know we've virtually always built our databases that way and now that we're building you know. More and more of us are in the business of building. At least small distributed systems. Were finding that that. Commit log at the center of things is useful paradigm and it solves a lot of problems and allows us to make some simplifying assumptions about the system and gives us ways of growing the system that we wouldn't have otherwise the earliest days people thought of kafka is just as thing for publishing and subscribing to message topics since. Then there's been higher level abstractions. There's been different. A is different systems believed around it. Describe some of the abstractions that have been built into the kafka ecosystem. Yeah I love this history. So it's what you see. Is you see the core thing gets built. And that's this distributed lock. And that's that's what kafka was at first it was a distributed log and like you say at that point duke was a big deal. I have a big giant hdfs cluster. I have some things that aren't in the hdfs cluster. And i need to get them into the hdfs cluster. And so you know. Kafka was sort of handmaiden to dupe always getting data into it as a big scalable pipe and part of the what we call the big data ecosystem. Then people noticed things right. People noticed for example. Well there are these kind of standardized legacy interfaces that. I need to write code for all the time. Like there's a relational database. And i don't mean to call relational databases legacy. But you know. There's a legacy system with a relational database. And so i need to write some code. That does a select on the database and finds new records and produces them into a cough topic. Well you're only going to write that two or three times before you realize you should extract that into a framework and then maybe you've done some work in kafka and you've got some topic that has processed data and you wanna put that into files in an s. three bucket or something and so you know you're only going to write that. Api code two or three times per four. You realize there's a framework there. And this was the first major piece of kind of application framework that kafka as a platform added and that was connect so the first piece of evolution that you're talking about was kafka connect and that was this plug -able extendable standalone integration framework because the community saw that people were solving that problem over and over again and everybody always ended up with their own buggy partial implementation. And so now there's a good one and there was like this rich ecosystem of connectors. So that's kinda cool after that. Okay i can get data in and out from standard things and everybody doesn't have to right the same as code or that same. Elastic search code. That doesn't differentiate. you doesn't add any value. You just do it one time. You get that connector. After you got that people realize okay well now my consumers those are the application programs that are reading from kafka topics. They end up doing interesting things right like they aggregate messages. They the group messages. They compute aggregations. You might enrich one topic with data in another topic or something like you get a message on a topic and you need to go. Look thing up so this just happens. If sometimes i get the the privilege of talking to enterprise architects big bank or something like that. It's always a fun conversation and you can say hey folks you know. What are your consumers. Do and those three things that i just mentioned while filtering aggregating and joining. That's like what everybody does and the recognized. Oh wait you know. We're doing this over and over again and it's really hard and there's a bunch of problems we haven't solved at so kafka streams emerged as of kafka zero point ten as the standard framework for doing that kind of stuff in your consuming applications and doing it in a scalable way. That's a pretty sure you've episodes on that. That's its own topic. But those are examples of kafka growing platform components and leaving its origins as scalable pipe behind and starting to say to the world. Hey look. I have an agenda for how you build applications. Let's talk. I think one of the reasons why kafka has become so prominent is that you often have in modern application development a large disturbance system and you want to synchronize state of that distributed system across your different clients across your different services. Can you talk about how kafka fulfills the purpose of synchronizing state across a large application. Yes yes and by the way. I think that is an insightful. Account of what a distributed system is it is sort of an attempt on the part of many programs running on many synchronized processors to create an evolving state. And try to come up with an account of it so the problem with state is that it changes and if you have lots of copies of that changing state in different places negotiating the changes to those copies is very difficult to come to the conclusion that you know yes as a system we have a consistent view of what is true right now.

kafka kafka kafka Kafka
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

07:11 min | 6 months ago

"kafka" Discussed on Software Engineering Daily

"It's great to be here. It's been too long for which i accept full ability. It has been too long and today. We're gonna talk about applications of apache kafka you and i have talked about kafka in other contexts. In the past and more broadly on this show. We've done plenty of previous episodes about the basics of kafka kafka's distributed replicated event queue. We'll get into the more advanced topics but speaking from a industry perspective. Why is it distributed. Event queue an abstraction that is so widely applicable. Why are people using this thing and many different ways. Yeah the will. It's how we built databases right. You have to remember when you think about a database as a developer. You think about the api. And maybe if you think about a database of course that api sequel and you know you've got underneath that you've got some sort of tabular data model or whatever data model and maybe if you think of a database as da as a person operating it you think of what's going on in the file system and maybe indexes and and concerns like that but inside kind of at the center of the tootsie pop inside of every database is an event log a commit lock right. That's things start and this is a little bit. I think sometimes little bit cutesy of of me to give this explanation but if you think of a database it's sort of like there's this log of events and you build up. These materialized views on top of that log. Those are tables indexes and things like that because reading the log would be a terrible way to live so you know we've virtually always built our databases that way and now that we're building you know. More and more of us are in the business of building. At least small distributed systems. Were finding that that. Commit log at the center of things is useful paradigm and it solves a lot of problems and allows us to make some simplifying assumptions about the system and gives us ways of growing the system that we wouldn't have otherwise the earliest days people thought of kafka is just as thing for publishing and subscribing to message topics since. Then there's been higher level abstractions. There's been different. A is different systems built around it. Describe some of the abstractions that have been built into the kafka ecosystem I love this history. So it's what you see. Is you see the core thing gets built. And that's this distributed lock. And that's that's what kafka was at first it was a distributed log and like you say at that point duke was a big deal. I have a big giant hdfs cluster. I have some things that aren't in the hdfs cluster. And i need to get them into the hdfs cluster. And so you know. Kafka was sort of handmaiden to dupe always getting data into it as a big scalable pipe and part of the what we call the big data ecosystem. Then people noticed things right. People noticed for example. Well there are these kind of standardized legacy interfaces that. I need to write code for all the time. Like there's a relational database. And i don't mean to call relational databases legacy. But you know. There's a legacy system with a relational database. And so i need to write some code. That does a select on the database and finds new records and produces them into a cough topic. Well you're only going to write that two or three times before you realize you should extract that into a framework and then maybe you've done some work in kafka and you've got some topic that has processed data and you wanna put that into files in an s. three bucket or something and so you know you're only going to write that. Api code two or three times per four. You realize there's a framework there. And this was the first major piece of kind of application framework that kafka as a platform added and that was connect so the first piece of evolution that you're talking about was kafka connect and that was this plug -able extendable standalone integration framework because the community saw that people were solving that problem over and over again and everybody always ended up with their own buggy partial implementation. And so now there's a good one and there was like this rich ecosystem of connectors. So that's kinda cool after that. Okay i can get data in and out from standard things and everybody doesn't have to right the same as three code or that same elastic search code. That doesn't differentiate. you doesn't add any value. You just do it one time. You get that connector. After you got that people realize okay well now my consumers those are the application programs that are reading from kafka topics. They end up doing interesting things right like they aggregate messages. They the group messages. They compute aggregations. You might enrich one topic with data in another topic or something like you get a message on a topic and you need to go. Look thing up so this just happens. If sometimes i get the the privilege of talking to enterprise architects at big bank or something like that. It's always a fun conversation and you can say hey folks you know. What are your consumers. Do and the three things that i just mentioned while filtering aggregating and joining. That's like what everybody does and the recognized. Oh wait you know. We're doing this over and over again and it's really hard and there's a bunch of problems we haven't solved at so kafka streams emerged as of kafka zero point ten as the standard framework for doing that kind of stuff in your consuming applications and doing it in a scalable way. That's a pretty sure you've episodes on that. That's its own topic. But those are examples of kafka growing platform components and leaving its origins as scalable pipe behind and starting to say to the world. Hey look. I have an agenda for how you build applications. Let's talk. I think one of the reasons why kafka has become so prominent is that you often have in modern application development a large disturbance system and you want to synchronize state of that distributed system across your different clients across your different services. Can you talk about how kafka fulfills the purpose of synchronizing state across a large application. Yes yes and by the way. I think that is an insightful. Account of what a distributed system is it is sort of an attempt on the part of many programs running on many on synchronized processors to create an evolving state. And try to come up with an account of it so the problem with state is that it changes and if you have lots of copies of that changing state in different places negotiating the changes to those copies is very difficult to come to the conclusion that you know yes as a system we have a consistent view of what is true right now.

kafka kafka kafka Kafka
TikTok deal deadline not extended, but talks expected to continue

Techmeme Ride Home

01:31 min | 7 months ago

TikTok deal deadline not extended, but talks expected to continue

"Tiktok can what what the hell was. Is this all just kabuki theatre like was there any substance to this at all. Do we expect this to never be brought up again. What do you think well recording this. On december fourth which i think is a new deadline for for for the tiktok deal to go through. It's pretty fascinating. Because obviously the trump administration did not really care about tiktok. it was clearly a it was always evident. That was a a thing. They thought was beneficial to them to talk about politically And now they've moved onto to imaginary fraud. And there's no one there to push this through but there are thoughtful people who say yeah. There's really is a problem with tiktok. being essentially a chinese company ben thompson is one of them And so something should be done here. There's a whole wider sort of us. China thing gets worked out so this is just vaping and bam nothing not in the morning I do think that assuming this gets sort of left on the biden administration's desk that it doesn't go away and that they'll be some kind of change to tick tock. I don't know if oracle and walmart being sort of investors have operators is. This is the is the solution in clearly. Tiktok doing just fine Even though it has an issue

Tiktok Trump Administration Ben Thompson Biden China Walmart Oracle
Streaming Data Integration Without The Code at Equalum

Data Engineering Podcast

04:24 min | 7 months ago

Streaming Data Integration Without The Code at Equalum

"Can you give a bit of an overview about what you're building at equal am and how it got started and you mentioned that you've been there for a few years so maybe a bit of how you got involved with the business as well case old building a platform for data ingestion physically. Etl system with the goal of providing open source benefits to the enterprise domain. I'm sure that everybody who's tried to use soles in the enterprise and found on the difficulties in how chips around implementing it. And we're trying to bridge the gap and get the best out of Into enterprise ready application product. So that's foiled goals as for myself. I've been working for about three years. Almost as implem-. I'm started out a halfway william equal actually started with that goal but had quite a few steps around it. We started with a very simple system and ended up with a full stack of spa kafka and all the jason competence and a food system so solid very simple the overall space of data integration and. Etl has become relatively crowded in the market. And they're a number of different approaches where some people are advocating for e. l. t. Where you just do extract and load and using something like maybe five tran or the singer set of tools or some people are focused on batch oriented workflows using more traditional et l. approaches and. I'm wondering if you could give a bit of context as to how equal and fits in that overall market and some of the differentiating factors that engineers should consider when they're debating what tools to use. What approach to take the main differentiator for us is a customer that once a system that is mature and a lot of indications of open source products and open source. Capabilities are still in the making and still growing as you grow with them. And we'll aiming at providing the whole system and to end to someone who wants etl rather than a lot of moving parts thing. That's the main thing we don't want to provide yet another software that relies on five to ten different vendors so for example if you're doing streaming you might implement calf and you might need zookeeper and you might monitor it and i'm sure that everybody who's done that. Seen model confidence that you end up with. And i would say that when you start with a data engineering project usually end with. They'd engineering plus a whole division of devops. You want to end that mess and provide one product with one vendor gives you the whole thing end to end everything and relevant to the use case rather than to the in terms of the overall ecosystem of data you mentioned wanting to be able to the benefits of open source to the enterprise. And i'm wondering for people who already have started down. The journey of building out a data platform they might have some capacity for data integration in place what are the components of the overall ecosystem that equal is designed to replace outright and which are the ones that it is designed to integrate with augment. Let's stop replace serves as an ingestion system so the workplace depends on what you're doing. I can certainly give you. Examples from implementations we have replaced system using open source spent the whole on top of our and a lot of station around building the flow in bento executing the flows in monitoring and getting everything working together. We have replaced the whole thing with just one system and again one vendor for the whole thing not mingling too many companies and getting them to work so i would say for that use case the replacement would be for the end to end system so it depends on the use case itself but we are aiming to replace the whole integration and to and from source to target to get data transformed in reached and managed to the level of. I read it from the source whether it is a string soul veg souls luck as three or even see souls and writing. It's to whatever they might be snowflake for data warehousing data lakes so it is an end to end solution that is aimed at providing the full stack that you require full. Data integration

William Equal
Interview With Yahia Lababidi

Spark My Muse

04:21 min | 8 months ago

Interview With Yahia Lababidi

"Welcome everybody sparked by muse. And today i have a guest. Yahia la vida de. I hope i didn't mess that up too badly beautiful. Who is a writer. An egyptian who's come to america as a young adult eight critically acclaimed books of poetry and prose. he's an authorised an sas and most recently he sent me revolutions of the heart literary cultural and spiritual which is just a treasure trove of little gems. Some smaller pieces some slightly larger pieces and to begin speaking about it. It's hard to know where to choose at this banquet table where to pick but you so much for joining me for the podcast. Thank you for having me over here in new to and also. I want to make sure that we tell listeners about this book being the book for january. Twenty twenty one and meeting up with you in february on the third for a book club. Discussion and fighting. That'll be really fun. What's nice is that it's recorded so anytime someone wants to come back. And listen or it can be embedded on your webpage even or or any web page. Yeah it could be revisited and enjoyed over and over you define aphorisms as what is worth quoting from the souls dialogue with itself and you also say that you hope that might serve as a form of peace offering and bomb in these troubled times and for people who are not quite aware or quite. Have a handle on what aphorisms are. Perhaps you can just explain that a little bit and then speak about what that offers us today. Well it's it's basically it has currency without being recognized for what it is so anything when people have these quotes or inspirational sayings or even what they call it. A witty wise one liners. That's an aphorism if if it doesn't have a name attached to it and it's a maximum or proverb in the assuming some great sage cited then it's an another category of instruction but but enough for them. They're certainly more people who are aware of what they are. And who use them consciously now than when. I began writing them. Let's say thirty years ago at this point as a teenager. When i don't think anyone even knew what that meant but i grew up reading. People like braun. Nietzsche and blake and kafka and pascal who tended to write in offer 'isms and they basically i mean wild has some definitional skar wild about how he had some existing a phrase. I do not presume to some olives in a frozen any of my offers. But it's this. It's this idea of trying to encapsulate a great conversation. And that's why. I define it as a competition with the souls conversation with itself really so you go off. You're thinking about something dreaming meditating possibly weeks years even and then at some point. There's one line that you can extract from all that that can stand alone by itself that will be a key or a door or window or invitation for a complete stranger to have that conversation with themselves so a good aphorism doesn't in my understanding of it at least in everyone's got their own definition is just as suggestion or you to sort of the spark your own conversation With with your with your soul so to speak. And that's why. I really appreciate reading Books of aphorisms where there's few on the page and a lot of blank space because it's understood that they are in need of diluting the way you dilute. It is by bringing in everything you know. Suspect you know we're just breathing alongside it

America Kafka Nietzsche Braun Pascal Blake
Justice Department Hits Google With Antitrust Lawsuit

Reset

08:12 min | 8 months ago

Justice Department Hits Google With Antitrust Lawsuit

"Few weeks ago we were all on pins and needles expecting a lawsuit from the US federal government against Google and on Tuesday after a fourteen month. Long investigation that lawsuit arrived the US Department. Of Justice and eleven states have filed suit against Google arguing that the company has used unfair practices to preserve its search and search advertising monopoly. Here to break all the details down with me, records Peter Kafka Hey Peter hit Teddy. So how a deal is this is a very big deal. This is the most consequential antitrust action against a big Tech Company that we've seen since nineteen ninety, eight I don't know what you are doing in one thousand, nine, hundred, eight I was actually still falling technology news back. Then that's when that's when the US government sued. And we've gone twenty plus years, and now we've got the government's suing Google and we knew that this was coming for a while right and we talked about on the show with Shrink Afari a few weeks ago we didn't have specifics we didn't exactly know. What the lawsuit was going to allege that Google's done you've spent the morning digging in talking to a bunch of parties reading the suit tell me what the main argument, the US federal government is making. So I'm going to caveat this. This is the beginning of a process that's going to take years, and so the initial complaint from from the US government. Is the initial complaint it is going to change over time through discovery. Okay. people are going to add to this. It's going to be a living document as we say, but the big picture here is is this to start with the US government saying Google has an illegal monopoly and search that it maintains it focuses mostly on the idea of Google paying. Mobile, phone manufacturers like apple like Samsung fees, either directly or indirectly to give it pole position for its for search engine which then makes it search engine dominant, and then it gives it a an unfair advantage in search ads. So it starting with search and adding on search ads in it is mostly focused on the idea of Google spending billions of dollars a year to give itself, prime? Placement. Your phone and yeah and to the average person of sitting here thinking what does this case kind of boil down to it's basically that Not just that Google is too big right but that Google has unfairly made itself. So essential in all of our lives. That's the overall thrust of all the antitrust and all the tech reform conversations we're having, which is all of these tech companies, Google and facebook, and Amazon, and apple are have gotten so big and they're not really restrained by government in any way, and this has been going on for years and decades and someone should should do something about it, and this is really the first. Concrete who've we've seen from the government in sort of the modern Internet era trying to rein in these big tech companies. All right. So I'm guessing Google to not throw them a big hug. What's their response to this Google response is, Hey, we have an awesome search engine. What's your problem with this by the way it's free so we're not harming any consumers with their awesome Search engine by the way when we spend billions of dollars and give that money to apple and Samsung or do people who use our android software for free, they get to deliver awesome phones to you for less money than they would have normally because we're helping to subsidize that what's your problem by the way? The idea of giving people money to get better placement is not. A new idea it happens all the time. When you go to your grocery store, your target or Walmart people have paid to have their products on the shelf or on an end cap they pay different prices depending on where they get placed. This is not a new idea, and by the way we've done all this in the open for years. Yeah. You get the sense that Google. Sees this as an existential threat at all I mean. I. Wonder to what extent This is a financial risk for Google. Stock Market said no big deal Google share prices trading higher than it was Google has faced these kinds of accusations and lawsuits in Europe for years there is definitely a possibility the the US wins a lawsuit and there are structural a reform is made in. Google. was forced to do something drastic like split it's android business from its search business. We're not sure as possible. This could be a way to break it up. Yeah. It's it is possible. There is a wedge here that you could use to break it up. There's also the possibility that. Spends years fighting the DOJ and it gets it takes its eye on the ball and or it doesn't make moves that it would want to make because it's in a lawsuit and it has one upset the government anymore, and this is what people at Microsoft said happened at Microsoft when they were fighting the US government twenty years ago. Microsoft ended up winning that because sort of there was going to break up settlement for for Microsoft and ended up not happening in Microsoft kind of got away with punishment except that Microsoft ended up missing out entirely on the mobile business and there are Microsoft people who say we did that because we we took our eye off the ball and There's an argument that just bringing the lawsuit and just occupying Google. Is Sort of a punishment in and of itself Peter you mentioned your story which talked about the fact that there's a lot of people on both the left and the right who've been calling to break up big tech. He's going after Google. Specifically, you made a great point which is that you had this sort of unlikely alliance between the trump justice department, which is bringing this lawsuit and folks on the Left like Elizabeth Warren who are. Glad. That Google is getting some skirting for the first time in a while. They're very glad and to be clear you know it's not just that it's a left right thing. I mean, the Lisbeth Warren Detests Bill. Burr she calls him a corrupt trump crony who should resign. And then literally, in the next sentence in a statement, they gave to me said and he should go ahead and pursue this lawsuit. And you know it's as basic as as anime enemy is my friend. The slightly more sophisticated argument is we can't wait to work with the DOJ that we love. We need to work with the DOJ we have. We've been waiting to restrain google for a decade or more, and we can't wait on this. It's gotta go. I want to ask about the timing of this. there. have been reports that this was sort of sped out right that there's an election coming up once every possible accomplishment that he can talk about to happen before Election Day. Do you see any evidence that this was rushed at all or what do you make of kind of reporting that? This is politically motivated to at least begin to unfold before Election Day there was a report from the New York Times in particular said. There's dissension within the DOJ career staff wanted to go slower and that bar was moving to make this go faster and he wanted to happen this fall. The intimation is this would be sort of a political action. It's possible. We certainly have seen Donald Trump rail against big tech companies especially in the last couple of weeks, he has to set anything about this suit today and and to be fair this is a little a little abstract I think for people to sorta understand and you know. I'm not a political expert, but it doesn't seem like this is going to move a vote. That said, you know it's Donald Trump era everything's very weird and and it's To ask questions about something like like the timing on this right? I mean. The timing is step back. Though as you mentioned earlier, this is a years long game. You know he trump I get to say he's begun the lawsuit, but obviously, this is not happening. Yeah. A more practical way of thinking about or another another way of thinking about it that folks have suggested to me is this is likely the biggest case in bill bars career and that if he doesn't file it now or at least before. He may not get a chance to file it because Donald Trump may not be president come January. So the rushes for Bill Bar to get this thing through and have his stamp on it. This a lawsuit that is gonNA live for a long time. Right it could live in Abidin Justice Department right and that he wanted to be the one to bring yes it may sir it is very likely to continue a Biden Justice Department and that bill bars looking at this sort of with a historical frame like this will be my contribution to antitrust.

Google. United States DOJ Government Donald Trump Microsoft Us Department Peter Kafka Apple Samsung Walmart Bill Bar Abidin Justice Department Biden Justice Department
'So Hard To Prove You Exist': Flawed Fraud Protections Deny Unemployment To Millions

NPR's Business Story of the Day

02:06 min | 9 months ago

'So Hard To Prove You Exist': Flawed Fraud Protections Deny Unemployment To Millions

"During the pandemic state unemployment systems have become a target for organized crime rings, they steal money through fraudulent claims but arguably a bigger problem is that some of the systems in place to prevent fraud like that have been hurting millions of innocent people. NPR's Chris Arnold reports when Sevi- guas lost his job as a food and beverage manager. Marriott Hotel near San. Jose he figured locale apply for unemployment. This was back in March he went online put in his info waited for weeks couldn't get through on the phone after more than a month he was told to mail and more proof of his identity mind driver's license picture of my past poor copy of my w. two she said the more documentation that I could put. In there to prove who I was would help out my case out his case Gouache had clearly lost his job with a big company had ide- what was the problem but this dragged on and on weeks would go by they need another documents and six months later, gouache still hadn't gotten any unemployment money manny can't find another job I had about seventeen. Thousand dollars saved gouaches thirty two years old, and he'd been saving up to go back to community college to try to become a computer programmer. He moved into a smaller apartment to save money but he still had to drain that entire savings for college. There's not enough left to pay rent next month to watch what I worked really hard to get dwindle away. I don't WanNa get angry in front of you for the interview. But it has been really really frustrating and the whole thing to seem so Kafka ask avoidable to him. It's so hard to just prove that you exist in California alone millions of people are having a hard time proving they exist as they struggle to get the unemployment benefits that they deserve and it turns out washes right? A lot of this was completely unnecessary.

Manny Jose Gouache Marriott Hotel Chris Arnold Fraud NPR Kafka Sevi- Guas California SAN
TikTok and WeChat: US to ban app downloads in 48 hours

Techmeme Ride Home

03:18 min | 9 months ago

TikTok and WeChat: US to ban app downloads in 48 hours

"This morning the Commerce Department announced that it will ban US downloads of and business transactions with Tik Tok and we chat on Sunday. So are these stories don now probably not even close quoting CNBC, the announcement comes ahead of an expected statement Friday by President Donald Trump on whether or not the government will approve a deal for Oracle to take minority stake and TIKTOK and become a trusted technology partner for the company in the US. It's unclear if the Commerce Department's announcement means there's no possibility of a deal going through. Before this Sunday deadline, it could be an aggressive move from the trump administration to push for its original intention to force Tiktok to become fully owned by a US company. The Commerce statement said that starting Sunday US companies will be banned from distributing we chat tiktok meaning the two major mobile APP stores run by apple and Google will have to remove the APPS from their libraries. The statement also blocks US companies from providing services through we chat quote for the purpose of transferring funds or processing payments within the US and quote. But. The announcement also lays out a separate timeframe specific to tiktok giving it until November twelfth to resolve the US national security concerns the rules that start November twelfth include provisions that block US companies from providing Internet hosting and services. For TIKTOK, this could be directed at the deal being negotiated between TIKTOK and Oracle which would provide cloud services for TIC TAC if trump approves and could give Tiktok and Oracle more time to hammer out a deal that will. Satisfy the president in an interview with Fox business on Friday Commerce Secretary Wilbur Ross said, the bands will affect Tiktok and we chat differently at first. He said Tiktok will still function, but users will not be able to upgrade the APP. It's still unclear what kind of functionality we chat will have in the US after Sunday but it's unclear whether or not TIKTOK will still be allowed in mobile APP stores but not allowed to provide updates to users and quote. Not The date of that extended tiktok deadline November twelfth certainly that gives all sides more time to negotiate a deal. But as Peter Kafka pointed out on twitter, we chat enormously popular with Chinese Americans owned by a Chinese company will be crippled by the US on Sunday night TIKTOK enormously popular with Americans including some trump voters owned by a Chinese company trying to do a deal with trump's supporters it will be okay through election day and quote. More headlines and rumors have been bouncing around over the last twelve to twenty-four hours bite dances apparently planning on a US IPO for whatever new business is carved out if it's allowed to be carved out. An agreement has been hammered out between dance and Oracle that includes the creation of an oversight board approved by the US government and a continuous third party audit, and finally most juicy sources are telling the New York Times that instagram founder Kevin System, has had preliminary talks about becoming tick tock new CEO if tech talk is allowed to continue as Josh. Bernstein. tweeted INSTAGRAM's Kevin System. Becoming Tick CEO and crushing reels would be the ultimate revenge for Zuckerberg stripping his autonomy. So Spicy and quote.

United States Tiktok Donald Trump Oracle Commerce Department Us Government President Trump Tik Tok Cnbc Wilbur Ross Partner CEO Kevin System Instagram Apple New York Times Twitter Google
WeChat Officially Banned On Sunday. TikTok Only Kinda Banned.

Techmeme Ride Home

03:18 min | 9 months ago

WeChat Officially Banned On Sunday. TikTok Only Kinda Banned.

"This morning the Commerce Department announced that it will ban US downloads of and business transactions with Tik Tok and we chat on Sunday. So are these stories don now probably not even close quoting CNBC, the announcement comes ahead of an expected statement Friday by President Donald Trump on whether or not the government will approve a deal for Oracle to take minority stake and TIKTOK and become a trusted technology partner for the company in the US. It's unclear if the Commerce Department's announcement means there's no possibility of a deal going through. Before this Sunday deadline, it could be an aggressive move from the trump administration to push for its original intention to force Tiktok to become fully owned by a US company. The Commerce statement said that starting Sunday US companies will be banned from distributing we chat tiktok meaning the two major mobile APP stores run by apple and Google will have to remove the APPS from their libraries. The statement also blocks US companies from providing services through we chat quote for the purpose of transferring funds or processing payments within the US and quote. But. The announcement also lays out a separate timeframe specific to tiktok giving it until November twelfth to resolve the US national security concerns the rules that start November twelfth include provisions that block US companies from providing Internet hosting and services. For TIKTOK, this could be directed at the deal being negotiated between TIKTOK and Oracle which would provide cloud services for TIC TAC if trump approves and could give Tiktok and Oracle more time to hammer out a deal that will. Satisfy the president in an interview with Fox business on Friday Commerce Secretary Wilbur Ross said, the bands will affect Tiktok and we chat differently at first. He said Tiktok will still function, but users will not be able to upgrade the APP. It's still unclear what kind of functionality we chat will have in the US after Sunday but it's unclear whether or not TIKTOK will still be allowed in mobile APP stores but not allowed to provide updates to users and quote. Not The date of that extended tiktok deadline November twelfth certainly that gives all sides more time to negotiate a deal. But as Peter Kafka pointed out on twitter, we chat enormously popular with Chinese Americans owned by a Chinese company will be crippled by the US on Sunday night TIKTOK enormously popular with Americans including some trump voters owned by a Chinese company trying to do a deal with trump's supporters it will be okay through election day and quote. More headlines and rumors have been bouncing around over the last twelve to twenty-four hours bite dances apparently planning on a US IPO for whatever new business is carved out if it's allowed to be carved out. An agreement has been hammered out between dance and Oracle that includes the creation of an oversight board approved by the US government and a continuous third party audit, and finally most juicy sources are telling the New York Times that instagram founder Kevin System, has had preliminary talks about becoming tick tock new CEO if tech talk is allowed to continue as Josh. Bernstein. tweeted INSTAGRAM's Kevin System. Becoming Tick CEO and crushing reels would be the ultimate revenge for Zuckerberg stripping his autonomy. So Spicy and quote.

United States Tiktok Donald Trump Oracle Commerce Department Us Government President Trump Tik Tok Cnbc Wilbur Ross Partner CEO Kevin System Instagram Apple New York Times Twitter Google
Who really killed Blockbuster Video?

Land of the Giants

05:37 min | 1 year ago

Who really killed Blockbuster Video?

"I'm Peter Kafka and I'm Ronnie. Mola and this is landed the giants. The net flicks effect a podcast on Netflix's disrupted. Hollywood change the way we watch TV and movies. And how it should have been squashed by giant competitor, but ended up turning the tables and killing the video store. Okay Peter. Let's go back in time long. Before we had netflixing chill, we had blockbuster nights. Remember this tonight. Make a blockbuster. In the nineties and early outs, blockbuster was the world's largest video rental chain. It was a huge part of American culture at its peak, it was bringing in six billion dollars a year in revenue and had more than nine thousand stores around the globe. It was the place to rent movies it was. Social place like teens would meet their. You know people you know because that was one of the few things that you could always do as a high school student. Was You know you could get together with your friends and rent a movie and go to the cool, parents, house, and watch it where there's a copy of the wiz that I would like hide in a corner at blockbuster because I always wanted to be there when I come back for, and we would go in there her car then go into blockbuster and pick out a movie, and then we get a little treat you know I don't know. We pick two to three movies scary WanNa funny one and then in action. Nobody has the movie I want I. Even Video Blockbuster probably hasn't I mean? We have over ten thousand videos? Five six o'clock on a Friday night. Phones are ringing off the hook. It's never what do you have? That's good. It's always what you have. That's new that last voices Jason Bailey nowadays. He writes about movies for places like vulture in the New York Times, but years ago, he used to work at a bunch of video stores, including blockbuster, which he says wasn't as great as we might remember. I have much more nostalgic for the video store. Then I do for blockbuster in particular which really in a lot of ways. Ways killed the video store, flattening it into the sort of McDonald's version of the video store, right? That's what I remember about. Blockbuster killed my local video store replaced with blockbuster, which I did not like blockbusters were everywhere, and everyone rented from there, but that didn't mean it was a great customer experience, one of the big things that you always hear people who who don't remember blockbuster through a Golden Glove nostalgia talk about where the late fees or as they tried to rebrand them additional day fees or additional rental fees. They were outrageous. Boxer made a ton of money on late visas or additional day fees running. At one point late fees made up seventy percent of blockbuster prophet, and along with this highly fees, there are a long list of other problems limited new releases, long lines, shitty customer service all which is lousy for customers, which also made it lousy for employees. I got cursed out a fair amount. Again for you know just doing what I was told to do by corporate, but yes I would, I would be told that you know that. We were monsters that we were bloodsuckers. I had that thing back on time I saw you. You take it out of the box I was like I got called out like that I had people tell me I saw you in here when I dropped it off. You tell me you didn't check it in on time. I'm not paying that if it got heated, but this is all factored into blockbusters business model. The company even had a term for it managed. Dissatisfaction managed to satisfaction is a term that John Antioch. Oh the CEO Blockbuster explained to me, and that is is long as you give a consumer and. And of what they want. They will ignore the fact that they're not always getting what they want. This is unique heating. She's a journalist who covered enough for Reuters she also wrote a book and made a documentary about Netflix's history blockbuster understood that only twenty percent of customers who came in would get the movie that they wanted, and they would have to get something else the other eighty percent of the time. They weren't happy, but they weren't horribly angry. Managed dissatisfaction is one of the great corporate euphemisms for screw you give. Give us your money. You are ever going to hear and if you can't remember what it was like to go to. Blockbuster occurred analogy. Be Like an airplane. You WanNa. Airplanes are like it sucks every way, and if you want to improve it in any way, you have to pay additional fees for everything that is blockbuster in a nutshell in the nineties, yeah, customers felt trapped, and with all the smaller mom and pop stores being squeezed out. They didn't really have a better alternative, and then all of a sudden they did. Again you've gotta remember. That netflix emerged in the late nineties when the Internet still felt pretty new. Amazon was just becoming release accessible selling books online. They're doing it cheaper than the competition and they didn't have to operate stores, so Reed Hastings and Marc Randolph to tech guys in the bay area are surveying the landscape and they thought hey, we could be the Amazon of something else. There's a better way to rent movies as many as you want. Go to Netflix DOT COM. COM Bake a list of the movies you want to see and about one business day you'll get three. DVD's kept him as long as you want. Without late fees DVD's had just come out. And suddenly there was this new way to watch movies that didn't involve these bulky VHS tapes. DVD's were smaller, more durable, and you could ship them for price of a postage stamp. So? Hastings Randolph thought. Hey, we could be them on of movies. This may seem obvious now, but at the time this was a big deal. Suddenly instead of having to go to the store and deal the crummy customer experience, he could stay home. You can hit a button and someone somewhere sent you the dvd you wanted instead of the one you had to settle. for which Netflix customers loved by the way they tended to order? The kind of movies at blockbuster didn't feature or even carry it all they had all the indie movies and older movies and blockbuster was focused on what was new.

Blockbuster Ceo Blockbuster Netflix Peter Kafka Giants Hollywood New York Times Mola Jason Bailey Hastings Randolph Amazon Reuters Mcdonald Reed Hastings Boxer John Antioch Golden Glove Marc Randolph
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

20:01 min | 1 year ago

"kafka" Discussed on Software Engineering Daily

"Many of these tools required a high throughput system for publishing data and subscribing to topics Kafka was born out of this need all the data.

Kafka
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

08:17 min | 1 year ago

"kafka" Discussed on Software Engineering Daily

"Because it's event driven then generates alerts from what's and just like push notification saying in the last five minutes there have been ten attempted. It's access to the server again. This is just written using Kafka connect and in this case using case equal so I didn't have to write any code to do that. Kafka connect sits there and it listens for sizzler connections it rice in Africa Topic Casey takes that Kafka topic processes it it filters for certain conditions such as the string you get an slog doc. SSh attempts and then performing the aggregation supper fats and then that's rice back to a Kafka topic which connects then hooks up and pushes outs yes they notifications on is Kafka more broadly useful as a buffer or a middleware system for log data is it's used in so many different ways because it is so flexible and adaptable so I mentioned earlier on the idea of database offloads. That's what I was huge cases but it's also widely used for this idea of aggregation. Look Centralization. I'm processing because you can do that pricing on the events as they arrive. You've also written some about how Kafka can fit into a data pipeline that also includes data warehousing tool so google big query. Maybe you've got snowflake snowflake at the end of this. Can you describe some patterns around involving Cough Khou with your data warehouse. Show so wherever your data's eight is coming from whether it's coming from a straightforward relational system the on premises whether it's from a cloud based platform whether it's from your own applications writing into Africa directly directly you have your transaction events in Kafka you can then enrich them and transformed them as you want to and then Kafka connects can push data to to the appropriate analytics platform so it can push it too big query can push it to snowflake this connectors have both of those that are part of the Catholic connect framework so you can have real time data aggregated and and rich if you want to use his stream processing but then London area time in these target data into our houses I have read several of your articles where you frequently use oracle databases in your examples. I don't know if that has to do with your background in in particular databases but do you encounter a lot of cough users who have oracle installations yeah so it's with Oracle doing consultancy with Oracle analytics product so a natural affinity so it's and it's interesting. I think how the apparent Marcus is changing ranging or usage with kaffirs changing originally a few years back when I started looking at it it was very much the cutting edge stuff so it was people who are really really heavily interested in the cutting edge dispute systems and the traditional enterprise things like Oracle and got less of a mention now you see more and more more conferences that meet ups on community platforms people asking about integrating with Oracle with territory with sequel server because that eleven eleven of adoption with is there that is actually making huge inroads into the enterprise. It's not just a cool kids Tagliani Mar it's a serious platform. Men's prices are adopting adopting onions prices tend to have prostates vases. I think there are people who would like to move away from their oracle installation because Oracle can get quite expensive and the database market has gotten really competitive so there are a lot of good alternatives to Oracle. Do People Use Kafka to migrate away from Oracle Sutton Agree Steph shift to that's if it's a straightforward migration. This question does come up quite a Lotta conferences as if it's simply a one time big bound thank Africa's not necessarily appropriate way to do it because there's plenty migration tools out there on the markets where after fits very well is where you want to retain an existing system without impacting out but take changes from system and start pushing them to the new one so you can run side side-by-side so people use approach Murphy from one database to another they'll see us moving from on premise to cloud platform auto earning separates POW platforms because you've got your events from your sauce whether it's database whether it's an application right into our database into Kafka those events can then be pushed her whatever your target WanNa want to send you can run side by side without impacting your original application and then once you got to go you can switch off that sauce so Catholic massive benefits that you written some about about the blurring lines between analytical and transactional systems so this is like where historically a you might have been running a nightly job to aggregate. Maybe a recommendation system. Maybe you've got your transactional data system them and in order to build recommendations based on that transactional data system you do an et l. job into a data warehouse and then use the data warehouse to generate those recommendations but these lines are blurring because we want to do more and more up-to-date analytical whole workloads. Can you explain how the patterns that you're seeing around Kafka fitting into this improved latency of the analytical Nicole workloads. I think historically we've always separates technologies because you had to a mentioned sometimes technology fast is how we design systems and in the pass you had to decide. Am I building a transactional system. Don't want to get the data in quick but it was not read are my building on medical system. It will be quick to get the data arounds but it might be slow to go to an and you had to choose and so organizations and teams grew up around this way of thinking and developers and engineers had this way of thinking it's right well. I'm an application developer I- knees application technologies nothing to do with analytics I can ignore those data warehouses and stuff Kafka brings the data brings the events front and center and now applications can access that data enero time analytics can also be driven by that same data so I think the move away from doing things in separate worlds is actually happening. I think he's happening slowly because you still have this inbuilt way of the the people have a thinking about what I'm building this application. It must be an application type architecture. I don't need to think about analytics. I'm not going to integrate with let's ex- except for one time bachelor once tonight out of it now. It's much more tightly integrated. You've been at conflict for a while now. What's something new that you've learned about Kafka in the last year also question I think that my answers and that would be around case equal which is a product? I spent lots of time working on their last year so that's built on top of Kafka streams so understanding a lot more around around how halfway houses concept of humbling streams of data instead shade the states on top of that and how you can actually do that through case equal is probably my main learning from the last year okay last question. Do you have any reflections on what it's like to be at a rapidly growing infrastructure for structure company because conflict is growing really quickly yeah. It's a fantastic place to be if you want to just kind of turn up and do something and then go home and not care about it then and I guess you could do that but you're not gonNA get the most out of it. There's some super super smart people some Super Nice people as well. It's just very welcoming friendly place to be a where. There's a lot of very exciting work going on. I feel privileged to have a job where the technology is awesome can talk about it enthusiastically because I genuinely think it's really me good so does that one company working on it's very smart. It's just exciting place to be okay. Well Robin. Thank you for coming on the show. It's been really fun talking to you. Thank you so much. Software can improve our lives but the business motivations of software sometimes conflict with user desires and may hurt US instead of helping.

Kafka Oracle Africa Oracle analytics cough google US Casey Sutton Robin Marcus developer London Murphy Nicole five minutes
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

14:29 min | 1 year ago

"kafka" Discussed on Software Engineering Daily

"Once you have your cough data pipeline nine setup. Does it constantly update and then you can reliably have those sinks the read side of this data pipeline line. Is this data going to be constantly updated. That's right so as comes into Kafka it will get streamed out. If you not doing any processing on it okay stream. I'm straight out to the targets using like elastic search for example. You're going to see that's almost instantaneously you can also have your aggregates populated that way onslaught as your target system allows item potent updates then you're aggregate swell updates in place so if you're building an aggregate of what am I current sales within the last hour Woah for example than the aggregate K. K. You'll get over it on each time and we'll just increase per hour any late-arriving data would also get updated against the appropriate McKay as well okay so you've given another example here of a search system so you could see potentially taking your transactional data data store and if you wanted to have search indexes created against that transactional data store and you want to have the search index is constantly updated you you could have the data being buffered from your transactional data store into Kafka and then you could have your search system reading from Kafka and updating dating its search indexes. Yes absolutely again. That's one of the common use cases for Kafka because since Kafka persists this data for as long as you want to the two with about forever with that's for ten minutes or based on the certain amount of size you persist that day and you can use it multiple times so for example populate your search index if you're searching next goes bang or if you want to scale us out and additional instances you replayed update from Kafka down to the target but but you're also updating it constantly from the system so it gives you the ability to populate a maintain those caches but that same day to the populate your search indexes as you can also be using that same data to populate a graph database or unless six database however you use that data you can use it multiple times from that same topic and I want to ask a question here. Why do we actually need Kafka if we're just trying to get our data from a transactional data system into into a data warehouse or trying to get data from a transactional system into a search index why not just throw that data straight into the search index or straight into the data warehouse warehouse? Why do we need this middleware of Kafka? That's a good question and it's one that's does come up because on the surface of it seems like that's the thing to do. We're introducing apparently currently additional complexity by using after in the first place the answer is that it's never just one pipeline. It's never just the set of data is used in this just one place over there. It's always all this data. Hey in east populates search index and we want to use it for analytics and we want to share it with this other departments and we want to use it Dr Association so you have two options us something Kafka which lets you take a real time feed of those events from the source system and use it multiple times concurrent concurrently against different target applications and consumers or you start to rebuild yourself a spaghetti architecture by taking the data from the source system to one place copying it to another and making it available to another unhooking all these different dependencies together which then makes it very difficult to change any of those pieces to scale them out and data chick it to replace one of those or move them all of these dusty dependencies so Kafka. LSU decouple those tendencies it also gives you many additional benefits. It's like providing buffering between sauce and targets so it does make a lot of sense when checklist examining the alternatives to using it one other thing that you can do who is data enrichment so if I- buffer my change data capture log into Kafka I can have data enrichment processes running over that data that's incorrect explain the role of data enrichment show so it lets you take events as they come in which can be from any source system or application so it could be stuff happening in database it could be applications writing directly to Africa and those events you can then start to process you can start to cleanse the data as it comes in and filter out but records you can start to reshape modify the schemers as you can start to join other data that could be data from a different database from a different system from a flat file from another application but because it's all in Kafka it can then be worked with together regardless of where it came from she can start to join an ritual your data you can start concatenation create deprivations off that data undoing undoing all within Kafka. The processing of data within Kafka is often done through a system called Kafka streams dreams so stream processing is the act of doing changes to streams of data that are coming into your system mm-hmm and there's a large number of ways that you can perform stream processing you could just set up a simple python script to just re data eight off of Kafka and make some changes to that data and read back to Kafka you could use a distributed streaming framework like flink you could use spark streaming streaming or you could use Kafka streams. Why should people use Kafka streams or could you present some of the different alternatives that people can use to due stream processing against data in Kafka sure so there's various different ways to do it as you say on your list at some of the ones there to an extent? There's not one right light one. There's going to be technical reasons why he may choose one over the other. There's always going to be pragmatic. Reasons like skills and experience but Kafka streams is part of Apache Kafka uh-huh. It's a Java library so you just bring it into your existing applications and by being part of Apache Kafka benefits for much tighter integration when it comes to things things I security things like transactional processing and exactly once in on sex so those are some of the reasons why often people start out on a greenfield project just just using extremes because it makes sense if they already have things like fling Kim place or they already must be brought into spark streaming they may okay we'll choose to use those unless they hit up against some of the technical limitations which think well actually we'll reevaluate this and we're going to use caffeine streams one or the other common ways that people do stream processing using Africa is using case equal and this doesn't require any coding as such at all using a sequel type language to interact interact with the data and declared stream processing applications so that means no Java gnome setting up kind of spot classes and stuff like that's such another good rate. The people do often take win. We're taking the change data capture log from our transactional database and we're getting into Kafka and we're GONNA use that change data capture log to update sink databases. Are we getting the entire database reading the entire change log into cough. That sounds like putting a lot of data into Kafka. It depends entirely on how you configure your CD too so if you're using something night to be easy I'm just very good popular open source Kafka Connector Sports things like postcards my sequel and so on you simply say I'm interested in these particular tables so so you're putting the whole database if appropriate but often times obey well. I want to sales table. I want product table. I want the customer table on you. Just get the events related to those particular tables bills but it comes down to how you configure the particular integration. Let's take the search example so like let's say I want to have my transactional database. Get a search index built over that transactional database if I'm doing that am I starting by seeding the searching necks with with a copy of my database or is the entire process based around that change data capture log. Do I have the entire historical change data you to capture luck. I guess I'm wondering you WanNa have this search index that you're going to be updating over time to be a reflection to be a searchable reflection of the transactional database that you have but when you're bootstrapping that search index do you just take the actual database or do you just take the historical historical change data capture lock so the way that many of the tools Jack is they'll take snapshots of the current database states they recalled the end the points in the transaction log at which that was taken to the system and then from thereon end capture any events out of the transaction log so it's kind of current state to throw a select whatever plus transaction log from that point said using the tonight show. You don't miss anything in between okay. Let's talk a little bit about this interface between a database ace and Kafka and that's Kafka connect. That is the interface point of how you're getting your data from one of these sources into Kafka Africa or from Kafka Into One of these sinks explain. What Kafka connect is Kafka connects in a nutshell? Lets you do streaming integration between source systems and Kafka Kafka and your target systems. It's just configuration file base so you'd have to write any code and it's part of Apache Kafka so if you're wanting to get data from Kafka down. HDFS Kafka tastes three from database into cars go from catheter search all of the possible permutations of pipelines you can think of of almost always you want to be doing that through Kafka connect so connects souls a loss of the problems of integration between systems it it does things like fault. Tolerance distribution system is built on top of carcass semantics self so you can scale it out it also handles the more tricky things like schemers and offsets and all of the kind of things if you decide to write it yourself us. Many people unfortunately start off by doing and then I'll write myself a spark job to get the state from here to here are writer Java Program to get data from this database into here. All of those tricky now things Kafka connects has solved already so it's it's a solved problem. Rally also has additional capabilities like transforming the data as it passes through so whilst we've talked about stream processing frameworks works for the kind of the modern front stuff Kafka connects also has the ability translations on the data as it passes through these called single message transforms and you can do some pretty cool stuff you can start to mask the data or drop fields or and rich them change data types all through Africa connect all through configuration files so actually makes it much more accessible to many more people other than just those who are going to write a Java programmer writes a spark code. Oh that's useful so you can begin in the transform process at the entry point of data from the source into the coff- connect system absolutely so for example standpoint if you're putting in data from a table with hundreds of columns which is not uncommon and you say well. Actually I only need a subset of those you could just drop out all of the additional ones or or if you've got data coming in and it's heavily identifiable information it's got credit cards. It's got addresses and you don't need data and actually holding that day so within Kafka and has additional implications vacations you can say well. We'll just call themselves on at the same time this column over here. Let's Kostas and change the dates type this one over here. Let's add in some meditators lineage information about where the information's come from and again ultras through configuration files connect runs as a distributed process so there's multiple nodes for a Kafka connect process is it always true could could explain the parallelism model of Kafka Connect Show so connects connects and this is an important point to make actually does not run on your brokers. Nothing runs on your Kafka brokers except possibly zookeeper and then there's plenty of people would disagree with that not as well but Kafka connect run separate from your brokers. You can run a single instance of Kafka connects if you want to but as soon as he wants fault tolerance as soon as you once additional capacity you then deploy additional Kafka connect workers and apply them as part of the same group Kafka connect will then distribute the workload across those instances this if you have a single connector like a sink process that taking days off topic out to a target you can have multiple Waco acas which will then form a consumer group and read that days from powerful if you're again data from database for example Kafka connect compare lies in I'm just I'm Rachel multiple tables at the same time so the powerless misdefined within connects unless you eat scale not win. Would you need that parallelism awesome. Is that only if you have a database that's changing really rapidly so it depends on what kind of threw you want to get on your data. If you call it happy slept Kafka connects sit unstuck through the data then I guess you don't need us but it gives you the ability to do so if and when you decide you want to get the data if the data's being created a greater rate eight that has been ingested on a single workload so now that we've talked through some of the finer points of this streaming. ETL process could you zoom out again and describe the streaming ETL process for let's say let's got a transactional Mongo database. That's my source of truth database. I've scaled up over time and now I want to build a search index on top of that database. I WANNA use Kafka as the middleware described at the end to end process of getting that data for Mongo into a searchable index shaw. There's two different answers to that's. I'll give you the simple one first and then I'll come bucks the second one the simple one is you deploy Kafka connects you use the BPM connector which is plugging Kafka connect so I should have mentioned Kafka connect a plug in based architecture you plug in the appropriate connective technology so we plug into be easier..

Kafka Kafka Kafka HDFS Kafka Africa LSU McKay Kostas Kim place Dr Association caffeine Jack coff writer
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

14:29 min | 1 year ago

"kafka" Discussed on Software Engineering Daily

"Once you have your cough data pipeline nine setup. Does it constantly update and then you can reliably have those sinks the read side of this data pipeline line. Is this data going to be constantly updated. That's right so as comes into Kafka it will get streamed out. If you not doing any processing on it okay stream. I'm straight out to the targets using like elastic search for example. You're going to see that's almost instantaneously you can also have your aggregates populated that way onslaught as your target system allows item potent updates then you're aggregate swell updates in place so if you're building an aggregate of what am I current sales within the last hour Woah for example than the aggregate K. K. You'll get over it on each time and we'll just increase per hour any late-arriving data would also get updated against the appropriate McKay as well okay so you've given another example here of a search system so you could see potentially taking your transactional data data store and if you wanted to have search indexes created against that transactional data store and you want to have the search index is constantly updated you you could have the data being buffered from your transactional data store into Kafka and then you could have your search system reading from Kafka and updating dating its search indexes. Yes absolutely again. That's one of the common use cases for Kafka because since Kafka persists this data for as long as you want to the two with about forever with that's for ten minutes or based on the certain amount of size you persist that day and you can use it multiple times so for example populate your search index if you're searching next goes bang or if you want to scale us out and additional instances you replayed update from Kafka down to the target but but you're also updating it constantly from the system so it gives you the ability to populate a maintain those caches but that same day to the populate your search indexes as you can also be using that same data to populate a graph database or unless six database however you use that data you can use it multiple times from that same topic and I want to ask a question here. Why do we actually need Kafka if we're just trying to get our data from a transactional data system into into a data warehouse or trying to get data from a transactional system into a search index why not just throw that data straight into the search index or straight into the data warehouse warehouse? Why do we need this middleware of Kafka? That's a good question and it's one that's does come up because on the surface of it seems like that's the thing to do. We're introducing apparently currently additional complexity by using after in the first place the answer is that it's never just one pipeline. It's never just the set of data is used in this just one place over there. It's always all this data. Hey in east populates search index and we want to use it for analytics and we want to share it with this other departments and we want to use it Dr Association so you have two options us something Kafka which lets you take a real time feed of those events from the source system and use it multiple times concurrent concurrently against different target applications and consumers or you start to rebuild yourself a spaghetti architecture by taking the data from the source system to one place copying it to another and making it available to another unhooking all these different dependencies together which then makes it very difficult to change any of those pieces to scale them out and data chick it to replace one of those or move them all of these dusty dependencies so Kafka. LSU decouple those tendencies it also gives you many additional benefits. It's like providing buffering between sauce and targets so it does make a lot of sense when checklist examining the alternatives to using it one other thing that you can do who is data enrichment so if I- buffer my change data capture log into Kafka I can have data enrichment processes running over that data that's incorrect explain the role of data enrichment show so it lets you take events as they come in which can be from any source system or application so it could be stuff happening in database it could be applications writing directly to Africa and those events you can then start to process you can start to cleanse the data as it comes in and filter out but records you can start to reshape modify the schemers as you can start to join other data that could be data from a different database from a different system from a flat file from another application but because it's all in Kafka it can then be worked with together regardless of where it came from she can start to join an ritual your data you can start concatenation create deprivations off that data undoing undoing all within Kafka. The processing of data within Kafka is often done through a system called Kafka streams dreams so stream processing is the act of doing changes to streams of data that are coming into your system mm-hmm and there's a large number of ways that you can perform stream processing you could just set up a simple python script to just re data eight off of Kafka and make some changes to that data and read back to Kafka you could use a distributed streaming framework like flink you could use spark streaming streaming or you could use Kafka streams. Why should people use Kafka streams or could you present some of the different alternatives that people can use to due stream processing against data in Kafka sure so there's various different ways to do it as you say on your list at some of the ones there to an extent? There's not one right light one. There's going to be technical reasons why he may choose one over the other. There's always going to be pragmatic. Reasons like skills and experience but Kafka streams is part of Apache Kafka uh-huh. It's a Java library so you just bring it into your existing applications and by being part of Apache Kafka benefits for much tighter integration when it comes to things things I security things like transactional processing and exactly once in on sex so those are some of the reasons why often people start out on a greenfield project just just using extremes because it makes sense if they already have things like fling Kim place or they already must be brought into spark streaming they may okay we'll choose to use those unless they hit up against some of the technical limitations which think well actually we'll reevaluate this and we're going to use caffeine streams one or the other common ways that people do stream processing using Africa is using case equal and this doesn't require any coding as such at all using a sequel type language to interact interact with the data and declared stream processing applications so that means no Java gnome setting up kind of spot classes and stuff like that's such another good rate. The people do often take win. We're taking the change data capture log from our transactional database and we're getting into Kafka and we're GONNA use that change data capture log to update sink databases. Are we getting the entire database reading the entire change log into cough. That sounds like putting a lot of data into Kafka. It depends entirely on how you configure your CD too so if you're using something night to be easy I'm just very good popular open source Kafka connector sports things like postcards my sequel and so on you simply say I'm interested in these particular tables so so you're putting the whole database if appropriate but often times obey well. I want to sales table. I want product table. I want the customer table on you. Just get the events related to those particular tables bills but it comes down to how you configure the particular integration. Let's take the search example so like let's say I want to have my transactional database. Get a search index built over that transactional database if I'm doing that am I starting by seeding the searching necks with with a copy of my database or is the entire process based around that change data capture log. Do I have the entire historical change data you to capture luck. I guess I'm wondering you WanNa have this search index that you're going to be updating over time to be a reflection to be a searchable reflection of the transactional database that you have but when you're bootstrapping that search index do you just take the actual database or do you just take the historical historical change data capture lock so the way that many of the tools jack is they'll take snapshots of the current database states they recalled the end the points in the transaction log at which that was taken to the system and then from thereon end capture any events out of the transaction log so it's kind of current state to throw a select whatever plus transaction log from that point said using the tonight show. You don't miss anything in between okay. Let's talk a little bit about this interface between a database ace and Kafka and that's Kafka connect. That is the interface point of how you're getting your data from one of these sources into Kafka Africa or from Kafka Into One of these sinks explain. What Kafka connect is Kafka connects in a nutshell? Lets you do streaming integration between source systems and Kafka Kafka and your target systems. It's just configuration file base so you'd have to write any code and it's part of Apache Kafka so if you're wanting to get data from Kafka down. HDFS Kafka tastes three from database into cars go from catheter search all of the possible permutations of pipelines you can think of of almost always you want to be doing that through Kafka connect so connects souls loss of the problems of integration between systems it it does things like fault. Tolerance distribution system is built on top of carcass semantics self so you can scale it out it also handles the more tricky things like schemers and offsets and all of the kind of things if you decide to write it yourself us. Many people unfortunately start off by doing and then I'll write myself a spark job to get the state from here to here are writer Java Program to get data from this database into here. All of those tricky now things Kafka connects has solved already so it's it's a solved problem. Rally also has additional capabilities like transforming the data as it passes through so whilst we've talked about stream processing frameworks works for the kind of the modern front stuff Kafka connects also has the ability translations on the data as it passes through these called single message transforms and you can do some pretty cool stuff you can start to mask the data or job fields or and rich them change data types all through Africa connect all through configuration files so actually makes it much more accessible to many more people other than just those who are going to write a Java programmer writes a spark code. Oh that's useful so you can begin in the transform process at the entry point of data from the source into the coff- connect system absolutely so for example standpoint if you're putting in data from a table with hundreds of columns which is not uncommon and you say well. Actually I only need a subset of those you could just drop out all of the additional ones or or if you've got data coming in and it's heavily identifiable information it's got credit cards. It's got addresses and you don't need data and actually holding that day so within Kafka and has additional implications vacations you can say well. We'll just call themselves on at the same time this column over here. Let's Kostas and change the dates type this one over here. Let's add in some meditators lineage information about where the information's come from and again ultras through configuration files connect runs as a distributed process so there's multiple nodes for a Kafka connect process is it always true could could explain the parallelism model of Kafka connect show so connects connects and this is an important point to make actually does not run on your brokers. Nothing runs on your Kafka brokers except possibly zookeeper and then there's plenty of people who disagree without not as well but Kafka connect run separate from your brokers. You can run a single instance of Kafka connects if you want to but as soon as he wants fault tolerance as soon as you once additional capacity you then deploy additional Kafka connect workers and apply them as part of the same group Kafka connect will then distribute the workload across those instances this if you have a single connector like a sink process that taking days off topic out to a target you can have multiple Waco acas which will then form a consumer group and read that days from powerful if you're again data from database for example Kafka connect compare lies in I'm just I'm Rachel multiple tables at the same time so the powerless misdefined within connects unless you eat scale not win. Would you need that parallelism awesome. Is that only if you have a database that's changing really rapidly so it depends on what kind of threw you want to get on your data. If you call it happy slept Kafka connects sit unstuck through the data then I guess you don't need us but it gives you the ability to do so if and when you decide you want to get the data if the data's being created a greater rate eight that has been ingested on a single workload so now that we've talked through some of the finer points of this streaming. ETL process could you zoom out again and describe the streaming ETL process for let's say let's got a transactional Mongo database. That's my source of truth database. I've scaled up over time and now I want to build a search index on top of that database. I WANNA use Kafka as the middleware described at the end to end process of getting that data for Mongo into a searchable index shaw. There's two different answers to that's. I'll give you the simple one first and then I'll come bucks the second one the simple one is you deploy Kafka connects you use the BPM connector which is plugging Kafka connect so I should have mentioned Kafka connect a plug in based architecture you plug in the appropriate connective technology so we plug into be easier..

Kafka Kafka Kafka HDFS Kafka Africa LSU McKay Kostas Kim place Dr Association caffeine coff writer programmer
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

11:43 min | 1 year ago

"kafka" Discussed on Software Engineering Daily

"Kafka. It can be useful for a company. That's moving towards a micro services architecture and I. I'd like to discuss that with you but before we talk about `bout that actually want explore a quick side note because you go to a lot of conferences. We make a point of communicating with a lot of people throughout the ecosystem. I've we've had conversations with a few engineers. I respect a lot recently about the quote Unquote Micro Services Industrial Complex which is kind of this idea that not to paint conspiracy theories but there is an incentive of vendors to encourage encourage people to buy things that caused them to spend more money and perhaps micro services could be categorized sometimes in some cases as one of those things the maybe there could be an alternative path that we as an industry. Maybe should be going down like like how could we architect are monoliths better. Is there a possibility that our industry has moved too far in the direction of micro services yeah yeah no that there always is for hyped thing like this and I feel like a little part of me dies every time I say the word just because it's such a hype type thing and I know where we are in that cycle right now and I feel like I'm a part of it and you know you're. You're absolutely right and here's the ads for these products. That's true. That's true you're you're. You're much more a part of the industrial complex than I am but you know if it's total when I talk about micro services Kafka and this kind of broader the architectural sure properties you can get out of a system by making Kafka your your system of record. I try to start by talking about what a terrible idea. Micro services actually are and I want want people to realize that like this is awful and there has to be some really good reason that you would do this because there are you know for one part of your monolith to call another part of your monolith. There's literally hardware in silicon to make that operation perform well and take take like a small number of clock cycles and there's nothing could be more optimized than a method call yet. That's not good enough for us. We're we're gonNA take what used to be a method call and turn it into this lumbering traverse through a network stack and a network interface and and VM container a vm like what is the matter with us or you're right yeah. I always really try to dramatize how horrible that is and I I also try to tell people. If there's any way you can possibly live your life as a software. Engineer will pursue that calling and not build distributed systems. Please take that option. You know you're GONNA have more time for hobbies and you're going to go home earlier and probably sleep better and maybe live a few years longer. You know it's it's it's better and nobody ever listens you. Make a good point and I think there's gotTa be some truth that it's a you know as architectural trends and language and framework. Selections Become Status Indicators then there will be you know maybe lower integrity software architects and developers who make decisions to adopt them because it makes them look or feel good and that's going to be the case. He's here like there's going to be some adoption. That's a thing right. Obviously there's a but coming in that but is I don't think micro services aren't irrational bubble. I I do think despite the Gross Stupidity of taking a method call and turning it into a call across a network and to making everything a distributed system you know all that's bad and has real costs. The benefits actually exist for a certain class a system in that class system is not that large in the days as when we said big data I think it turned out that most of the systems we were talking about. Were really useful not all that much of the time right all the web scale examples. We always gave an and I I've been a part of this are actual web. Scale companies like Netflix Amazon. There aren't too many of those. There's this the whole rest of the world of people working for medium-sized companies writing enterprise applications. What about us. I think that US we actually do get utility Outta micro services and our as an architecture. I'd love to dig into that more but I think it is actually worth it but we need to get there through the gate of remember ember how bad this is going to make your life and if it's not radically improving some other part of your life don't go there so let's assume. I do want to break break-up. My monolith and I think you know we could go a lot. Deeper on things you just said. I think they'll be very interesting. Conversation probably not totally uh well suited to the focus of this conversation because there's a lot more. I want to explore with you so let's assume we do want to break up the monolith. We've got some result. What's that we're doing it. We maybe we are that architect. Maybe we're giving a prescription for malicious software architects to spin their wheels for several years but let's assume we WANNA break up the monolith. WHY IS KAFKA USEFUL FOR BREAKING UP MONOLITHIC SOFTWARE SYSTEMS GOT IT okay well. The first micro services deployments communicated through shared database and and the reason they did that is because well we had one lying around and we already knew how to use it and everything was there and it seems so obvious right seems so obvious that I need to pick up new orders and I need to query the user table and you know find. There's a service that maintains updates to users table and the ship and service is just GonNa go query that sucker and get what it needs and it's real fast and it's great and what happened with those was that they became extremely extremely inconvenient. monoliths in the sense that the shared. Schema of the database tended to lock the version ing of the services. If you change change one thing that that involves a scheme change now you have to change every other service that relies on that Schema and all those had to be released at the same time maybe in a particular order in really bad cases and it was it was all of the bad things about micro services and none of the good ones. I was not able to independently evolve and version and think about a single service I still had to think about the whole thing and by the way I think that's fundamentally. The reason why monoliths fail as monoliths is that they're just too hard to think about and so that one thing that I was trying to get I couldn't get that was we didn't know any better we tried that we quickly discovered a community immunity terrible idea and nextstep was and really is having services that don't share a database they have maybe local databases his but they talk through some kind of PC mechanism gop or rest or something and that was better that allowed made it easier here to independently version services and didn't have this this locking together through a shared Schema but it created crater problems of service discovery for which any number of tools have emerged to help alleviate and failure cascades became a problem. You know if you're synchronous. Lee Calling one service and it's synchronicity calling another and another another failure six services down the line is is going to cascade up. There are ways to mitigate that but you know these are just some of the balls you had to juggle service discovery. Failure cascades were harder now that that restful immigration services was still super easy because is calling a service feels like calling a function calling a method. It's just over the network instead of you know through called structuring on a processor so part love. I think why that paradigm is feeling some strain is because it's easy and all the API's we used to do you know that that RPC integration make it look like a method call yet. It is entirely unlike method call in terms of its operational characteristics. It can fail L. In all kinds of ways takes literally orders of magnitude longer so getting the developer in the mood of the method call while it not really being a method call. I think is turning out to be a bad thing so that gets us to Wyckoff and micro services matter if instead every micro service is to use another buzzword what a reactive micro service or we could avoid that simply say event driven then you know the system at some point receives an input and it doesn't matter what that interface this is some service receives input. Maybe a Web form is submitted or whatever it does some computation and produces a message now whoever is interested in that message like my shipment service a moment ago you know the first service may be as an order validation service in the ECOMMERCE front end throws some Blob of Jason and an an endpoint in that service and it validates it produces a message to Akaka topic. Well the shipment service. Now can pick that up and you know do its thing saying that we talked about with the user data able to share data that way and it's able to do its work simply win. That event arrives in the topic. Now that's clear enough enough but the amazing thing that happens here is those orders are in the topic now and in our little minimum viable product ECOMMERCE system here like we get it. There's orders. There's users you gotta ship things might need to make payment service at some point seems a good idea for mvp what other concerns might arise that have to do with validated orders. Just stop and think about that for a second well. The answer is literally. Anything and you don't even need to be the person who knows that I could have some mobile. APP that is location unaware and if you just bought something that has some product compliment that's sold by a partner retail business. That's near where you are. I'll send you a notification indication. That actually sounds creepy. Edge made that up but you know that's another fairly complex service that you might develop that is going to process. This topic validated orders. Maybe fraud detection is another one. Maybe I want to look at the history of orders and see if something looks suspicious has nothing to do with payment has nothing to do a shipment off. The top of of our heads here were just kind of making up some other reasons to consume from that topic of validated orders as the system evolves as the team grows as the company grows. You know scale this in any way you want. Here's that topic with those orders in it and they can last in in that topic for as long as you want them to you. Can you can make that retention infinite. If you want and any service anybody wants to write that can do something with that will they can write the service subject to authorisation and any regulatory concerns or anything like that the data's there and now new services services can kinda grow up in that..

Unquote Micro Services Industr developer Kafka Kafka. Netflix fraud Engineer Wyckoff gop Lee mvp Jason Akaka partner
"kafka" Discussed on Recode Media with Peter Kafka

Recode Media with Peter Kafka

03:29 min | 3 years ago

"kafka" Discussed on Recode Media with Peter Kafka

"This is Recode media with Peter Kafka. That is me. I'm parked on the vox media podcast network and not advice, media headquarters in New York City somewhere in Brooklyn in the slate panoply empire. Thanks for letting his interview here guys. If you like the show, tell someone else about there. I'm done with the pitch. I want to get right to interview with Nicole hall of center. One of my favorite writers directors. Thank you. The whole reason I do podcast is, is why theoretically them. So I can talk about tech and median their collision, but Joyce to talk to people like you so Yemi and yeah, you for showing that if you have an excuse to meet the people, you wanna meet. They tell you shouldn't meet people you really like. But I don't think that's true. Disappoint you and we'll see how this goes. Okay, I'll do my best. The reason you're talking today as you have a new movie out which you can see as you listen to this right now, you should listen to this and then go watch it on net flicks it's as called the land of steady habits. Do we call this net flicks movie, just call it a movie that is streaming on Netflix. I don't think I would ever say it's a Netflix movie, but I would say it will be on net flicks so it's still call it. I made a movie and you can watch it on net flicks and net flicks made it used to go to to things like the landmark theater and sunshine cinemas to go watch your stuff because you are you a veteran of the nineties indie movies seen. And now I get to see your stuff at home, which if you had your druthers, what would you prefer that I watched something like like, this would you rather that I watched this movie at home or theater? I prefer you watch it on your phone. I mean on your watch, expand your limits. Watch, of course in theater. Of course dark with an audience trapped? Yes, but we'll this will do it. It looked great on my TV. Yeah, go good. No, I'm not feeling bad about it streaming on Netflix, you know, of course, the preferences that theater, but net flicks was terrific to work with and four and more people will see the movie and I'm not sure I had a real choice with this one because I really wanted to make it my way I want to talk about how came to be we should I, it is the New York Times reviewed. I saw called achiever esque which means it's about a middle aged guy, having a middle aged life crisis in upstate in Connecticut, actually, in Westport, Connecticut wasn't more Connecticut. Ben Mendelsohn great actor, many of you seem maybe don't know who Ben Mendelsohn is, but he's one of those those guys go. Yeah, it's been Mendelssohn. He's great normal. You make movies, Catherine keener, she's not in this movie. What else should we talk about the plot of the movie or should we just let them watch it? I would say. Watch it and it, it kind of starts a little methodically. Most of my movies sort of do like, okay, okay. And then by the time it ramps up, it really packs a wallop don't give up on it. Things happen. Good. It's own pace. Starts off with Ben Mendelsohn looking. He's wandering around a bed, bath and beyond. As you make you make it look incredibly intimidating. It isn't timid. Ating it's ghastly and fund can be both, but it's about this guy and his problems and how he wreaks havoc on other people's lives. Very narcissistic unhappy man want to spoil it a little bit. There's no marvel tie and there is no superhero of any sort. I don't think much metaphor, although he plays a marvel villain. Yes, in another movie. But no, it's about families in parenting, and it's funny, and it's very sad as well..

Ben Mendelsohn Netflix Peter Kafka vox media Connecticut New York City Catherine keener Nicole hall New York Times Brooklyn Joyce Yemi Westport
"kafka" Discussed on Recode Media with Peter Kafka

Recode Media with Peter Kafka

01:34 min | 3 years ago

"kafka" Discussed on Recode Media with Peter Kafka

"This is Rico immediate with Peter Kafka. That is me. I'm part of the vox media podcast network here box media headquarters in New York City. If you like this show, please tell someone else about thank you here. Anthony would CEO founder of Roku, welcoming Anthony. I beat her second. Yeah. Remember the last time we sat and talked for a while. I think it was years ago and he was on a stage LA. That's right. I remember. And I asked you all the standard questions, which is you're gonna get crushed by apple crush by Google, crush by Amazon. What's going to happen that it was years ago since then you went public? Uh-huh. Yes. Successfully so far. Check the market cap this morning. The company's worth about seven billion dollars. So things have gone. Well, yeah, we went public year ago and you know businesses, great. I mean, it's it's an awesome time to be in the streaming business. Yeah, I'm writing about another company that wants to raise a bunch of money, and you are now part of their pitch. Which is how Roku can go public then then we ought to be raised this much money at a billion dollar valuation because there's room for sort of lots of companies don't about Natta participate in the streaming, boom. Yeah. I mean, you know, one of the challenges with Roku going public, was that this perception that were hardware company? I think it's largely gone away because you used to sell hardware and you still do we still hardware, right? But our goal for selling harder is to build active accounts. We're not trying to make money on on hardware, but my point was just that that was a challenge for us. And so we spend a lot of time with investors explaining our business model, which is about services and advertising.

Anthony Peter Kafka vox media New York City Rico LA apple Amazon CEO Google founder seven billion dollars billion dollar
"kafka" Discussed on Recode Media with Peter Kafka

Recode Media with Peter Kafka

01:54 min | 3 years ago

"kafka" Discussed on Recode Media with Peter Kafka

"This is Recode media with Peter Kafka and I am obviously not Peter Kafka. I'm CARA Swisher. The editor at large of Recode and the host of Rico, decode. If you're wondering why I'm here instead of Peter listened to last week's episode this week, you will once again be in the hands of Recode senior social media editor Kurt Wagner. Before we get to this interview with discord CEO Jason Citron here's your weekly reminder. Tell someone else about the show. I have to say that or else Peterkov will. It will be extra grumpy with me when he gets back and Peter Kafka is always extra grumpy. So it's worse. So that's all I've got for now. I'll be back later in the show to read some ads taken away, Kurt. Thank you care. I am here with Jason Citron who is the CEO, discord, Jason, welcome to the show. Thanks for having me. Thank you so much for being here. We're going to talk about a ton of stuff today. US have some news that you're going to talk about around talking about gaming. We're gonna talk about messaging, text, voice, all that stuff, but I have to ask you the most important question gaming right now, which is. How many hours a day are you playing fortnight? Oh, not enough. You know, running this whole company think takes a lot of time. Yeah, his four night. Are you a fortnight guy? I'm really not. Actually. It feels like an I have. I had a ten year old trying to explain to me how fortnight worked, and it made me feel older than I've ever felt in my entire life. Yeah. But it seems like the game of the moment, right? I mean, I'm not missing that. Yes, that's right. That's right. Are your employees playing it or everybody? Everybody's playing at me, I think. Yeah. Well, you need to stop working so hard at your company, apparently. Cool. Well, I think where I want to start today is kind of what discord is and who you are. I think outside of maybe the tech world, there's certainly a large group of gamers who know who you are, but I don't really even know much. We've met before few times and chatted, but I'm when consider myself a hardcore gamer by any stretch..

Peter Kafka Jason Citron Kurt Wagner US CEO editor CARA Swisher Rico ten year
"kafka" Discussed on Recode Media with Peter Kafka

Recode Media with Peter Kafka

02:00 min | 3 years ago

"kafka" Discussed on Recode Media with Peter Kafka

"This is recode media with peter kafka that's me i'm part of the vox media podcast network appeared box media headquarters in new york city by the times of as it comes out code media will be over promoting could be there for many weeks now it's done to make sure to go to recode dot net fraud coverage there's videos was podcast raves the interviews it's an amazing event i can tell you that now even though i'm talking about it in the future will go read it won't say thanks to everyone who came okay that's the host promotion of the event that happen here's the thing that's happening now i am talking to lauren duka live in person high alert how yeah i'm excellent delighted to meet you in person thing i've been reading about you for a year plus pay the task fifteen minutes of ending here how do you know know know know know a year does it we gotta get warned duke gun the podcast mr window you are the person who rose to national consciousness for writing a single article for teen vogue dot com this channel everyone knows the article besmirch tell us the the headline of the articles donald trump is gas leading america still jail definition of a viral piece of konta oh gosh you know there's like numbers on it and i had thought based on the numbers i've been taught at huffpost that i had gone viral before but my joke about this as it's like an orgasm one you know you now this was very different just of this it was like a tidal af i mean the the sheer magnitude and of reactions it was still kind of haven't gotten over it it's still kind of going however here that's where we got it and there was the carlsson and narrowest incident and then tucker carlson kept of a labor lawyer for you for for many months how he's stunned up pieces my violent tweets on the part of the a violent left turn serve that that is what i want to talk i wanna talk about how you rocketed into public consciousness uh a lot of people's lives have been changed by donald trump people in the media business is but a good thing for that it's obviously there's a little.

peter kafka vox media new york fraud lauren duka duke donald trump america tucker carlson fifteen minutes
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

02:02 min | 3 years ago

"kafka" Discussed on Software Engineering Daily

"Those things and rewrite to the same kafka cluster the locations because the latter you can take a latitude longitude and you can turn that into a location maybe this person like maybe you wanna take the lat long and for during the real world instead of a virtual world you could query yelp real quick and find out what is the business or what is the pharmacy in your example with the pharmacy that's closest to that long or is that person at the pharmacy can you detect that maybe that's your form of enrichment that's a very basic query that can be made at scale in parallel by a stream processing system you got to choose between your different stream processing system so i think that's what will get into next how do you choose between difference you've got spark streaming you've got kafka streams you've got flank you've got all these different stream processing systems that in some sense are doing the same thing they grab a big chunk of data and they perform map reduce like operations on that chunk of data what are the tradeoffs between different streaming systems that can do that kind of operation yeah that's a very good question because it satan formed outside it seems like oh we have owned those two process hours ends are on the same and if you don't to anyone who is asking the business and knocked an engineer his oh absolutely say oh it's all the same just pick something like may be the ones that we lack the support called trucks who is the most but if you twice driving into the dayton differences in deductible those zones differences in be supported programming languages and if you are a on fanatic because then it's kind of narrows your a choice of time it and then also in terms of day action dot the modern and comparability they really differ and even zoll at the end of the day and oats on touring machines who can do every single with anything and obstructions a provide are quite different and they can make.

yelp engineer kafka dayton
"kafka" Discussed on Software Engineering Daily

Software Engineering Daily

01:56 min | 4 years ago

"kafka" Discussed on Software Engineering Daily

"So help us understand the hosting model here and what we're getting out of like a lawyer kafka as a service thing 'cause i think what you'll lotta people do these days is they have their own kafka deployment that is running on aws along with the rest of their infrastructure and thrown aws or googlecloud or whatever it is mostly awswhat would we get out of a this the kafka's service conflict cloud like what is the business offering so the business offering is that conflict cloud results fully managed streaming data service that offers kafka as a service today and in the future will offer the entire council platform with cuba registry and the rest roxie in lots of connectors and so the the uh the way it offers diet is essentially light just like any other managed data service where what you do as a user is specify what you need in terms off compute storage of reliability and and just depend on conflict cloud to them wrong your cough gov sort of our or your skin registry fully managed manner so is it is it still running on servers that the buyer purchases from aws no eat it it is a fully of service in the sense that we take care of provisioning service onto need the covers on the clark provider their droughts said some sense it's just like any other monte tenant hosted fully manet'sarvis that o'clock provider might offer except that it is cloud provided naas stake in it will support all the opensource conflate the eyes and the opensource kafka protocol.

kafka aws cuba clark