Courier with Troy Goode
Imagine yourself using a GIG economy application like Uber that application generates lots of notifications there's SMS there's mobile phone application native updates. There's emails if you order a ride from Uber. You might receive a text message and a push notification at the same time and if an APP overloads the user notifications the user might end up annoyed and delete the APP from their phone altogether. But perhaps all of these notifications are necessary. You would rather get three simultaneous notifications from your food. Delivery APP then failed to get your food on time. If you're a mobile application developer you might not know which channels are working for you. You might not know which SMS notifications are reaching the user and which times the user needs to get an email update. So how do you solve this problem? What choice do you have? But to send the user all these different kinds of notifications at large companies like Lincoln or like Uber. There are actually entire teams devoted to figuring out how to optimize the notifications that they send you. It has a surprisingly large impact on the usability of a mobile application troy. Good is the founder of currier a company that provides this kind of notification optimization as a service and this might sound like a small trivial problem but it actually has large impact on the usage of our APPS and. It's not an easy engineering problem. Troy joins the show to talk about the problem that courier solves and the back end infrastructure. That powers it courier is built entirely on service. Api's this is a great case. Study in how to build a completely scalable infrastructure product based on service tools. And we've done many other shows on service products and things. They have been built entirely in service. If you WANNA find all those episodes you can go to software daily Dot Com. You can look at our mobile APPS. They have all of our episodes organized by categories and in terms of which episodes of the most popular. You can check those out at software. Daily Dot Com this episode of Software Engineering Daily is brought to you by data dog a full stack monitoring platform that integrates with over three hundred and fifty technologies like Gremlin Pager duty. Aws Lambda spinnaker and more with rich visualizations and algorithm. Michael Arts data. Dog Can help you. Monitor the effects of chaos experiments can also identify weaknesses and improve the reliability of your systems visit software engineering daily Dot Com slash data dog to start a free fourteen day trial and receive one of data dogs famously. Cozy t shirts that software engineering daily dot Com slash data dog. Thank you to data dog for being a long running sponsor of software engineering daily Troy. Good welcome to suffer engine daily. Thanks Jeff I'm glad to be here whenever I use a GIG economy service. I get lots of notification absolutely. This is true whether I'm using grocery delivery or I'm using ridesharing I get SMS. I get mobile phone updates second emails and sometimes it gets so many different types of notifications that I ended up. Silencing all of them and then I get a phone call because the service can't reach me any other way. How did we get to the point where we're drowning in notifications? Well I think that if we rewind the clock maybe even only ten years there was only one meaningful notification channel and that was email for most offer products and I think that what has happened with the advent of mobile devices and the ubiquity of them is now it so easy for technology companies to reach you and frankly those notification channels perform so much higher than email in terms of creating an engagement. Basically you clicking on that and driving you back into their product that it's a bit of tragedy of the Commons situation where it's optimal for every one of those individual products to maximize the likelihood that you're going to engage reaching out to you with many notifications on the other. Hand that when everybody does that. That creates a culture where you become very fatigued by notifications one where you didn't start to think. About UNSUBSCRIBE ING and I. I typically blocked badges because that drives my ocd crazy and all of a sudden. Because everybody's doing it now. It's maybe not the optimal strategy for any one of them. But who's going to be the first one to break and stop sending as many because that's not going to cause the rest of the industry to do the same. Let's say I'm building a application. That has a lot notifications. What are some anti patterns of people who are building kinds of APPS? I think the easily the number one anti pattern is if you are a product if you have a product that supports multiple channels. So let's think about you know. Typically it would be maybe email. Sms voice push those would be typical channels. You might sport sending notifications across all of them. That's probably the biggest anti pattern in one of the biggest ways to drive on subscribes. Certainly you know there are scenarios maybe like emergency scenarios where that might make sense if your pager duty. There are scenarios where this makes sense but for most products. You need to be more selective about which channel you're using and win right so like grocery delivery. There's a few APPs aware every time the groceries are going to be delivered. I get a message on SMS and in the APP. And that's actually. I don't know if it's bad because I do want to know that they're being delivered but it seems like maybe it varies from person to person from customer customer. From grocery delivery. Grocery delivery are their systems for. I know you're working on a system like this and this is this is the kind of use case that drove you to start courier. But prior to currier what were people doing to customize their notification systems. Even today we're still in the early days I'd say most firms are just building this out internally and there's one I can talk about or there's a couple that I can talk about because they've they've talked about what they're doing publicly one of them is twitch so Amazon own streaming company they the year before last launch. What they call smart notifications prior to smarten identifications. They had three channels. They had in-app notifications. So that appear while you're in the twitch experience either on the web or in your mobile APP they had mobile push notifications. That would show up to the home screen of your mobile device when you weren't using the APP and they had email notifications. They were sending. Let's say I follow your twitter stream and you come online. They want to let me know that right because it's their business model to drive me inside of the APP. I'm going to see advertisements. And that's how which is going to drive revenue if. I don't know that you've gone online. Then there's no value being produced. It's not youtube or I can come and check it out later. Really so it's oriented around. This live event in that notification is incredibly important so they started out by sending notifications to all three of those channels at the same time. You know boom boom boom. And we've experienced this with new grocery delivery and other APPS and what they found was that the unsubscribe rate skyrocketed. And when somebody would unsubscribe now it wasn't they were unsubscribe from one of those channels thereon subscribing from all of them and all of a sudden. Now you can't reach them at all so the next time jeff comes online to start streaming because troy unsubscribe prior time. They can't reach troy. Troy doesn't come back to the platform. Potentially ever there's no other way to reach him. And now you've lost that revenue. And what would be the alternative to that like? How could twitch avoid the scenario of somebody completely unsubscribe from does notifications? Yes they're smart notifications platform that they released what it does pretty simply as keeps those three channels in place but it just adds some logic around that and they have other things that they do. That's more sophisticated but at the simplest terms. What it says is okay. We're going to prioritize these channels based upon the likelihood that it's going to cause troy to engage in come into our experience we know from benchmark data that the highest performing channel that you can have is inap- notifications if you have long session times right. So twitch has long session times people come. They watch they stay. Let's say I'm watching somebody else's stream and now jeff comes online and I followed Jeff if I'm online in that experience than which will show me and in-app notification while I'm in that experience inside of twitch and they will not send mobile push notification they will not send that email right and that's going to be the highest likelihood activity to get me to reengage and stay longer within twitch. Let's say I'm not I'm not online. I'm not watching which I'm walking down the street. I'm at lunch. Then they look and see okay. Has Troy downloaded the mobile APP? Has the accepted push notifications. Has He engaged with the mobile experience within the last period of time to consider him an active mobile user? If so now we in mobile push right and if none of those are true then finally use email as that fallback mechanism. It's the worst performing channel but it is ubiquitous in. At least it's a last ditch way to try and reach me and before we get further into the mechanics the details of this. Let's just drive this home a little bit further because currier is pretty beautiful. You just showed it to me. And you're spending a lotta time on this and I know that there's people listening right now. Who are like this is definitely not something I'm interested in why? Why would I care about notification optimization? How can this really be that important? You know but as I've understood from looking your product and just talking to you. This is actually something that impacts a ton of application CETERA. So can you give more of a survey of the landscape and explain why maybe called notification optimization or whatever whatever you WanNa call this category of the problem. Why is this important? Yeah so I think that it is important to any product whether you're a technology company or some other service where what you need to do is drive your customers and users back into your software experience right a lot of technology companies but this bill right. Bbc sast products absolutely a lot of BBC products media products for sure but also maybe more traditional services we think about things like United Airlines and all the notifications that we get often on many channels at the same time about a gate change that are trying to drive us in for more information into their APP so that we can be informed about what has happened. Fundamentally businesses usually need to reach and communicate with their users. And the way you do. Soviet technology today is through notifications whether that be email or mobile. Push or one of the many other channels. What has happened though over? Especially the last four or five years is that the number of channels that are available to us. As engineers and product managers have exploded but the complexity of integrating them is incredibly high. And so what frequently happens isn't the earliest stages of a software product. We just stick to email or we just stick to mobile plush. If you're just in the mobile push landscape so let's save a mobile first product. That might be pretty good for you. Actually because mobile pushes very strongly performing channel there's ways at career can add value there but where we had the most value is if you're starting on the web and you starring just email because email is the worst performing channel. If you're doing well at email you can get about a three to five percent. Engagement RATE FOR CLICK throughs. If you add another channel any user you interact with channel you can expect in the twenty to thirty percent range of click through so you're looking at five x or more increasing engagement rates just by adding additional channel. Now the question comes. How easily can I add that channel? And how easily can I maintain it over time? Because as I add more channels have more templates. More content. More copy that needs to be created controlled So as a company goes from email and SMS to email and SMS and perhaps intercom what happens internally at the company as they're getting more and more messaging based systems. Maybe they're they're adding Twi- Leo. They're adding maybe some kind of slack. Integration they're adding ENTERCOM who inside the company is managing all those notification flows. How is the company Keeping Track of okay? This happens and then it you know it tees off a text message or this happens and then it opens up a desk ticket and it's going to send them an email and open up in ticket over email. Is there somebody inside the company? That is the central point of understanding the user experience of notifications. It's a really interesting question. Because so in the earliest days it ends up being the engineers. Typically IF YOU'RE A small company. The product manager is probably. Who's making these decisions? Because the product manager is the only person incentivized to really be thinking about engagement later. Stage companies have growth teams and those teams. Maybe take a little bit of that over. But in the earliest days the product manager wants this but the engineers the only one that can implemented so as you look. If you're going to add email it's simple sign up for secret account or male gun postmark. They have really simple. You IS AN EP. Is to connect into your platform. You had that second channel you add. Sms Nelson. You have to either do it. We talked about not doing which just send it to both of those. Api's and plug in both API's and just send it both or you're putting some sort of control layer of on top of that and it could be as simple as switch statement or if then but you'll start to think about okay. Well what are the conditions that are going to control this? If then statement right is it going to use context things like is the user online which means you need to have a presence system is going to use historical data about which of these channels. The user has engaged previously or preference data. So you've got to start to pull from a persistence layer to inform this decision. You start to do things like okay. Well I haven't been but what happens. I've picked one of those channels and delivered to that channel fails so that. Api is offline well. Do I Q that. Backup to deliver it on that channel later or do I fail over to a different channel. It might depend on notification in the urgency of that notification. Some you might be okay with that notification being delivered tomorrow others you might say look I just want to get it out on any of these channels immediately so there's a lot of this complexity that the engineer has to build out but engineer honestly often doesn't care right it's the product manager who cares an engineer just wants to get back to building value instead of the core product and they kind of just get pulled aside and we see we see in the largest organizations. They have full dedicated teams and departments. That literally work on this kind of infrastructure all day every day for years he say more about that absolutely so there's a couple public examples. I'm happy to share. Lincoln has a internal infrastructure named air traffic controller. And it's it's available online on their blog. You can go check it out. And they talk about the billions of notifications that they send every day all of them. Lincoln teams across the world across their campus down here. They all use the same internal notification infrastructure. So they're all kind of calling this internal. Api There are thirty engineers that work full-time on that infrastructure. All Day every day and have done so for years you're talking tens of millions of dollars of investment into control plane that can understand. Should I send this notification to Jeff over mobile push or over email? Should I send this notification to Jeff at all? Should I send its batch with other updates right of if somebody looked at Arlington profile? Do we send you a notification immediately or do we wait and say thirty seven people looked at your profile Jeff and include that as a different notification. When do we send it if somebody looked at your profile or sent you a message at three? Am Do we hit you up immediately or do we wait until nine? Am Your local time and deliver that notification then all of that infrastructure that makes these decisions is abstracted away from the core product engineers into this air traffic control system that is again built by thirty plus engineers within Lincoln then provided as a service to the rest of the organization. The idea of currier is that that kind of air traffic controller. This kind of person who is deciding. The user flows of receiving notifications interacting with those notifications. Currier does that as a service Q. Explain what your company building in more detail. Yeah sure so our Weiss. He pitched was segment for notifications. So the the core idea originally and I think still is take all the channels that you're already using or aspire to us and the actual providers for them so we don't send us a mass. We don't send email our customers plug in their twitter account. They're send Grit Account there slack. Bought they plug those into career and then we offer a single API our customers and developers publishing events to the courier API and we use that context and those rules in this benchmark data to figure out which of those channels we deliver that notification to Jeff Right now in addition to that one of our key value propositions and something that's just ingrained in our culture is enabling non engineers so taking the work that engineers do and helping them to empower their other teammates. So that somebody who's closer to the problem that product manager that growth team member they can take ownership over these questions. These challenges they can add new channels. They can change the content and the copy of these notifications without filing Jira tickets saying. Hey Troy can you change the text on these twenty templates to from x? To why do the tools like Twi- Leo or send grid or whatever else is being used to manage notifications? Do any of them have workflow managers like this or is this just? This is a piece of middleware that nobody has built as far as you now. All the marketing side of the world has had this forever marketing automation. My background. I led engineering for a company called Ella which helped create the marketing automation space. Took that public back in two thousand twelve and ultimately sold it to Oracle if you're sending marketing email marketing campaigns or even today multichannel marketing campaigns. You you have infrastructure. You have middleware to do this. But that's really a belt promotional activities it's about scheduling campaigns. That are going to go out at certain times. It's about then responding to out of product events things like did you engage. Did you fill out? This gated entry form to download this white paper. And if so we're going to put you into this separate drip campaign and follow up on this cadence. What we haven't had is the equivalent of that for product businesses for triggered events that are happening and coming out of your product. Some get close things like customer dot. Io and intercom can do aspects of this but they are still more focused on those marketing use cases. Rather than the driving internal product engagement use cases that product management and Growth Teams caravel example you gave with twitch so like I get twitch notification and I opened that vacation that I can understand getting feedback or the feedback being driven back into twitch and twitch being able to make decisions based off of win that user. Open the notification and how they interacted with it. Is that true for all channels like can I get feedback on a in-app notification as well as an email as well as an SMS? Do all of these things give me enough information to be able to have insights about what a particular user prefers so the answer is mostly yes in that while the method varies each of those channels have ways that you can access that engagement. Data fundamentally did the user engage now email is the oldest and most mature of these channels and it has the most options available sometime. Sue the Chagrin of users so. We've Seen Superhuman. How some some controversy over there open rate tracking right in? That's been a feature of email for longtime. Do people open this. You can't generally tell that in many other channels but you can always with different methods. Tell did the user click did they. A good notification has call to action. Right it's not an informational F Y. It's Hey Jeff I'd like you to do something I want you to log in and watch this twitch stream. I want you troy at mentioned you. I want you to reply to him or at least look at what he said. It's you put stuff into your shopping cart two weeks ago and it's still sitting there going to complete that purchase. There's something that you're asking that user to do so you can tie the action of. Did they perform that action? Or at least take the next step toward that to those notifications across those channels the challenges the literal mechanics of doing so is different in various channels and the data that you capture silo. D- in each of those providers today so send grid out of the box and this is true for just about every transactional mail provider is going to track this for you. We've probably all seen it when we get an email me hover over that link and it says you know track Dotson Grid Dot Com you. Click that it's tracking. Did the user click it? You can instrument the other channels to do similar things but that data is not flowing to a single system. That's one of the things that career provides you can start to understand. What is that recipient profile? Look like across all these channels so that we can make more intelligent decisions than you. Otherwise could right because Singapore might know how effective email is for you but they don't know how effective slack is for you and probably never can likewise twi- SMS snow can no to certain degree how effective SMS for you. But they can never know how effective facebook messenger is as a channel for you. You probably do not enjoy searching for a job. Engineers don't like sacrificing their time to do phone screens and we don't like doing whiteboard. Problems and working on tedious take home projects. Everyone knows the software. Hiring process is not perfect. But what's the alternative triple bite? Is the alternative. Triple Bite is a platform for finding a great software job. Faster Triple Bite. Works with four hundred. Plus Tech Companies Including Dropbox Adobe Core Sarah and cruise automation triple bite. Improves the hiring process? By saving you time and fast-tracking you to final interviews at triple by dot com slash s daily. You can start your process by taking a quiz and after the quiz you get interviewed by triple bite if you pass that quiz and if you pass that interview you make straight to multiple on site interviews and if you take a job you get an additional one thousand signing bonus from triple bite because you use the link triple bite dot com slash. S. E. Daily. That one thousand dollars is nice but you might be making much more. Since those multiple on site interviews would put you in a great position to potentially get multiple offers. And then you could figure out what your salary actually should be. Triple Bite does not look at candidates backgrounds like resumes and where they've worked and where they went to school triple bite only cares about whether someone can code so I'm a huge fan of that aspect of their model. This means they work with lots of people from non and unusual backgrounds to get started. Just go triple by DOT COM slash. Save daily and take a quiz to get started. There's very little risk and you might find yourself in a great position getting multiple on site interviews from just one quiz and a triple bite interview. Go to triple by DOT com slash daily to try it out. Thank you to triple bite now. You as a routing layer that wants to be able to make decisions between these different providers you need to be able to have access to the rate of response for you and for San Grid and for Facebook Messenger and for slack do those. Api is like if you want to integrate with all those different players do they all expose the analytics information that you need to build currier. Yeah the answer is sometimes yes in and other places what we've had to do layer our own tracking infrastructure and analytics infrastructure on top of those platforms. Right so that we can drive that activity data into our own data stores that we can then use for a the intelligent routing in be the analytics that show you. Hey Okay for this notification these are the four channels that you're sending this inefficacious to which ones performing best right as well as for Jeff. Which one is the best specifically for him as a recipient? Wow so if I integrate with currier and I need to get data on how users are responding to a FACEBOOK MESSENGER. Ping you would probably have to give them. They're probably have to be some kind of redirect right. You'd have to have a currier and then have a redirect to the facebook messenger link right. That's correct yeah. So we'll wrap that in a custom domain. That you you help provide right so we will wrap the euro that you're trying to send the user to with a tracking link that looks like your domain and then that will drive a tracking event into career. Who I see Ray Right. So let's start to get into the usage so like if I am using currier and I have. Let's say it's a media company like let's say it's the it's the software engineering daily APP. We've got mobile APPS and we released new podcast episodes. We release some articles and I want to drive engagement on those pieces of content into my APP. What is my experience for integrating with courier? Let's say let's have got you know for when a given user signs up I get their email address. I get their phone number and they all have the mobile APPS I've got. Let's say three delivery systems. What is my process of integrating? So there's two sides to it. One is the integrating into your products basic calling or API. The second side is setting up the notification templates and that's actually done inside of our user experience so that's a full drag and drop experience. Where in theory? The engineer doesn't have to be the one to do it now. Maybe for you you'd be doing both but in large organizations that might be the product manager or the growth team member or maybe even a product marketer. They're the ones setting up the notification templates for all these channels using drag and drop without having to write any code and the engineer. All they have to do is is publishing event to career. Api so the typical walk through of of how to get started is sign up when the career website. It's free to get started free forever. Plan you just click logging with Google logging with get hub. What you're then provided with is the ability to create your first notification. You create that notification you plug in at least one channel right. So I don't know if you send grid or male gun or who you're using for for email. Typically people add one of those providers. I but you could start with mobile push or you could start with SMS. It's up to you. You set up that first notification with one channel you add the tax you add the call to action button whatever else and branding and images that you want to be part of that template now the area you can kind of click a little button and it gives you a little coates says send this code snippet you know. Call this to call the courier. Api We have some libraries for you know which back injury using we can. We can do curl Joss Ruby. The pure rest. Api So pretty easy to access you. Then take that code. Snippet you pop it into your Code Base. And now it'll start flying through gotTa do one other thing though and that's either when you make that call to careers. Api To say delivered this notification. You can pass. Troy's contact information along with that or you can separately send that over to courier. We'll store that in a profile database. And you only have to send that to us wants right so you have to set up each of the user's profiles within currier because you need to associate the user. Id with all of the different delivery mechanisms for that user. Exactly so what? We like to see our customers. Get to the point. Where maybe they have a hundred different notifications across their product that they send out we see typically tend to a hundred is Liz normal range of notification. Count for you know. Let's call it. The equivalent of an early stage to mid stage product. Competition does a ten to one hundred different types of notifications. They might completely different templates right. So forgot password. Somebody friend did you. Somebody followed you. You know you've purchased something. Whatever the case of a media company right. A new article is posted. New podcast was posted. A new comment was left right. And all those account based ones as well as you mentioned the forgot passwords in such billing. That's right so what we like to see is that our customers shouldn't have to go out and when they're sending that notification to our API we don't want them to have to go query their user database and pull all of this information together themselves. You know what choice. Sms what's his email. What's his mobile push token? We think it's easier for them to pull that once. Send that over to the courier. Api just once now. Whenever they sent him a vacation they can just say deliver this notification to Troyes user. Id and we'll take care of the rest interesting and as far as sentiment about how the users interacting with those different notification types is there anything other than they opened it or do you try to get more complex like they opened it and then they engage. They opened it then. They bought something. How deeply do you get into the sentiment of the response to notification? That's a great question. Today it's really the click through tracking and saying did they actually click on the call to action engage get driven back into the experience. That's really as far as career goes one of the things we're working on right now is actually a segment. Integration in that segment will be bidirectional so that could then take that event. Push it into a segment. Have that then. Route to say mixed panel or amplitude or heap or another system where you want to track engagement across your product and then you can start to do analysis to say okay. Once a user has engaged in this notification did they then go through the next four steps to say complete the purchase of their abandoned shopping cart to get an engineering. Can you give me an overview of your stack and maybe some of the key technological decisions? You've got to make yes. We Are Java script shop. We're a NO J. US based back end. I'm a longtime no jazz developer on the Express Jazz core team or teams about five engineers. Today all of us have long backgrounds and no Jessen react and so that's that's our stack were hosted on. Aws a fully service stack so. Everything's running aps lambda so that we are. We've learned a lot from our past again in the marketing arm side of the world and one of the things that you have to be capable of is rapidly provisioning for scale at kind of an lumpy lumpy kind of timelines right so every early startup founder dreams of having to support scale right. When you're in the notification space you have to support scale immediately because just one customer you know they might not send that many notifications if you were to spread them out evenly over time. But that's not the way this actually works. What will often happen is they will send a large majority or a large fraction of their notifications kind of all at once. And so you either have to keep a ton of you know idle servers just sitting there or you can leverage some of these more function as a service kind of platforms to dynamically spin up capacity as needed. There's a lot more work that goes into it than just using Rawal Lambda. But it's an amazing start and it kind of product is is what we had to do the marketing automation side of the world. For a long time. That fundamentally boil down to just having idle servers the issue with building courier on lambda. If we were talking for five years ago would have been. Is this thing reliable? Is this thing going to start up fast enough? Is this thing going to respond to a call fast enough? My understanding is that lambda has gotten much more reliable. The cold start is much less of an issue but notification routing is really time sensitive like you needed to be as responsive as possible. Is the cold. Start an issue at all for you Linda. So cold start is still. You're right that it's improved significantly and I know. They have other improvements coming soon but it is still a thing. It's still not as good as just having idle servers. Standing by there are few things that you can do to mitigate that one is we. We fundamentally have an asynchronous. Api So our API's calling other API's and those API's might fail in which case we need to Req- and retry delivery. What that means is our customers are calling the career. Upi and then moving on they don't wait for us to successfully deliver that notification. Because I could take a long time right. Some of them might be scheduled to go out tomorrow. You don't want your server waiting for that to happen so one we benefit from that too. We put a layer in front that all it does is. It's job is to be as responsive as possible to your request to queue up right so we make sure that there's no cold start time for you calling. Api You call the API. It's always responsive and then we kill it and then those queued messages are picked up by Lambda functions which might have a cold start time. The third thing is and this is relatively recently released by. Aws is you can set a provision capacity with lambda. And what that means is you can get back to the. I just have idle servers sitting around world where you can say. Here's a minimum threshold of floor. If you will of lamb does that I just want just warmed up sitting there. Now you're paying for those right so you lose the hypothetical benefit of not having to pay for that capacity but instead of scale zero. It's like skilled at two one. Exactly exactly that's cool. So do you have any idea how that scale to one or skill the two compares to like the pricing of having a single like far gate instance running? Yeah I mean. It's it's pretty comparable when you look at costs basically by CPU hours right fundamentally you know this is all more or less running on same infra. So the container. Yeah the pricings pretty comparable the nice thing is you can far gates still lumpia right because you're you're spinning up you know real full servers. They're all virtual of course but it's not just a function it's a real lesson you can do some really meaningful things which is going to impact the times for far gate and just some of the costs of how much memory you're going to need. How much sleep you you're going to need. We've specifically about courier. Each of the we have really two sides to our back end. One is our delivery pipeline from when you call. Api to win. We've delivered that notification to our downstream providers like Williamson Grid and then the second back end is the run the design environment in and that's a more typical web application in that's for the users that are logged into career and are dragging and dropping contents around on the template editor for the delivery pipeline. We are really really specific and really purposeful about keeping each step of that as small as possible and a small function as possible with as few dependencies as possible so that the spin up time the boot time and the tear down time for each of those is minimal so that we can keep the latency from when you call career to win. We deliver it to partner. We want the partner to be the biggest part of that in terms of of time. Not currier just going back to what you said about far gate. I lambda I know like some of the details like black boxed to us by aws but do you know more about like. You're saying that that far gate is Kinda thicker or a bigger instance when you summit compared to lambda. I thought that in both cases you're effectively getting a container. Yes but one is a thicker container in that. Let's say forget you are actually going to. You can do some of this with layers today in in lambda in far gate. It's a full as you can install large software pieces in you could install engine on it. You can stall whatever you want on. It's it's basically it's the equivalent to EC two right it's just it's dynamically provision. Ec Two instances fundamentally whereas obviously for lambda its function as a service. So it's it's lighter weight yours you have more limitations in what you can do. Yes you could. Use a layer to install software. You know a server. That's running behind the scenes or something like let's say image magic. That's pretty typical use case. Because you WANNA DO IMAGERY SIZING. You can do that with layers. And sometimes that's the right choice. But you're really if you're building lambda functions the right way that image magic layers only applied to one function right and that's not applied as a layer to the other functions in your chain whereas you would be incentivized with far gate to probably follow more traditional Web Server Rollout where that Web server has all the things on it. More of a monolith like deployments to those Fargo instances. But you could then use far gate to dynamically scale horizontally C. B. Said Far Gate is like dynamically provisioned. Ec Two. I thought it was like catheters containers. Ec Two is is so so it is containers so you can plug far gate into you know. E- chaos which is Amazon's hosted version of Cobra. Nowadays you can plug it into EC was acs which is their worst version of puberty is proprietary. Yeah so it is. It is a container right. But it's just like docker container. You know thinking about the the best practices of docker containers. It's really keep those then as possible right because the more layers of that container you add the more things that are running the background. The longer the spin up time for that container will be so. So you're right that it's all kind of turtles all the way down all containers. But it's I think really the use cases that they are meant for and the way that that I've seen far gate positioned is as the successor. Tc too so you. Don't you think about like Kuban? Eddie's as control plane for for servers for full servers that are essentially more or less permanently provisions. Obviously like one of the things it does is it will tear them down and spend the mopping needs. Make sure that it feels like twelve factor. That doesn't like you don't just write your data to disk and expected to still be there tomorrow but they're full servers. That are meant to sit there and wait for responses. It's not meant for invented architecture where the event happens the things up. It does the thing and spins back down which is really the function as a service lambda use case. And so I think that what that leads to while they're both similar in terms of technology when you're using far gate your you'd be likely to install in genetics and other things on a foggy instance actually run a real traditional server there whereas in. Lambda you are abstracting the web server side to API gateway and you're just receiving Jason. John David Structure in is your input doing your thing and sending Jason as output in expecting that container the spend down right but from what we know about the scale to wonder skill to whatever you call it minimal provisioning or what is it called for lambda. I think it's capacity. Yeah so do we know anything about what the the medium of runtime is a container or is it like it is like a far gate instance or are we. Just don't know I I I don't know okay the rule we might have raw goat's right. Okay and then so does that. Give the degree of reliability. You need to be hyper responsive or are you saying that. Api Gateway is your highly reliable fronting combination of API gateway plus the layer behind API gateway. That's responding to those events in queueing them up right in. The lambda stuff takes over once. Things have been queued. And that's what then sees the process through to the final ultimate delivery to the downstream partner provider. And what do you use for the queuing us? Okay all right so hits. Api Gateway then gets cute on us and that cueing action triggers lambda. And I guess do you have like different lampposts for the different providers whether it's like a Lambda Lambda dedicated to send grid? Or how does that work? That's right yeah so you can. You can bind lambda functions to Q. Q.'s so that as you pop messages onto that cue. The lambda can just rip that off and so when our land will function runs it's basically effectively being passed in that message by the infrastructure processes that message potentially puts it on another queue depending on what step of the process. It's an or that's responsible for delivering to one of our integrations. So yes we so we have about twenty different integrations with different partners like slack and send grit in Tokyo and each of those are their own land function pulling off of Accu- provisioning a pulling the configuration data from that customers installation. So what are Your Api keys for Toledo right and then making that outbound? Call to Twi- Leo. Then we have to receive that response and deal intelligently with it so what happens if the tweet is down. Well depending on the scenario we might either. Req- that for later delivery or Req- that as a reroute to a different channel. So we take an example like let's say are ordered a grocery delivery and it's coming to my apartment and the grocery delivery service uses courier and the system has been configured such that since the grocery delivery is on. Its Way to my apartment. There's going to be a text message that needs to be sent to me. And so give me a sense of how that message is getting routed or first of all how that triggering you know how the grocery delivery service is having that triggered on their back in that there needs to be a text message sent or there needs to be a notification sent to this person and courier abstracts away what the notification is going to be whether it's going to text message or it's going to be an inapt notification or it's going to be an email the API of the grocery delivery service says okay notification needs to be sent to Jeff that notification hits your API. What happens next so the grocery delivery service calls our API to pass him an event ide- right and they might call that delivery notification might be Ryan without event which would be one of those ten to one hundred notification types. They have exactly they'd pass in Jeff's recipient. Id this typically be the user ID from their authentication system right so whether using zero or something else. They've got to use their I. D FOR JEFF. They pass that over to career and then they pass over. Whatever data is relevant to the so might be a list of the items that are being delivered the final price and maybe a estimated delivery time right so they passed that over just as Jason data structure to courier so now courier accused that up so we receive that we cue that up. The next up the happens is a career. Lambda function picks at a Q- pulls data about that customer. That tenant as what we would call it internally. We look at configuration for that tenant we look at configuration data for that template that matches that event. Actually have a whole event mapping systems. You can start to push events to career without notifications existing on the other side and then you can later come in at an invitation. Say whenever that event comes in the next time. I want you to send this notification of these channels so we have to look up the mapping to figure out okay for that delivery confirmation event which notification or notifications are going to map to it. We pull in the configuration and templates for that notification. That's all part of what we prepare stuff. We kind of bundle this up into a package that we then Req- the next step is what we call a route package. It calculates through looking at the user's profile data which we actually also pulled in the prior. Stop looking through the configuration. What what channels have been configured for this notification which of those channels are still active. We have the API keys and all the data we need and then it kind of builds out here are the possible destinations and go through the kind of privatization and rules. Engine to figure out. Which of those are we going to select? There are ways to configure. Curtis say to send multiple so we have to figure that out as well and then next up is basic queue that up for a rendering pass which is okay. We've decided based on this process. This notification is going to be delivered over SMS. So we basically boil down in that routing stuff to figure out okay it's GONNA go rest mass and so now it passes a message off onto a cue for this mess including the template that needs to be rendered and the data that needs popular that template we then have a per provider rendering engine. That can you know if we think about. What is the format for vacation? Let's html was format for an SMS notification let's Plain Tex was the format for slack notification. What slacks proprietary Jason Format called Microsoft teams has a proprietary Jason Format Etcetera? We have renderers and serialize for all of these providers so we take that same template data in render it for that specific channel. Then pass it off to basically to call downstream writer in this case. Let's say Twi- Leo for SMS assuming that the responds and says it was success. Then we're done and we can mark that as a success and wait for the engagement event to kind of update that record later if it fails then we kind of go back a few steps and we go back to the routing process to figure out whether we want to go back to Tokyo again or whether we want to go to a different channel. Okay cool so that's a pretty detailed explanation. You've given just to revisit the architecture you're fully server less architecture you've got to decide what data services to us so. I'm curious if you've got you know I'm sure stuff that you need to access really quickly. So maybe you're using Alaska cash or I mean maybe just walk me through the data services that you're using on wwl side. Yeah so you've probably got the impression already that we're pretty heavily. Invested in aws and features that accurate. That is very accurate. So Lynn or pretty all in. Yeah I know. A lot of companies are trying to build to be agnostic to the underlying hosting infrastructure. I think that that will be seen in a few years as the same kind of messed up. We made in the ninety S. Orum's we're trying to abstract away. Which database layer that. We were talking to what that generally meant is that you spent a lot of time building an inferior way to use the power and capabilities of the technology. And then you never actually switched anyway right. I think we'll see the same thing happening. With hosting infrastructure we can take a benefit and advantage of a lot of the key capabilities. And frankly you could for GP or your if you actually just commit an build for the infrastructure. So let's let's table that for. I'll go deeper there so on the persistence. Layer for that delivery pipeline. One of our goals is that we have basic maximum number of queries that were willing to call per function. Because again we're trying to reduce the latency throughout in every single one of those call is queries needs to be key value up. They can never actually do a scam. The can never do a multi-record query. We have to know exactly which key to pick up and we only do up to three calls per functioned peopling minimal. We've dynamo as the persistence engine. For All of those key value look ups. The data that we right into Dynamo is also streamed over to an elastic search cluster. We use that in our logging. Ui for actually quarrying and fasting data but all of the kind of run time things that happen when you call Api between then and when we actually deliver that message to Twi- Leo Justin Grid all of that is key value look ups and Dynamo and the latency is not an issue. I think we did a little demo before the past and when I click on send in our little. Api EDITOR LEASE INSOMNIA. But it's like postman. If we remember the slack use case the slack notification showed up before I was actually able to tab over to my slack. Claro true why is it so fast I like I mean I assume if you compare like Dynamo visit maybe gonNA sound like a naive question? Be Compared Dynamo. Db to read this. You know I would think that you would need something you know some kind of memory. Data access layer in order to get the kind of latency. You would need out of notification infrastructure. I think as long as you're restricting yourself to just pure key value ups you know. It's it's fundamentally Hash table and if you're just indexing directly into the exact record in the hash table database servers are really good at doing that right and so that is all you know in some ways we are using dynamo as effectively erectus infrastructure right. Because we're not writing queries into the underlying Jason Structures within these records. It's all key value. Look and we benefit so could we use Lhasa cash? Could we use the renaissance literature we could but by using dynamo we can do things like dynamically spin up lambda functions when certain rows are updated and we use that for example to than stream creation record creation and record changes over to elastic search? Right so again. By committing to that infrastructure able to benefit and tied in to all the other services and reduce our operational complexity in our engineering costs and so what using elastic search for elastic search is inquiries so anything. That's not a key value up where we are potentially returning multiple results. Those hit elastic search. Those are generally only happening within the user experience when you log in as a developer or administrator to see the list of templates and then listen notifications also see the logs so every time you send notification through courier we track that as a log you can log into cur can see those logs you can click into that law you can see. What did you send the courier? What'd career load for the profile? Why did career route it to a specific channel? What did that channel partner send back? So let's say you're called Twilla was failing currier would show that to you. In the dashboard you click into those notifications. And then you could see that. Twila responded back and said your API key is invalid so maybe somebody rotated the key and forgot to update the carrier installation or something else has happened. You'd be able to see that in those logs because we'd capture what Twi- Leo and what's in Grid and slack with a returning to you digital makes infrastructure simple. I continue to use digital ocean because of the low friction and attention to user. Experience Digital Ocean has kept the experience. Simple and I can spin up a server in less than a minute and get high quality performance for a low price for an application needs to scale digital ocean has CPU optimized droplets memory optimized droplets managed databases managed Cooper Netease and many more products digital ocean has the flexibility to choose the right instance for the right workload and you can mix and match different configurations of CPU and Ram if you get stuck digital ocean has thousands of high quality tutorials responsive QNA forums and a customer team who treats customers respectfully digital ocean. Lets developers focus on what they are building visit? Do dot co slash s daily and receive one hundred dollars in credit over sixty days. That one hundred dollars can be put towards hosting or infrastructure and that includes managed. Databases a managed. Coober Nettie Service and more. If you WANNA get started with coober netease digital ocean is a great place to go. You can use your hundred dollars to start building your distributed system and you can get that one hundred dollars in credit for free at. Do Dot Co Slash Sec daily. Thank you digital ocean for. Being a sponsor of software engineering daily are using Amazon elastic. Search service we are. What do you think of the whole idea of like that Amazon? Taking open source projects and product. Izing them and perhaps not contributing back to the open source project and perhaps taking away business from the companies that are funding. The open source project. You know as somebody who is advocating the all in aws perspective and somebody for whom elastic search service Amazon elastic search services. Good enough what are the cases where you know a company would need to go to elastic and get better support? I guess we can explore you to those topics sequentially. Yeah so first of all. I think that it's actually quite problematic. What's happening right so I would agree. I think that what we've seen are many of these companies. Go to the dual licensing model right and that's obviously created some controversy within the community. Because it's it's not a strict open source license anymore. Once you kind of go down that path but you know I think frankly Amazon has forced their hand and I think that either we need to go down that path or we have to abandon the idea that open source projects can generate product businesses. The Red Hat model could still work right. In terms of being a primarily professional services oriented business. Obviously we see a lot of open source. Projects go that path but when we look at a Mongo with that lesson we look at elastic with their own hosting infrastructure in many many other opens were servers are going out that path especially persistence servers. You know I think that they probably just don't have a choice but to go down the the dual license path. Because obviously they're not gonNA say oh well we should just abandon our our business model and so the open source project is is really a considerable enough set of functionality. And it's easy enough for Amazon to spin it up and offer the basic level of you. I and basic level of services on top of it such that you think it forces a company like elastic to go the licensing model rather than what if they built additional services on top of elastic additional product? Offerings on top of elastic better search experiences vertical research experiences or in the case of Mongo. Like we just did a show with the web flow former. Cto and he was saying that you know Mongo is so crucial to them and they really want the best Mongo experience in and Atlas provides enough of a differentiated Mongo experienced that. They're not GONNA GO WITH DOCUMENT DB. They're not gonNA go the off the shelf thing even though they're big fans of aws so it's surprising to hear that perspective. Well I mean you know. We've seen red us. Go down the path of trying to add additional capabilities on top and separately license those capabilities. We saw a fair amount of pushback from the community there as well. I think there's definitely a pick your poison in terms of. How are we going to get out of this bind where we want to make sure that project owners of open source projects are economic incentivized to continue working on it? And the only way you know yet you've listed a few different options. I think that fundamentally adding new capabilities top and licensing those separate isn't to mean not substantially different than just a dual licensing model. And I think that we in terms of who it hurts more. I think the dual licensing model hurts Amazon. Right hurts people who would want to host these services and be paid for it instead of the company that created the Software. I think the separate licensing for new features on top of the open source platform which is more of the rattus model. I think that hurts. More of the consumers right it hurts the companies that actually went to use and benefit from red us as as users developers. Because now it's okay. Well you're kind of forced me into do. I want the free version or do I want this. This premium version. That feels to me less than the spirit of source. I do think you've illustrated one other possibility. Which is maybe. You're just better at hosting than Amazon. Which is really an operational excellence challenge and I think is probably a fairly separate skill set than building an amazing set of of software right. And if you can do it well and it sounds like maybe the team over Mongo was with atlas. Is doing it well enough to succeed there than maybe that is a great model in those use cases. But I don't know that as a community. We want to say not only need to be a software engineer that is world class at creating a different shaded. Persistence layer and soffer service. But now you also need to be rolled class better than Amazon better than Microsoft better than Google at being able to host that and spin up capacity and provision servers. Which is you know. Relatively lower layer you islay alpert experience fairly orthogonal as concepts so more power to them if they can do it well but I don't know that we wanna put all of our eggs in that basket so in other words you are basically saying that the dual licensing model is the most viable path forward for a lot of these kinds of businesses. Well I'm biased. It's more lines to my incentives as a startup founder. Who wants to utilize this technology right? The dual licensing model either forces the Amazon's make stuff and googles of the world to pony up and actually kick back money to the creators of that software or to build their own competitors slash the creator actually hosting it themselves. So we'll see what path goes down but it feels like a preferable path from my perspective but I'm I'm just consumer of all of this. What you said about the all in on. Aws Perspective are onery. Visit that I. I'm with you there and I think that what you're suggesting. Maybe it alludes to the fact that we perhaps have a hangover from. I don't know what the motivations were for. Oram's but I assume it had something to do with like Oracle and Microsoft and just the sense that lock in is pervasive problem. That we can't get past you know that's hangover from a time when there was not really a check on on. Microsoft's power or power and there really was not alternatives. I mean design extent if the market goes in the direction of going all in on. Aws It would put us in a position where there's not you know people who are all in on the aws ecosystem you know. There's no check their and they're totally locked in. And then just a matter of like do we trust. Aws enough not to pull the rug out from under US and increased prices a lot but arguably we should trust aws not to do that because it's so ingrained. In the culture of Amazon to consistently lower prices just expand horizontally and strategies worked so well for them. Why on Earth would they undermine their culture for the sake of a quick buck and still such a competitive space? And you know everybody's competing on price right. It's relatively commoditised. And I think that as long as Microsoft and Google or in the game. I don't think Amazon Ken do that. We've seen Microsoft azure grow by leaps and bounds in terms of. You know they've clearly surpassed. Google now is the number two to Amazon. I think there's still plenty of competition in the market. That Amazon's not incentivized to Jack. The prices and it's just kind of like the historical trend lines. Your point is that's not the direction it goes it goes it goes the other way. I'm not a soothsayer I don't. I don't know what will happen in three years. Maybe the prices will start to go up. Doesn't seem likely though and I think that's like true of just computing general computing capacity. Cpu capacity. Moore's law might not be an effect anymore but it's still trending towards it gets cheaper over time. I expect that same to be true for for hosting and you know your point is also just accounting for the fact that there's just no free lunch you cannot insulate yourself against platform risk. If you try to do that you're gonNA fall behind the people who are not insulating themselves from platform risk and you're going to seal your fate regardless you're paying. It's a tradeoff right. You're paying cost now for a future possible savings but it's a very real cost that you're paying out you're going to spend more time and you're going to have a worse result today and it may benefit you someday down the road and I'm skeptical that it would. Are there any more knee? Sh-she aws based decisions or aws products that you're using that are worth highlighting saute one. That's relatively niece's cognitive so use cognitive oh as authentication store so all of our user accounts are stored incognito. We support email password. We support get hub aloft. We support Google off. And all of that is controlled by through COGNITO. It's been okay. I'm not in love with it. We do love the fact that by being all in including the authentication layer on. Aws All of this is automated with as code. We can spin up entirely new production. Stacks from into end takes about ten minutes. Every developer has their own complete end to end. Stack it literally you click a button. Boom it's up wound spin it down spin up as many times as you want. That has been nice and there's effectively. No configuration click a button. Boom there you go is that cloud formation template reading service DOT COM really is cloud formation under the and so you're using the service framework. Are you paid customer service? We're not currently paid customer service so we use the open source library. I think the paid service is they have a hosted offering which we do not use. The nails have a support offering which we might use at some point. And what else do you use outside at the ABC go system? Let's see so. In terms of additional tools. We like many companies use ENTERCOM. We have our own set of of notifications that we send through courier to downstream providers like slack and send grid and Expo. So we use those services. But we won't count those for now. Let's see so we're we're actually about to add. We want to add some more. We have a new you. I experienced coming out soon and we WANNA be able to see the customer journey as they test that we use segment. We're also thinking about adding something like full story. During our white commoner class. There was a company called Essay. Or we want to check them out as well. It's kind of like a full story but targeted engineers. So we're going to test down a few. Those options log rocket. I think is in the space. So we'll take a look at a few of the data dog US get home Blake. Yeah the full story log rocket sector those tool seem so useful but I wonder if I mean it's kind of invasive right like it's like you're videotaping entire browser session like we sure. Do I want this behavior? Normalized like yeah. Do you feel conflicted at all about I do. And you know like as a call back to earlier. It's a bit of a tragedy of the Commons situation. Where do I want to be the only one that doesn't benefit from this? I would also and perhaps this is just rationalizing but I think you know we are a social media company. You're not writing really private stuff like these are templates. You're not even like writing user information in user experience. We obviously have. We can see your lights anyway. So I I like to think that our customers would be more okay with US doing it than say facebook. Probably they would all prefer that nobody can well full story previous sponsor Kelly. They should still sponsor daily. That's definitely a lot of potential customers on here but let's talk a little bit about the future. So how do you see this notification market evolving over the next five ten years both in terms of like new platforms and like developer? Tooling developer usage. Yes I think one way to answer that is that we are seeing rapid growth of new channels and that growth is only continuing to increase and so as an example of this a year ago. Microsoft teams didn't even make my pitch deck in terms of channels and today it's one of the most commonly requested channels because of the growth of teams as a way to reach Non San Francisco companies. Like everybody's got office three sixty five so we see massive teams. Ms Teams growth in the mid west end and outside of Silicon Valley. So if you're a B. Two B. Company being able to reach into that kind of always on presence at your users are sitting in all day and they might not be sitting in your product. All Day is really high value. So that's a channel that kind of came from nowhere is now frequently asked for in education space for B. Two B. So new channels will continue to pop up so I didn't see that one coming. I'm sure that there will be others that I don't see yet we've got you know what's APP is still trying to figure out what they're going to do here. It's really nearly impossible to get even approved further. What's at business license? You can fast track that by applying through Tulio and some other ways but even once you do you have to have. Whatsapp approve every template that you want to send and it's incredibly limited in what they're willing to support the general kind of thesis is they don't actually want to be in the notification game and they don't know whether they're going to stay there or not so we'll see we'll see where that goes basic messenger is somewhat limited but is much further long so we'll see some of these channels we chat line especially as we look outside of the US. Some of these are now starting to focus on vacations line. Notifications from media companies is one of their primary drivers of revenue so like The New York Times and many other media publications send their notifications in Southeast Asia. Overlying that's how they communicate with their audiences because because nobody has the APP. Well we forget that outside of the US right like in the US. We start with emails notification channel because emails ubiquitous. That's not true elsewhere. Wow and so you have to start to think about well. What channels are ubiquitous? And oftentimes answer is none. And now you're in this weird you remember that Carnival game where you'd have all these circles where you had like fit the circles on top of this bigger circle so that you've perfectly overlapped that circle you're basically like creating this vin diagram to cover up a circle. I feel like that's what's having to happen. You have to happen with these communication channels especially outside the US where it's like. I have to piecemeal all of these different channels together. Because there's no one channel that I can use to reach all of my audience so we'll see a lot of growth outside of the US overtime. We'll also see more sophisticated orchestration of notification strategies so as an example here. I described earlier. What which is strategy is in terms of in-app mobile push email slack has almost the exact same one about three years ago. One of their employees Matt. How posted a diagram to slack of their exact flow chart for how they decide whether to send a notification to you in on which tire ever. That's a complicated. Yeah it was. It was a funny one but right because Matt how he was like. Hey you know whenever I see somebody on Hacker News Say. I could build that in a weekend. It's like no man. You couldn't even build our notifications and so I've spent some time with the notifications team at slack. There's again about eight people that work full-time on notifications all day. Every day. Lord when you become a big successful tech company you're going to have a team that's building out the internal version of courier. They are working on some really sophisticated notification infrastructure and doing really cool things that I think are out on the bleeding edge and when we talked to other companies like slack companies are looking at things like doing multi channel distribution and then retracting from the channels. You didn't engage in right so by pushed you onto channels and I wait for you. Engage with one of them. I retract the others. So it's like it never came right. How can we optimize for speed of delivery and likelihood of engaging while minimizing the impact? Distraction that you might have in. Frankly your annoyance right. This is an area where I think user preferences are well aligned with what will benefit the business right because they do not want to annoy you because you have the power. To unsubscribe so it's kind of like. How do I get on that knife edge of making sure you say as informed as possible you keep us in mind you can keep coming back into our experience but I never cross over that into that realm of annoyance where you might just cut us off last question? What would you build if you're not building currier? Oh that's a good question. What I really love building tools that make it easier for people to start businesses and start products. I think I waited too long in my career to create developer tools. And I've definitely been bitten by the bug. It's when I started to think about what I was going to create. Next courier was the top of my list and one of the reasons why is because I was like built things for marketing teams and logistics teams and political campaign teams. I'm an audience. I buy stuff. Why have I never built something for myself? So I think it would be definitely something else in the developer space and something that makes it easier for young engineers young startup founders to go from you know the proverbial zero to one. How do we get that idea out into the market with out having as much overhead of things I have to build? That aren't core to the idea. And so I think career fits within that model. If it wasn't courier I'd be looking for another idea that achieve the same effect. Try thanks for coming on the show. It's been great talking. Yeah it's been great job. Thank you Apache. Cassandra is an open source distributed database. That was first created to meet the scale ability and availability needs of facebook Amazon and Google in previous episodes of software engineering daily. We have covered CASSANDRA's architecture and its benefits. And we're happy to have data stacks the largest contributor to the CASSANDRA project since day. One is sponsor of software engineering daily data. Stacks provides data stacks enterprise a powerful distribution of CASSANDRA created by the team that has contributed the most to Cassandra. Data's decks enterprise enables teams to develop faster scale further achieve operational simplicity ensure enterprise security and run mixed workloads that work with the latest graph search and analytics technology all running across hybrid and multi cloud infrastructure more than four hundred companies including Cisco Capital One and Ebay run data stacks to modernize their database infrastructure improve scale ability and security and deliver on projects such as customer analytics. Iot AND E-COMMERCE to learn more about Apache CASSANDRA and data stacks. Enterprise go to data stacks DOT COM slash S. E. Daily. That's data stacks with an X. D. A. A. S. T. A. X. At data stacks DOT COM Slash Se. Daily thank you to data stacks for being a sponsor of software engineering daily. It's a great honor to have data stacks as a sponsor and you can go to data stacks dot com slash s daily to learn more.