Linux Kernel Development with Shuah Khan
An operating system. Kernel manages the system resources that are needed to run applicastions. The Lennox Colonel runs most of the smart devices that we interact with and it's the largest open source project in history shoe, a con has worked on operating systems for two decades, including thirteen years at HP and five years at Samsung, she's worked on proprietary operating systems and a variety of Lennox operating system environments, including mobile devices, Shula joins the show to discuss her work within Lennox and her experience contributing to open source Shula has made significant contributions to k- self test. A set of tests for Lennox testing, the links Colonel is complicated because there's so much depth to the code base and such a variety of ways that Lennox can be used. There's also variety of ways that the operating system can get tested their smoke testing performance testing and regression test. Sitting there are trees of tests. And as a developer, you may only want to run a subset of the tests. The conversation with short it from the low level practices of testing, the Colonel to a high level discussion of how the Lennox Colonel can reveal dynamics of human nature. It was great having shoe on the show. I really enjoyed the conversation, and it was at the open source leadership summit put on by the Lennox foundation. So thank you to the lenox foundation for inviting me to that before we get started all mentioned to events that we are having in the near future. One is the software engineering daily meet up at cloudflare, you can go to software engineering daily dot com slash meet up to find out more. It's April third, and it will be a conversation with Seib Karachi, a crypto currency investor and friend of mine, and he's been an engineer at Airbnb and many other places. He's really a great guy and interesting to talk to so to find out about that event. You can go to software engineering. Daily dot com slash meet up and the other event is the hack on you can go to software engineering daily dot com slash hack. Thon. This Kathan is for a product. I'm building called find ca labs it's a product for finding people to work with on your projects. We're having an in-person Hackworth on at APPA kademi. And we're going to hang out. We're going to have some food. It's going to be on Saturday, April sixth, and you can find out more details at software engineering daily dot com slash Hackworth on. I would love to see you there. If you're in the city or in the bay area and so software engineer dot com slash meet up software engineer daily dot com slash Hackworth on if you're interested in either of those Nellis get on with the show. Digital ocean is a reliable easy to use cloud provider. I've used digital ocean for years whenever I want to get an application off the ground quickly. And I've always loved the focus on user experience, the great documentation and the simple user interface. More and more people are finding out about digital ocean and realizing that digital ocean is perfect for their application workloads. This year digital ocean is making that even easier with new node types, a fifteen dollar flexible droplet that can mix and match different configurations of CPU and ram to get the perfect amount of resources for your application. There are also CPU optimized droplets perfect for highly active front end servers or CIC de workloads and running on the cloud can get expensive. Which is why digital? Ocean. Makes it easy to choose the right size instance, and the prices on standard instances have gone down to you can check out all their new deals by going to DO dot CO slash SE daily. And as a bonus to our listeners, you will get a hundred dollars in credit to us over sixty days. That's a lot of money to experiment with you can make one hundred dollars. Go pretty far on digital ocean. You can use the credit for hosting or infrastructure and that includes load balancers object storage digitalization spaces is a great new product that provides object storage. And of course, computation get your free one hundred dollar credit at DO dot CO slash S, E daily and thanks to digital ocean. For being a sponsor, the co founder of digital ocean. Moisy Eretz KE was one of the first people I interviewed and his interview is really inspirational for me. So I've always thought of digital ocean as a pretty inspir-. Company. So thank you digital ocean. Hugh, Akon you are a Lennox core. Commissioner welcome to software engineering daily. Thank you. I wanna talk to you today about operating system. Kernels. Let's start by just explaining what is the role of an operating system. Kernel operating system is you have a piece of hardwood laptop, for example, in everything that a human needs to do on that operating system provides dry woods and unquote cardinal. You have memory in the laptop and how do you initialized the empty? How do you set up the hardware? How do you get the dry woods, different pieces of hardware to state working? For example, audio on the laptop awed you screen all of that is what operating system going to do foot you. It's essentially making the. Hardware. Usable footer you to do for the user to do what they need to do with that piece of Harvard. You've spent a lot of time on the testing process for the lenox Colonel when you're talking about these different elements of an operating system like media playing for example, how do you test something like that? That is a good question. I do contribute to Lennox media subsystem. And so how do we tell you how you have a dry with apart? Obviously the media drivers. And then you use various user space up legations in terms of to do the testing the different levels of testing. We do like, for example. We would just to do does it boot. For one thing the sofa slow of testing and operating system does the cuddle would come up to us a prompt. Can you log in can you start up legations rows of since? Oh on so say, you are talking about a particular media. Would you have a USB stick that does digital TV your and log TV and so on so he he connect not as a dozen pilot up with that. And then after that you go looking for actually starting in upper Gatien dot. Would start streaming perhaps and see audio streaming can stream can you switch between channels and so on. So does the different levels of dusting? So to get to that level. You have to go to different level layers of verifying, different pieces work could actually, and we also do some drivers datum would video drive would Soden TV distill mediate. I was they also tend to do complaints testing. That means is the highest level obligation. That is using the API Conally PI. Do they use it correctly? So there are tools that kind do compliance hosting dusting is a launch area, depending on what you're planning to do. So as a double up, but if I'm developing fixing a bug ought to developing new feature I have to learn to understand what are the pieces that I need to use to test that particular part. And cardinal self test itself. His addition estimate. Well, what do they we have multi Dulles as couldn't tell us? We come up with test and say, hey, this is the test. I want to run then I am taking a new patch on I want to be able to just quickly apply. The patch would the cuddle and run this test undis- shells. It could be a show script a C program. Just run it to make sure that the patch doesn't do any ingredients. And in some cases, you have to verify the new feature itself is doing what it says. It does. Right. So that those two levels of dusting so cuddles south test is a suite of doper tests. So to speak we go in do that kind of dusting. Is there a sense of continuous integration or continuous testing? In Los Colonel, I'm much more familiar with web development. And you have continuous delivery in in web development. How does that compare to operating system development? We do do that in the Lennox cuddle space. What happens is we how does strings that continuously teak patches coming in and run radius kinds of tests like for example, just to completion test. It could be Khanal has civil configurations. We have radius configuration options that can be turned on for example. You might have a patch that compiles just fine. Deduct Dulles compiles it and tests with it. And then once it hits the integration ring, it might fail to comply on some odd conviction out of some particular configuration, for example, I three eighty six it could fail on that. So we find those we have several gets master Linda's master, get the. On Conan dog then we have Lennox next, which is we call the continuous integration. So we're doing continuous integration of single day. We have an lyrics next release that we dollops us to test stuff continuously. So does that help you understand does? Yeah. You mentioned this tool case self test. If I'm a kernel developer, what is my usage for K self test. It is a test suite. So not so much test tool. So if you ought to cardinal della put what you would do is you'll use the cardinal repo, and you can kick off a it works from the cardinal make while mainly quite you will run you'll check out your cardinal sources. And then you'll compile the cardinal and say makes that case self test. What that does is it will build all the tests that we have and it will run through all of them. And then give you results for that. So the way I I use it. And also a lot of developers use it is they will use to run the entire test suite or they go into particular subsystem, and then run just those subsystem tests like, for example, somebody that it made it change to the cardinal space. They would just go on that particular subsystem. Can run into Joel subsystems tests as well using? There's multiple knobs that you can use. So you can go look at once this subsystem and say, hey, I want to just run tests in that subsystem or I want to just run in this on this particular route. So that's the kind of knob stay us. Then in some cases, it could be installed on any target does system in it can test can be drawn on the target, for example, cuddle Leonardo test forms. They used that. They take cardinal self-testing. They build it on the document system. And then they take it over to targeted system, and they run it right after they booed the cardinal. How does it fit into my workflow as a developer? Like, I am. I maybe I'm tinkering on Shum, I level element. And then what I want to run K self test on some lower level elements that are underneath that. How am I using this suite of tests, tactically? But so depends on who you are. Right. So if you had a. Cardinal deliver. And you made it couldn't change you want to find that cultural change before you send the patch up in that particular case, you will go you will make sure that the on the cardinal cell test, either particular to your area of the change you made and make sure that nothing broke as a user say, you are you have you just install the new cardinal, and you wanted to make sure that you could Noel nothing to rest on your side of things. So you could run it as a destroyed off a user or a ember a Delaware. You could go. To some eminent plot femmes, they'll use older revisions of the cardinal like, for example, releases long-term lease. They're not always on the main line Colonel. So they if they want to redefine by nothing, but will. Brutal street could act as users would use the whole suite. In the subset. When when I wanna use the subset this upset if you tweet tweets to something God something owed added a new feature. So most of the cardinal, Bruce. What would we do is development process? Then we knew feature we would in some cases, we would write new tests to go with that new feature. And then we make sure that the existing tests didn't as as we made this new change to the cardinal that we did not break anything. That's the regression part of it, which we already have in the cuddle Celtics. We add new test and say, hey from now on we want to test this feature. I we wanted to this feature that does it work the way to expect it to work and then going forward as other changes keep coming in. Does this fiction feature continues to work the way we want to work just to give the listeners some perspective for why testing is such a big deal for Lennox Lennox is, of course, running in extremely sensitive environments. You have mission critical things. That are life and death. And it's really important that bugs don't sneak in to lower level. Aspects of the code lungs is also very complicated. There's many different kinds of Lennox. There's different kinds of devices that run different versions of Lennox. And because of that the amount of tests and the different sub trees of testing types can get quite detailed. Can you describe the I guess the testing tree or how the tree of different tests across Lennox's managed, the thin the cardinal. We have cuddles helped us that's really one single place that we have us, right? Use it with us that can be run into jewel, dry woods and subsystems. They could have some tell softest the kickoff and run. Like, for example, you will have a test code driver. For example, a subsystem might expose certain test teen sort of interfaces to just testing programs. So you would the test of karnal softest tests used that to exercise the kernel, of course, cutting testing is very important as the part of rally dating qualifying cardinal is really important because it is the foundation piece in any of the lyrics ecosystem you want to help those bits. So it it. So we have various methods to in the cardinal to verify things like, for example, we have if I am writing ape changing driver, for example. I would go torn on configuration options in the cardinal that make sure that my locking is seen. That means I'm not locking here and leaving the law leaving that lock not unlocking. It those lock and unlock are balanced. So I can I can go into the cardinal and say I want to turn on those options. So which which which is I do so and then in other cases, they we have other DiBa GOP shins that will go and say look at case on for example, are just stunned as Asian so we can turn dot on. And then it'll go make sure it's these are all not something that you would turn on in the field. Right. But you would do it during the Dell of -ment or you do it as validation mechanism. Before saying everything looks good. So you would go and say, hey, I'm going to turn on a DOE sanitizer. And tell me if I have any memory that. I am using after free or. Any kind of memory issues? I might have. That's one example. So we have various tools within the kernel configuration options compile time configuration options that we can turn on and debugged. So we use all of these mechanisms. And in addition, we in some cases, we have tracing we can turn on tracing on Chrissy events. Like, for example. You are you have a host. And then you are you have a VM running. So you say, hey, I want to take this device and assign it to the VM. So we can if we are interested in looking at, hey, how this process is working is it correctly working so we can go and turn on Tracy events. So during run time and then turn them off. If we want tracing is an interesting one because. Done. Some shows about in distributed, tracing I'm assuming tracing somewhat just non-distribution raising somewhat similar where it's measuring the late and sees through the different paths that it's that a call might be taking or measuring at least that the call is is making its way through the different lower levels. I guess maybe define what tracing in this particular case, these Tracy events if for us to get a feel for how it particular user action might be taking paths it's similar to that. It's kind of taking baths in the Colonel. And we want to make sure we are facing the path. It's taking it could be used in two ways. It could also be used how fast something happens performance racing. And the other aspect is is it taking the right paths in the kernel that we expect you to take right? So as a debugging tool, so those are. Two different uses of that. I think what I'm what I'm getting at is that we have multiple ways to debugged and test the Colonel, and then we use depending on the depending on what we are doing as Colonel downpours. We all we do we use all of those avenues to test the cardinal doing dveelopment, integration and debugging. So if I had some change, and then I ran a performance trace on it. And there was a small increase in latency due to that change. How would you know, if that performance penalty is is like a bug or is something that would be considered a regression because sometimes it's tolerable increase in latency? It's not a big deal. Nobody's ever going to recognize it is there some judgment that's involved in that process. Yes. Yes. Depending on how bad the performance impact is. And why that? Whites introduced you have to fill the bay. I go go about it is I have to I root costs and say, why did why am I seeing the change? So what happened is that change can the change be explained? Or is it is it avoidable. Maybe that is something in the path. I introduced something that that I might have an alternate mechanism. That does not introduce the performance penalty. It all boils down to route causing in analyzing is it absolutely necessary to do. So and in which case the coordin- needs to be read it on. Right. I mean, you know, there might be a better way to do that peace accord or and so. That is that it depends on what kind of performance impact and how it can be solved. It also depends on the nature of it. HP? One view is a foundation for building a software defined data center. HP one view integrates compute storage and networking resources across your data center and leverage is a unified API to enable IT to manage infrastructure as code deploy infrastructure faster, simplify life cycle maintenance for your servers. Give IT the ability to deliver infrastructure to developers as a service like the public cloud. Go to software engineering daily dot com slash HP to learn about how HP one view can improve your infrastructure operations HP. One view has easy integrations with Tara form coober, netties Docker and more than thirty other infrastructure management tools. HP one view was recently named as CR ends enterprise software product of the year two. Learn more about how HP one view can help you simplify your hybrid operations. Go to software engineering daily dot com slash HP. To learn more and support software engineering daily. Thanks to HP for being a sponsor of software. Engineering daily. We appreciate the support. What are some other judgment calls? You have to make. So, you know, latency might be one like, okay. Is this latency go actually going to matter? Like, let's say you do a root cause analysis you find that. Okay. The root cause is actually like just a trade off that we made in the code, and it's actually just going to increase the latency. But it's not a big deal because it's something that's a synchronous, and it doesn't actually matter. So we're going to just leave it in there. But are there any other areas of testing that? Like, here's an example. You might say, okay. We introduce this new module and we've written one hundred tests for it. We could write a thousand more should we write a thousand more tests. So that's an example of subjectivity kind of question. Like the question of do we have enough testing? It's the testing is an interesting thing you really have to defy even the test. Cordray Dusko test code itself is could be buggy. So there is always a balance of how much testing it needs to be Todd good at testing in my opinion. I mean, you can have that's that's how I view testing. I go and look at how you can verify as took grains of dusting white box and black box. You probably heard about that functional. Testing would sus overall testing block box testing would be a lot of lot of the cardinal dopers. We are doing functional testing and white box testing. We kind of know what in some cases. What to look for some of the some of those tests are important because. If something shows up in a white box type testing functional type testing like lock imbalance. It will ultimately show up in a black box. Use a test case when use it as using they will run into it. So do they run into it after ten hours of us or one hour of us? That's a different thing is the reason why is it a race condition? Or is it going to happen all the time? So that is always a balance to me you have to. I there is a mix of white and black box type tests that would make a. Good test, suite. I can't quite defer the number on his saying, okay. Should you be getting thousands us? You might be getting thousand tests that really don't aren't useful or do you could be running one single test. That's really useful. So. I recently had to do this. I was I maintained USB your driver, and we have had some security issues that came up in that driver. And then I fixed several of those and then I was sitting known and looking at it. I thought is that if one shell script, I can run dot could exercise all of the different paths. And it took me some time to come up with it. I was kind of sitting there, and I'm going. Okay. This is what I do when I'm testing this. And so okay. Can I somehow put that in automated form, and then put it initial skip which I did. So now I run through that test. And that actually finds a lot of things like, for example, if I go. Do some kind of action like. Attaching device. That's what did I will does attach the device, and can I detach it or can I at attach you have to think through the process of things that could happen like, for example, somebody attack it's in particular state. So you have to have been I'm writing it us. I kind of have this state transition in my head. I might not actually put it down. But I kind of go, okay. Somebody could come in and ask for the device that's already in use. What do you do do with in that particular case? So you kind of have to think through and then automate as much as you can obviously you cannot automate everything. So we how as recently testing. I'm doing a media featured on the media these sharing feature recently. I have a patch dust batch CDs out for review. So I was looking at that Pat citizen. I'm going how do I test all of the exclusion cases? So I. Came up with a set of obligations like in us. So I'm running all the man who will Tess, and because a lot of these applications, I'm using are not low level enough that I pretty much have to have their green based or the unit to kick off a streaming, and such it takes a manual effort. So now, I have a document while I was doing that. I wrote this tape tablet form, okay? Of. If the device is in use by this uplink Asian can the other application use it or do you see busy? So I have this table sitting there that I need to go back in the next few weeks or couple months to see how I can automate it. So you always start with Emmanuel testing and say look at the manual testing and say, can I automate this somehow using the tools, I have or is it possible? I cannot omit. So these are all the processes I go through anyway. So in my when I'm doing work, we're about security testing. How does that fit into your workflow? This security testing is look for example. Hey, I mean in terms of denial of service is kind of a security kind of problem. Right. So somebody if if you could get a system to a state, it's not responsive that in itself is a is a problem. So you security testing spans. Range of general Colonel testing itself that is the specific aspects of so for for the driver. I was looking at any time user can get his system into a state where it's not responding or chronic sort crashes. Those are the our security vectors you have to look at networking, for example, networking, it's involves getting used packets from the user space, for example. So can somebody send you enough ending packet into data into the kernel we are in network packet. And how does the cardinal reactor is that's one angle. So you have to kind of think through that. And there is a lot of effort going on in the Colonel right now for Zing we go and look at the fussing, and we ended are fussing tests that get run. And then we have a backlog of bugs, of course. So we look at those and we. See how to fix them right in some cases. It could be injecting errors. We have every of injecting certain errors to see how the system reacts and some of those are security related, right? I mean, you could could you and also can use it not regular user. Can they get visibility into the cardinal memory in Ken, their security securities, multiple can a user view privilege today, though. So it's various aspects that I am. I'm starting to look at some of the the bug that we have at the moment. And then in the Google says bought a Saudi says about they are doing huge afford on finding some of the problems. So I'm starting to look at those and see what can I do? In terms of the first testing for testing is. Automated automated. It's a lot of it is autumn origin rated court like, for example, the so okay. One example chaos testing, right? I believe so so let me explain one of the one of the test. They do they'll they'll do a user. This is one example of test that they do. So system calls cardinal has bunch of system calls so use it program will just invoke all of the system calls and pass them say invalid data. How system calls react to it and dry were any dry will that has outside user visible. I e ELS I act so call I actors with. Bad memory are passing in bad, bad values or error injection really depend. How does the reactor? That's that's the those are two examples. I can think of at the moment. Fussing reveal what kind of problems in in one of the cases. This is a. This is a bug actually fixed. One of the cases is it would react boundary kind of conditions. What would what would in one of the cases when such testing happens if you have passing invalid fly invalid flag, for example? It godown cardinal path. It shouldn't take it, for example, should be able to detect that. It's an invalid and not access out of bounce memory. For example, say use this. I I'm just throwing this out. Right. So it's not necessarily a Leslie true. But are necessarily real problem in the cardinal. So you you have us user can pass in range of for value one through to fifty five or whatever if die after comes in. And then that I act specifies it wrong range does Colonel start to axe. This memory that it shouldn't access and go down pout and get an counter panic or a warning or a you know, it will. The driver itself run into an error that potentially could impact the other subsystems now would first testing help prevent something like a heart bleed for making it to heart bleed where I if I remember correctly, you basically had a buffer overflow problem. Right. That those of vulnerability because people could could actually access buffers that they shouldn't have been able to some of the idea talking about some of the sent problems that known I'm specter kind of no not specter who was the one before that heart bleed. I don't remember it very well. Okay. All right. That's fine. What about spector chose? What was spector was the? Manner. If you if you can if you can articulate with that was and. That is related to hallway. Right. I can't say I can't I can't speak to it that much because I don't. But it is it is a part of it is exposing we have several CPU's with different threads in threaded CPU in this is hardware. Right. So we'll one process. How can if one process user process can access to use the process will will be real have the ability to to look at? Data. That's not privy to it what it has privileged look at. So that at at a higher level that is the core of the problem. Right. So, but yeah, memory management isn't somebody complex piece. I can't say I know much about it. Okay. Do you know much about like testing it or how? Well, one contest we have house in in the cuddles Loftus suite that we do use to test. But when I am I don't I don't play in that area dot much to know. The extent of dusting, right? Okay. How do you test networking for a single node operating system? Like Lennox is is a single node operating system, but it's often networking with other nodes. And I assume there are certain tests, you would wanna run that would involve multiple nodes. What's the process for testing those? You're talking about networking between two different systems different operating systems. Oh, I see you talking about almost it. You're talking more about say, you have a almo- almost like heterogeneous kind of testing that you have different operating systems. Yeah. Running to operate networking stack. Yeah. What example exactly tuned to different operating systems talking to each other is is that in the purview of a self test or no, it's not. But I have done it in the fast. I used to work on NFS. Cardinal NFS part for issue axe. File system could act so in in that time when I was working on that we would actually how interconnective fests we call them hack fests or inter inter kind of duty pests, so we would that is one of the objectives. Being able to say it can NFL can you draw does entropy ability work at that level. Will you be able to emphasis just an example, but TCP IP stack, for example? So you will you be able to take a kid, you have a client running on Indo system, for example. And then you have a soda running on lenox. So how does the communication work? And if you have NFS is again packeted packets, right? You have like and any other APAC it up. You would take it packet. You're sending messages back and forth. Is that interpreted could actually on both sides depending on the stack? So in that particular case, you would run test to make sure that's that's one of the issue could do doing both client and server side have the reverse if Europe ligation save is third or client model, and if you want I don't know if we talk about settled clients anymore as much with the cloud, but but that is the kind of testing that you would do to make sure that even when you have different networking stocks in walled on either side that you are it's a TCP IP protocol. Right. You'll if if you follow that protocol, it should work, but and then. The other aspect of that either into I had to deal with is that if you have a. You might have the same operating system different architectures could come into play based on beginning. What's this little in? But that's all network. You have to worry about those aspects as well one year designing the messaging format messaging packets, I'm not talking so much about TCP IP. For example, you came up with your own using UDP as a base, but you're using your own application level protocol on on the top. So you have to worry about those aspect. So is that is that what you're looking for? Okay. You were at HP for thirteen and a half years could act would you learn about operating systems at HP? What can on? I have been. I was at HP. I was doing h we cuddle cord I worked on the low level experts drivers lot of the CPI brings system. HP Bill HP's version of Unix, right? It's a I believe I think it's probably at some point to be a steepest one and. Yeah, it's it's yeah. And then it's big Indian because that's the big thing that though so big Indian beginning. I'd say so we is sixty four this one of the ones early is sixty four hours I were down and risk as well. My fun. Excellent. Without was when when we were bringing up new system is sixty four system, we took the CPI, and then we had to make sure that big Indian worked. I mean into didn't worry too much about the beginning because they don't have any beginning systems. So how to go through and make sure I know what I did a lot of. Experts work where HP's servers there is lot sixty four way thirty two way sixteen weeks or even eight. We did on. But we supported online to placement on those experts online the placement what that means is you could Yankee call it out and replace it. And so all of that goes all the way from when you get there is the way sequence of things to do you push about an and you say, okay goes into driver has to go into a questing state and undrivable sat to detach itself. And then you go into CPI and make it make sure that you actually get it ready for removing the device, and then you put the new device back in and you'd house recognized. So it kind of spans from all the way from low level ACP all the way into the user level house one. I lost a lot of Tom's doing that. Because he had actually pull them out. So what was was fun? So that's that's the that's the work. I did. And then also worked on shutdown paths. For example, you know, how to gracefully shutdown in various cases system is just the way some of the internals of the operating system worked in terms of Twitter member even though so. Then you have is sixty four ways system or when you have multiple CPU's. You have the the way the operating system the internals of the operating system worked you had to gracefully shut them down in terms of various. It's not hard. It's not hard heart per se but coordination between different CPU's over just can go wrong. You you might wanna CPU might choose to not. Okay. So this is this is very internal to that particular obligations. This is not going. Oh, gosh. I don't remember. Okay. Accent you'd ask. So. Yeah. Samsung what about Samsung? So you worked there Samsung makes mobile operating systems. I guess at that point we're talking about Lennox, tell me about the world of mobile operating system development. It's the it's it's vide- different. So I came from doing of just sixty four way type Cicero send I don't think I would work done desktop as much but going all the way to going to really small phones. Right. So one of the things that one of the things that they encounter is. There is a lot of vendor drivers that sometimes vendors drivers that come into play. And there is a lot of pressure of time to market type things. So one of the things I realize I noticed is that it is hard for them to go. Take. Keep up with the main line in a lot of different cases. Like, for example. You know, you they they when they release the product and obviously everybody wants their phone to be working me. And I think I'll get mad if my phone isn't working. So it's harder for them to move to Maine new mainline cardinals as fast as they would want to because they have reasons to keep you keep up with the main line because mainline has new features coming in new. They do we all want to move to the main line because mainline Lennox cardinal. Has more features right is continuously. We're fixing bugs adding new features. So we want to move at the same time. You also want to keep your product stable. So it is a continuous balance of the two activities. And in some cases, some cases they the phone Wenders want to stay on. A long-term release you just continually try to merge with main line are you gonna get merge conflicts? Or are you just going to get errors and how the code interacts, you could get much conflicts? And then also in some cases. Phone Vandross might have they'd own creed that they're keeping internally because they might fix problems there on conquering. They might not have time to upstream those send it upstream the fixes, so. Yeah. The merge conflicts. Plus something might break. I was that is one of the things I was doing when I was at Samsung one of my activities was to make sure SAM's devices that are important to Samsung product teams. Continue to work on the mainline cut cardinals. So it would take mainline nine cannolis and make sure that that helped some of the product team small to two new releases quicker that activity itself. And when you say main line is it main line lenox, our mainline Android. I'm sorry. I'm talking about Linda, Linda kernel, we do from the Lennox Colonel dot org. Okay, mainline. Okay. And how does the managing the divergence between Lennox and Android work Android is that are two parts to the to all these phone platforms. Right. I have the underlying system which is the system software which is cardinal. Plus if you look at your phone and looked at system details, you'll see cuddle revision and three dot something or fold out something that is the actual cardinal lluhvision. That they might freeze on like, for example. We'll how we have do you do, you know, how the you probably don't have his ability to to all of the stuff we deal within the cuddle space. Right. So what Linda Susu may get mainline that is continuously moving. It's it has the new features coming in. And we do lease once in about a tweaks once every eight weeks in new cardinal release comes out so during those eight weeks seven to eight weeks, we are doing integration, continuous integration. You're talking about. So there is a two week window between releases this eight weeks is part of that first two weeks after a new release comes in. That's when all of the maintainers, and we are sending patches pull requests to us. Linda's post them in. And then the first two weeks is that is just ending this'll end the Sunday for five dot one. So after that. We are fixing stuff that we found in this the code base. We just did Linda just pulled in two weeks. So that continues depending on the comfort level depending on how many patches that are coming in fixes coming in that'll settle in the four to five weeks. And then the next release comes out very often the products like, for example, foreign sort of Samsung and Google or any other phone vendors, and even some of the destroys who to or they all to say, okay. This is what we want to be our next release on or this product does Samsung galaxy ten this is the kernel release we want to pace it on the usually always use is stable release and Greco. Hartman maintains maintains stable releases. He's been our show. Very interesting. So he probably went took you through that. Don't remember very much. Deep detailed show. So what Greg does is as soon as five zero for example, just came out like two weeks ago he'll start to maintain that as a stable release. And there's a couple three other leases that are always always stable release that keep keep getting fixes from the main line. So when we fix a say security problem or a driver bugger, and we look at it and say, this is maintain decide that are maintainers dollar pursuer fix that problem will decide whether this is should go into stable or not. And then it'll go into the stable soc stable releases. We have a couple three stabilizes, and then renders sometimes request and say, hey, we would like this particularly to be long term stable which is longer length. Then think it's like five ten five years. I think. It we it. It's decision is made at the time. How long it should be? So the way it works is a very new feature goes into the main line, which is Lynn asus. Cree and then stable trees get fixes from there. So product teams, I mean, individual companies than phone product teams, they decide based on the tried to get their dry worsen. So on that need to run into the one of the main lines, and then they use that whichever one house all of their content to get their product running. They will stabilize on that. Meaning they will pick that as their release to do the release. And after that, they will be they will keep moving they cardinal to with the fixes. And then they'll take March. They keep merging stable releases. This podcast is brought to you by wicks dot com. Build your website quickly with wicks wicks code unites design features with advanced code capabilities. So you can build data driven websites and professional web apps. Very quickly. You can store and manage unlimited data you can create hundreds of dynamic pages, you can add repeating layouts make custom forms call. External API is and take full control of your sights functionality using wicks code API is and your own Java script, you don't need HTML or CSS with wicks codes built in database in ID. You've got one click deployment that instantly updates all the content on your site and everything is SEO friendly. What about security and hosting and maintenance wicks has you covered. So you can spend more time focusing on yourself and your clients if you're. A developer. It's not a problem. There's plenty that you can do without writing a line of code. Although, of course, if you are a developer, then you can do much more you can explore all the resources on the wicks code site. To learn more about web development wherever you are in your developer career, you can discover video tutorials articles code snippets, API references and a lively forum where you can get advanced tips from wicks code experts. Check it out for yourself at wicks dot com slash S, E D, that's wicks dot com slash S. E D. You can get ten percent off your premium plan while developing a website quickly for the web to get that ten percent off the premium plan and support software engineering daily, go to wicks dot com slash S, E D And see what you can do with wicks coach today. Ever. Conflicts between these major companies conflict between Samsung and some other phone vendor. Or are they mostly pretty aligned when it comes to operating system stuff? It's open source said that at that point right is cut scuttle community that they were can so conflicts in what way do you judgment priorities. So as open sodas, Adele, photos, we. We are looking to what can open source, right? Like, for example, we might have to have. But the same time we'd always looking to work with we don't think of it as in in a sense, we don't necessarily think of our selves as belonging to one company or the other because that's not really productive because you want to do. You want to do advanced the Colonel the way that scales and improves for the betterment of large generally general features as opposed to specific features audit product specific feature, for example. And if there is a product specific dry were it's still a component. It'll come in. That will be in the driver's space for if it's there is the driver that's needed for the phone to run that will be part of the driver. And then for example, Samsung would contribute that code and they will maintain that. I don't we there is. No, we I can't think of a conflict that we will have about a feature that, you know, this is people are willing to listen say, hey, we want this. But it needs to be generous as well. Does that answer your question something? No where I think is interesting about Lennox is, it's this enormous project might be the the most people project significance intersection. The you know, the biggest project managed by the most people with the most impact in human history. I I don't I don't know of another can't think of another. What has your work in the Lennox project taught you about human nature? That's a good question. So it's again it collection of individuals, which is I find what king in the Lennox. Couldn't I have a lot of projects before I started working in working in the open source, and then contributing doodling Colonel my sense is that people are welcoming. They would want you to do best contribute. They will give back because you are putting yourself out there with the large number of when you send the piece of patch out. Right. You do not know what kind of comments you would get back. So you have to I have learned minutes, a growth process, you have to learn about you have to figure out is a self discovery as well, you kind of go Hardaway until you actually send your code out. And then when somebody says there is a better way to do this. You have to understand that you have to say ha yes that is a better way to do it. And then you have to self discovery process of somebody points out something would that bother your how do you react to it? So it's self discovery aspect. You should be able to take feedback and criticism. And there is always been you look at six thousand or seven thousand ten thousand people there is always a percentage of the people a large person to those people want to work with you. And there could always be somebody that doesn't want to. Or you'll see different kinds of personalities just like any community. So what I phoned that works. Well for me is. I like to the way like to work is that I would take the feedback. And then look at the feedback and say, hey done under the day. We all are working to introduce the best code the one operating principle. Right. So we might have different opinions on what is best. So at that point communication becomes spirit important. So if you want to survive in open source community to help good communication in terms of being able to explain being able to explain you're the problem, you're solving. Also other interpersonal issues as well in terms of how would you react to feedback and criticism or how would you go back and say, hey, I understand what you're saying. But this is what I'm thinking. So be able to have that conversation. And it's the process imagine delivering criticism. Mm-hmm. And delivering feedback is is also an art could act. Those are. Yes, obviously. Yes. So it is tough. It's a tough process. Both are tough giving giving feedback in a way that you want to give feedback in a way very constructive. It it it has to be taken viewed the same way as well. So both are tough. So you teaches you a lot about him just humans. We're all humans. We are not perfect and we make mistakes, and we have to be understanding of each other. And then keep going John's are great feedback loop though, personally for like personal self improvement of communication skills. Of course. Yeah. It is. I have learned a lot. I mean, you know, I yes, it is. It is a I mean, I do I have taken. I mean that is one thing. I remember from it while back, I took a technical communication. Type class how do you interact with? In teams or how do you? So one of the things that stayed with me, and I still use it is if you ever have a situation where it's emotional situation, or then you want to not match emotion with the motion try to because if you it's like adding fuel to the fire, right? So so if you you have to set by stood back and say, hey, what can I say that would doesn't add contribute? If somebody is getting upset about it about something. I mean, it's actually something to us even in personal life to somebody is getting upset about something. You don't want to say something that upsets them more because the point communication stops. You have to stop and think how could you end, thankfully in an Email based communications it is actually easier. Innovate right in a sitting in a meeting room actually talking about code. It's a little bit different Innova than doing it or Email because time to yes. Delete about on Russian. Well, here's delete fun. But somebody probably has an audit. Log. No. I mean, Email if it's you're sending Email you can always look it over multiple time. At least look at it. One more time right before you hit the send that's send button. Right. So you can kind of look it and say how am I how how would other percent ticket? We might not think about it. I mean, you know. So so Email gives you that little bit extra time. I'm saying based communication or that little bit of extra time to think about it and say, hey, you know. Definitely I like how Google the g mail recently. Now, you have an undo button after you send the Email which I mean only last for like three seconds or something like if you actually make the emotional mistake the undo button. Okay. Undo undo. A three second window could be emotional mistake on it could be something. I have done this. I would just be thinking something. And then I just come up with. Oh, gosh. This is the real problem. Right. I mean, the code and it could be mine. And then I would just respond in Russian and go men, I should have thought about it a little more. You know, it could be it could be not just emotionally it could be something else to you're not thinking thinking through and you kind of thinking going down the wrong path. And you kind of just like, oh, this could happen and go. No. That can never happen. You know? And then, but it's now that you know. So so that is and then another aspect I phoned. I mean, I have been I have done a mentoring within Samsung, and I'm used to entering HP as well, which I have on both sides I mentored, and then I was Mantilla's. Well, so I learned a lot in both aspects. So one thing I found is it's open source is great place to collaborate work. I mean, we'll doing so much clever work, but it is always a difficult for somebody to get over the EPA -hension and p fear of making their work public. So that is a journey. You just have to make if you want to be you have to you have to want to do that. Because it's so much easier to not have to do that. It's so much easier to work on a project that isn't open source, and you can do. Do the work you do and you have a few peers that look at the code, and then it goes in right? But it is a journey. You have to make when you are setting out to to be wall. Notable to have you work critiqued by lots of people. And I see that in the process, I have mentored lots of engineers and the process for them there. It's an upper -hension for one. I they don't know where to start because when I struggle with the to like, for example, I told you I don't know much about him or I don't know much about scheduling, you know, like lyrics accces, it's a huge project. And there is so many areas not one person can be. I mean, nobody can be an expert at everything. So we all however pieces that we are good at in. If you don't look at subsystem put a couple of years it moves that it is hard. Even keep up that that's how much Delman goes on. So what happens has result is a lot. So smart engineers. It's hard for them to kind of wrap around will do I start. This is a huge cookie. Will do I start biting from? Right. So that's the first thing I notice and the second thing is getting over that apprehension what if I make mistake. How would people react to it? I'm going to look dumb. You know? So you have to go through the process. So. Yeah. Last question. How has your work within the legs community changed your perspective on computer, science, computer, science when I the human impact off technology. So just open source, and Linda cts, and is mind boggling to me, we have changed in the last just ten fifteen years. It changed. Our lives so much that we the communicate has changed the way we interact which other it has changed. How do we the bay? We conduct business has changed and in touch our lives in a vase that we don't even know. Right. Oftentimes, I would talk to people people that aren't familiar with technology, or, you know, don't I mean, they use their phone, but they don't really know what that means. Right. And then so I think I am. And then the the whole and deep learning and all of these are based on computer science. So it is amazing to see in the last twenty years. How much how many lives it is touching? So I have always thought that computer science is a combination of art and science, and there is a lot of creativity in walled. So it's a different kind of creativity. You're not in a painting a picture, but it is similar to that. It's a creative process. So the rest of the it's one thing to write an operating system. And how it you how it gets used is. And is something that continues to amaze me. Sure, thanks for coming. In software engineer daily. Great talking. Thank you for. Tim d. Go CD is a continuous delivery tool created by thought works. It's open source. It's free to use. And go CD has all the features that you need for continuous delivery. You can model your deployment pipelines without installing any plug ins. You can use the value stream map to visualize your end to end workflow. And if you use Cooper netties, go CD is a natural fit to add continuous delivery to your cloud. Native project with go CD on Cooper, Netease. You define your build. Workflow? You let go CD provisions and scale your infrastructure on the fly and go see the agents, use Cooper netties to scale as needed checkout. Go CD dot org slash SE daily. And learn how you can get started. Go city was built with the learnings of the thought works engineering team, and they have talked in such detail about building the product in previous episodes of software engineering daily thought works was very early to the. Continuous delivery trend, and they know about Qatar delivery as much as almost anybody in the industry. It's great to always see continued progress on ghost CD with new features like Cooper, Netease integrations. So you know, that you're investing in a continuous delivery tool. That is built for the long term. You can check it out yourself at go CD dot org slash S. E daily.