Facebook, Steph, Josh discussed on Software Engineering Daily
Might understand. He's that around that time. Facebook really didn't have much testing and it's kind of ironic because you were the creator of extreme programming. It was highly dependent on the process of writing unit tests and then writing the features facebook clearly reversed that process and was able to be successful despite the fact that they wrote their features before they their their tests. You often hear that you need. Unit tests. In order to build things like continuous integration. Continuous integration. Let you move much faster because you have this battery of unit tests that runs at every new build but i. i don't think facebook that stuff until later. How is facebook able to move so fast with such a low amount of unit testing. Yes so that was. That was part of the puzzle. One when i arrived one of those. Hey this isn't what's in the books and yet it seems to be working and the answer that i came to is that while the there's a couple parts of it one is. How many of your problems can you test for. And how many problems. Only show up in production. So if if if you're writing a simple calculation but it might not scale in production you can't write unit tests for it so the facebook answer is don't so that's part of it is depending on the ratio of. How many problems is it possible to. Test for before. Production versus after if that ratio is skewed towards. Hey steph only fails in production then. Don't write any tests. The second part of the answer is tests are a form of feedback and facebook. Engineers had many many other forms of feedback. So you'd right on your development server and you'd try stuff out and then it will go through code. Review does second form of feedback. And then it would. At that point it would roll to the internal site. So you get more feedback from that and then it would go through the deployment process. Were it would get rolled out to a small number of machines and then more and more and you get feedback from that and then of a logging and like operational awareness was just part of the engineering culture. So you'd get feedback post production of like how your feature actually was was behaving. So there's a bunch of feedback loops in place. Unit tests occupied for most development facebook. Just the cost benefit the the amount of feedback. They added in the timeless timeliness of the feedback and the cost of achieving that feedback. Just wasn't worth it the in boot camp. You're supposed to code in production. The first week and i was very careful to write tests. Do everything properly. So i got an in a fair amount of of. He got a fair amount of heat because my first featured in land for three weeks and people are like. I don't know how this is gonna work out. One i was. I was wondering that too. I had a huge case of imposter syndrome. When i when i landed at facebook and realize just how different everything was and then the tests that i had written broke almost immediately and they were deleted. That was one of the things that surprise josh. If if you had if you had a test and it failed but the site was up they just delete the test if you had tests that. Were intermittent that that were non deterministic. They were just deleted and at first i was shocked like old. Delete a test. This is producing noise. And it's not producing signal if you eliminate this this noise production per definition. The situation is clearer all of a sudden the fact that you kind of wish that you add a test for something while you didn't and so yeah just chuck it. Let's move on now. You probably wouldn't wanna take that approach to nuclear power plant software or electricity grid software. That seems like a practice that is uniquely useful for kind of early facebook where this thing wasn't yet a communications utility. It was more of a fun thing to do akin to television. So there's an another lesson. I learned that facebook is the difference between reversible and irrevocable decisions so if you if you roll out a feature in new zealand say and people don't like it and you can easily turn it off. Then that's a reversible decision. Facebook engineering spent a lot of effort on a making decisions. Reversible like sometimes you have to do quote unquote extra engineering. To make a decision. Reversible talked with one of the managers in the network infrastructure and. He said when they got a new piece of networking gear in it went to the characterization. Labs just to put it through its paces. They mostly didn't care about its performance. What they cared about was. Can we turn this thing off safely. So and that's an investment in reversibility all the feature flag stuff. That was one of the lessons. I had to learn a people in code reviews sued. Say oh you definitely need to put this bind feature flag will what why my attitude is. Let test it. Make sure that it works. And then i'll just put it out. Well the learned hard learned wisdom. There was the. There's a bunch of stuff you can't test for so yes you have to introduce additional complexity to to put the flags end but if you don't you're going to roll something out and the you're only recourses going to be a hot fix rollout. Which makes the release engineering grumpy. And you did not wanna make release engineering grumpy. What's the role of a manager at facebook. So that changed the fairmount. In the time. I was there when i got there engineers. All at all levels were expected to exercise a lot of initiative. There's a beautiful article by ye shanggong about facebook's early engineering management philosophy and i can't recall the title of it maybe we can find it later. But the upshot is facebook deliberately chose to make engineering easier and engineering management harder..