266: Dodging Ubuntu End of Life & Ruby on Rails DevOps with Justin Snair
You're listening to the ruby on rails podcast on the five by five network. You're listening to episode to sixty six and I'm your host Britney. Martin Justin snares, the director of cloud infrastructure for the Pittsburgh cultural trust he's been with the cultural trust for ten years in this passionate about his job. He really has AWS certifications always learning more. This man, literally eats sleeps and breathes AWS infrastructure is also a devoted husband father. And in his spare time an avid woodworker. Welcome to the show Justin thanks for having. Absolutely. So Justin, can you? Please. Tell us your IT origin story. Sure. I mean, I think like most people in IT started at a very young age. I got that first computer. And I just wanted to learn everything about it. Learn how it worked learn how to program things especially on it. I had stalled Lennox on my family's computer when I was nine which was not didn't go very well with my parents since they didn't even know how to turn it on at that point. But I loved it. And that's kind of where it started. I fell in love with. That I really focused on like computer classes in high school in went to college for network ministration network security more focused when I started with the trust. I had a few other jobs started with the trust. I kind of was thrust into the whole cloud stuff and really fell in love with it. And kind of sparked a new passion was before it had been mostly the security networking side. Now now, it's more DevOps in and cloud infrastructure. So when you start at the trust everything was very much stored on site the day. I started was actually the day that we actually undertake it a big website redesign project that was my very first day. So it was kind of overwhelming, but we worked with the with the developers to kinda get a plan together, and what kind of environment we wanted to have. And yet that point everything was all self hosted. But that was kind of the my first experience with ever even using ruby for rails for anything. So. Interesting. So I think our listeners would enjoy hearing about all the aid of US certifications. You have so any tips and tricks on getting certified. I would say the biggest thing is to get your hands dirty actually get in there. I mean for me at least, I'm a I'm a hands on kind of learner I like to get in there and just tinker with things in can kind of see how it however they works. They'd against documentation is actually really really good. So have never had a problem finding the answer to something that I needed to know AWS, just looking at the documentation. And then also I take advantage of couple different training programs. A cloud guru for one in lenox kademi is the second one I really like lenox academy training because they've got virtual labs. Grated virtual labs at that. So they can tell you exactly what you did wrong out in graded for you, which comes in really handy. And then just my general strategy when I take a test is, you know, I like to take their path the eight of his practices exams and get at least ninety percent before I ever attempt the real thing. And then on the real test. I land through the questions, I know right away, the speed through everything if I pause for even thirty seconds on a question. I'll skip it come back to it at the end just to make sure I can get all the questions answered in the time a lot 'cause there are quite a lot of questions that you don't have much time to really spend on just one question without one thing that I've really enjoyed that. You've done I got the AWS certified developer certificate about three years ago. And then recently had to get recertified and since then you actually built a sandbox environment at the trust. So that multiple people could go into the accounts, spin up resources and not worry about. About it. Do you think this is something that most orgnization should be doing? Yeah. Absolutely. I mean, I think we're we're lucky that we have a boss that kind of understands the the importance of the training kind of hands on training. So I kind of asked her that for a couple years. And finally, you know, we sent quite a few people to date of. Yes. Conference this year. So I thought it was a good opportunity to roll that out. So that everyone has something to us, you know, just to do the labs at the conference. But then moving forward to experiment with whatever they need to you know, NATO Bs just kinda learn in grow yet. So as the trust director of cloud infrastructure, what has been the trickiest part of managing rails environment. So I would say definitely used to be like managing all the gems dependencies, but bundler has made that like a no brainer anymore. It also used to be a little difficulty raw try to run multiple versions of ruby on the same server, so we're looking RBM or something like that to control which I ever being used. We've simplified that infrastructure down. So that we're only running a symbol single application on anyone server. So that really eliminates the need to maintain multiple versions of ruby in parallel. So really, I mean difficulty at this part at this point. I mean, we have got it into autopilot so knock on wood here. It's pretty easy. These days yet of US when I started with the trust we had a custom deployment application that would manage all that. And one thing that I'm really proud of his team. I feel that we heap offloading things to existing AWS services. So that we can really smooth out those deployments. It used to be where we would choose deployed very strategic times throughout the day. And while we still tend to do that because we need to dance around big on sales and whatnot. I mean, if we have something that needs to go out to production. I feel that as a team we were just a lot more confident about getting those across an obviously continuous improvement deliveries a big buzzword these days. Yeah. You should be able to you should be able to deploy whenever you need to even if it is in the middle of on sale. Everything's built properly deploys work properly than deploying during on. So should not be Abro. Now completely. Agreed. And our listeners have heard many tales of me talking about getting ready for Hamilton. And just was a huge. General part and making sure that that on sale was a success. So let's dive into the reason that I brought you on today and talking about up grading rails environment. So at the trust were moving away from a bunch of fourteen four because it's reaching end of life. Can you break down for the listeners? What that actually means sure. So canonical, which the company that releases boon to they release new software new long-term supported versions of their operating system every two years and the long-term supported version, basically just means that it supported for five years from the day it's released then on the off years they release kind of they're beta versions where the really only supported for two years. So our strategy is always been only used a long-term support versions just because that gives us five years before we really need to do another upgrade they'll keep everything running during that period of time. There's no real reason to upgrade as long as everything works or you have a really compelling reason that you need a package that isn't available yet. But not run into that. Luckily in a long time, the when when when it actually goes end of life so right now we're on fourteen four which is end of life as of yesterday. That basically means that canonical is no longer going to release any bug fixes or upgrades security patches of any kind for any of the packages installed on the operating system. So. While it's not total buzzkill. It can it makes administration a lot more difficult. So you obviously usually package manager makes things a lot easier to install things upgrade things if once your end of life, the only way to do those types of security or bug fixes to actually compile things from source in. It's all things the old fashioned way, which while doable is likely to break a lot of things if you've been running packages the entire time up to that point. So our general rule of thumb is we'd like to upgrade before we're completely end of life. But the little bit delay this year on our schedule, but we should still be done in the next week or two. So we're not too far behind so just to make that clear so say next week something like Hartley was discovered what would happen to the people who are currently on fourteen four the only way they would be able to fix a bug like that would be to them one ever. Let's safe Tarpley open. This age would release their fixed source code. They would have to manually compile that from source and replace all their binary on their system manually with the patched version, which specially with something like open a sage which is built into many dependencies from almost every package on the operating system doing something like that manually is certainly air prone and just not recommended. Yeah. It makes a lot of sense. And I think there's a real division between our listeners of who is on a platform like AWS where you're managing a lot of these things, and then using something like, Roku or cloud, sixty six in order to manage your your environments, just because those platforms as a service will manage all that for you. But you're certainly paying for that expense. Sure end. So if you're lucky to have someone like Justin to have someone like Justin on your staff than you're able to keep on top of those kinds of things. But otherwise, you're real risk. If you don't keep. With the with the security taxes. So what is the trust plan on upgrading our review on rails environments because we do have four different applications that we run on almost the same ruby on rails versions. Yeah. I mean, I kind of touched on a little bit. But I mean, the the main thing is we we only use the long term support releases of two. We tend to be a little more bleeding edge when it comes to ruby and. Passenger an engine X. We kind of usually on the latest of all of those things, but in terms of the operating system, we usually stick with the the long-term supported releases. And we really don't upgrade from that. Unless until they go end of life, or like, I said before there's a compelling reason. Some software that's not available or aversion unavailable in packaged form that kind of forces us to upgrade. Once we've decided that we're going up grade. We usually take the path that we upgrade our test servers. I then obviously production. This particular is a little bit different change couple other things within our others feeling group. So it makes it a little more complicated. But the general thumb is pretty pretty much the same upgrade test them broad. So if we'd maintained everything, and we're really using the the ability to upgrade other things as just mentioned, we're gonna be upgrading our ruby version. There's been a lot of security patches to ruby recently think Ruby's at two dot six dot four at the moment, which is already different than what we're already starting to upgrade with. So we'll have to upgrade. But if we had kept everything status quo. We hadn't changed anything else would the upgrade path from above to fourteen to eighteen be difficult. Well, the way we chose to go at this was to not really do upgrade. We're kind of starting with a fresh bone to eighteen four server and scripted out the tire process to get that blank boon to eighteen o four AM. I from AWS to have all the soft to bootstrap it basically with all the softer that we need to run our application. I thought it was important. This time the script that all out when we originally built this environment. A lot of it was done by consultant and since that time while we kind of had an idea of everything that was needed in. We've upgraded things as we needed to. We've never actually had to build a server from scratch with all of this stuff. We kind of just use their images in kind of upgraded along the way. So this was the first time we've we've kind of I've kind of done this whole thing from from zero which was great because we really need to have all of this documented, we didn't cause they didn't really provide us any of that stuff when they left us as contractors tend to do. Let's see. Yeah. That's that's basically the jets why we decided you descript this time, so how do we how do we get to the conclusion of what sizing we should be using the servers is as you mentioned were starting over with new sizes. So how do we know what is the best type server type that we should be using for these applications? So generally would we figure that out in the past? And we're kind of locked in now to a service is we reserved instances over locked in for three years. So what before we make those decisions gonna be three year commitment, we do heavy load testing on the application, and we're testing, you know, how many users we can throw at it before it dies. And basically what how users we can throw at it maintain the performance that we'd like so we kinda throw a crazy number out to see what it takes actually kill it. Then we kinda see what the number is it can operate within our expectations without killing it. And that's kind of how we size it based on that. We kinda size it based handle hundred concurrent users per server. That's just kinda what we've done you could obviously make your servers bigger. But because we use auto scaling you're paying for that exercise. So we'd rather have each server handle a smaller number of users than scale up as we need more servers rather than just buying bigger service to start with so far listeners who have never actually conducted low testing before do you recommend that they spin up a separate environment and try to hit it as hard as they can. Or should they actually try to off load tests against their production environment on off hours? I can see the merits of both methods. If you have a way of completely copying your production environment. Exactly, which we do in AWS very easy in a lot of other hosted platforms. Make it pretty easy to spend a duplicate copy of the environment. As long as you're testing apples to apples that that strategy works, just fine. I still think it's worthwhile though to test against your. Production site, just to make sure that you truly are testing apples to apples in that. There's not some little difference that you're gonna find out down the road that oh we forgot that production has this other little setting on it that that this test over didn't have. So I would recommend you start with kind of not production. And then once you've got everything out and you've got your test out. Any know exactly what you wanna test. How many users you wanna throw at it? You do want. It's important test production as well. Agreed. So you mentioned that you're writing a custom script in order to upgrade our ruby on rails environments. And so I'd love to add briefly walk through the customs strip that you wrote in highlight what you're executing. Because eventually this is just going to be incredibly easy for us at the trust to be able to just execute the script, you know. Should we need to spin up new application, we actually just acquired a different application from a consultancy in it became incredibly, easy for us to integrate that over into our environment. Just because we have pretty much a set formula of how. Host ruby on rails. Applications the trust if you would mind out briefly walking through that script. That would be great. Sure. So it's it's just the bash script. It's gets attached as user data to the AM. I whenever it's launched. So basically, whenever the instance comes up all the stuff just runs automatically. So it's bootstrapping the instance with everything we need and. Yeah, just run through everything that we're doing here. First up is we're just going to do an update on the app. Get update which updates all the packages available to app get the package manager. And then an apt to get upgrade actually upgrades all those packages that are available. That's kind of the first step just because the AM I from AWS is boon to eighteen four, but it's boon to eighteen four at the time. That that aim I was created. So there's probably packages that have been released from the time that that came out till now. So it's important that you get everything up to date is fresh as possible for free start the process. The next step is installing Python-3 and the it'll be a CLI because we take advantage of VSEL life for a lot of other things move forward in the strips of asked against all kind of right off the bat. Then we said a few variables here. This is mostly used for us SM, which is AWS systems manager to do some automated stuff on the servers. So we set some keys for Sam on terms of what SM policies it should be downloading. We said if we create a variable for the region than the host name in the different tags that we want the servers to be tagged with. Let's see what he do next set the time zone eastern. That's basically what we've used for all of our servers. So there in eastern based business at not everybody's that way. Yeah. We're we're in the east coast, but you know, that's personal preference. A lot of people like to use UT CD's days. And then we get into installing some of the prerequisites for everything else is gonna get stalled. So. Certificates the build essential tool. So that we can compile software my sequel clients, and my sequel, libraries, then we start adding some customer, positively. So we don't get all of our packages from canonical repository, which is the boom to official repository. A lot of the software that we use is more up-to-date in some of the specialty repositories. So for instance, for no J s we use their repository directly and for yarn as well. For ruby. We use the bright box repositories, and they're usually right on top of things new versions are released within a week as opposed to canonical where you know, you're probably waiting a month or so before they get around to unless it's a security issue in there, pretty quick. But officers feature released you're gonna wait awhile. Also, then fusion passenger Wease customer positive from them. So we can get the latest compiled binary passenger quickly and easily. Then what we ATL the customer positive than we go through actually install the software from those repository. So. Image magic in curl in some libraries needed for ruby. Then we actually install ruby install Butler and update Butler, install no J s yarn passenger engine ex. We'd still post fixing configure that Email can be related through the server we use SAS for that. We also we use radius to Q authenticate from our Lennox servers back to our window service here on premise, so we then configure radius on the servers are used can log in with their same username and password that they log into machines here, which then I've been at the local users who should have access to the box. Next up is I install code deploy. Which is the deployment tool that we use from AWS to actually deploy the code on the servers. So this trapping ask doesn't deploy code. But it gets all of the tools onto the server that are needed to deploy the code. And then we let those tools do that. 'cause that's what they do this. And then finally cloud watch logs. We install that just for sending system. Logs to cloud watch for later analysis. In. There's a section at the bottom where we kind of do some bullying logic tests. If what kind of server it is so based on the tag on the server the test server, we may install some other packages that don't get on production. His one of our other applications, for instance, one of our applications has post presence old on it. So we have to sell the post libraries for that server, but not really anywhere else. So rather than creating a whole separate script for that serve. I added some logic into this to detect if this is that server install this. And that's pretty much. That's awesome. And I think I can I don't think there's anything there's probably a few things that need to scrub out of here. But I could share this listeners wonderful if they want to get head start on doing something like this themselves. There's just a few things I have the scrub out of this that are kinda secret. But I was just about to say that just reading that script. I think generated a large section for our show notes. But yes, I think it's probably share the script. But yeah, we'll definitely do that as a gift or something in the show notes of short a check that out listeners because Justice, but a lot of work into this definitely want you to be able to read the benefits from it. So we mentioned how experienced Justin is with AWS. And so I'd love to wrap up the show by asking him. What is the coolest thing that you're currently working on? Well, I guess that depends on everyone's interpretation of cool 'cause I've got through projects right now. In eighty minutes. I'm working on. That are big the first one is that which is really big is were currently in the process of migrating our CRM application, which is the main business application that we have here. It's basically our entire business to the cloud, which has been very challenging definitely probably the most interesting thing to a lot of people, but trying to fit a legacy application into modern architecture is quite a challenging thing to do. In. There were quite a few hurdles along the way. I just found that really rewarding to be able to. I mean, we're pretty close right now making that happen. We're about two months away from going live AWS Indo excited about that. I'm ready to be done. But it was it was really fun rewarding to to kind of get there. I learned a lot. And then the other thing that's probably a little more fun for everyone else is working with the deep lens camera to develop a kind of a facial recognition application long-term goals of that would be to maybe do ticketless entry at venues. So that people could basically walk in in your face is your ticket essentially, you show up to the show with nothing you walk through a special gate has a camera. And when it sees you at knows that you're supposed to be there for the show and kinda just lights up green light. And you pass on through concept of that. We've got a camera sitting in our in our office. Right. We're doing this podcast right now. And. It's Bisley watching people as they walk in the door. And then if it recognizes that person at greets them on slack. If it doesn't recognize that person, it sends a message to a slack channel to basically train the model who that person is it's we've had some success with it. I think what we've realized we're running into a kind of a limitation of the lens hardware the camera on the deep lenses, really only like three megapixel. So when you're training the images that you're going to be using facial detect detection later, if you're training them with poor polity images, you're gonna get poor quality facial detection out. So. I think our next up that we realize we can't use that camera. It's kind of just the prototyping camera was really meant for you know, full blown application. So I think the next step. So that project are going to be to get some different hardware with some better camera on it and see kind of where we can take that full. Well, how can our listeners follow what you are up to? Not real big on the whole social media thing. I kind of a recluse, but you definitely find me on linked in. Or if you're looking for help with AWS, you could find me on up work for sure Justin snare, both places also willing that mission. It's thank you so much for joining us on the podcast today. Justin listeners if you are on the older version of a bunch to now's the time to find upgrade path catching next week.