Opentelemetry discussed on Software Engineering Daily

Automatic TRANSCRIPT

Are off, then you need to know what has happened, right? And then that's what I think tracing comes to play, because it can fully connect the dots all the way from the user to whatever backend system you might be using. So we know let's say we can isolate where the problem might be coming from, right? So in a microservices architecture, and then user might be facing problems, but that might be from an upstream service that the user doesn't talk to directly. And of course, logs give us, let's say, the most free form visibility so that once we have isolated the problem down to a particular part of our system, maybe we can go look for anything that might be off or wrong during the period of time that we're interested in. So all these work together to troubleshoot problems quite frequently. Can you give me a sense of the usage of OpenTelemetry and maybe describe why a open-source project is relevant to a generally closed source company. Yes, I guess blind originally started obviously as cloud source provider software. But we have been embracing open-source more and more as a company, right? Including, of course, open elementary. OpenTelemetry started a combination company where one of the co creators of the project. And several of the founding members of open elementary are now Planck, either via the acquisition of omniscient, or people who join the subsequently. The reason we believed OpenTelemetry project like open telemetry was important is because, as I said, with observability, what we're trying to do is bring together the data across all telemetry signals so we can have more effective monitoring and troubleshooting of applications and infrastructure. And that is impossible if you rely on proprietary protocols and data collection mechanisms, right? So I might have one way of collecting logs. I might have another way still proprietary of collecting metrics. I might have an APM agent that is totally appropriate instruments my application. And collects the data and all of these end up in some data silos, right? That are impossible to connect. By design and definition in some sense. So we found that this was very important to democratize the data collection and give the user the power of what we want to do with that data. First of all, and second give us, let's say, give the ability to tools to connect these data together. So OpenTelemetry is, of course, the merger of open sensors and open tracing. Projects that started with similar goals, but at the end of the day, open telemetry is trying to necessarily create a set of standards and an implementation on top of it. For most popular languages, so we can instrument applications, whereas the case, auto instrument applications with auto instrumentation agents, and data collection, let's say, infrastructure for transferring all that data to whatever backend is the search of the user. And the backend can be a proprietary backend. It's our service or it could be an open-source solution like Prometheus, let's say. And it's worth saying that augmented elementary today is the second most popular project in CNCF in terms of contributions, only second to Kubernetes. It should have been fully embraced by the industry and a lot of end users. When you started up telemetry at omniscient, what were the goals of the project? The goal for us was I guess we had to believe that anything that goes into the user's environment, modern cloud environment has to be open-source, otherwise the users who don't trust it, right? And secondarily, we had the goal of democratizing the data, right? In the past, especially when it came to application monitoring, all the technology and IP that we had created as an industry was all about instrumentation. So how we can intelligently collect a very small subset of the data and based on that, troubleshooting application. In our view, in modern environments, that doesn't work anymore, right? So we wanted to move all that intelligence to how we analyze the data and to the analytics we were able to provide on top of that data, right? So our belief was let's have the goal of building an open-source solution that would be very powerful. And very simple, let's say to use in terms of collecting data as much as data as possible, and then give the users the option to use whatever backend, they believed would solve their problem best. So automation that was playing subsequently, we focused a lot on building, let's say, a very powerful analytics on top of this data and we think that's where the value is, right? And we decided to contribute all our IP, let's say, an effort in making open telemetry successful. So let's say that data collection is not any more differentiation because we don't think that's where the value of the users is. Can you tell me more about how the spec for OpenTelemetry was developed and exactly what it gave to people that did not exist in the open-source telemetry ecosystem before? Because there were other open-source projects around telemetry before. Yes. So first of all, open the elementary is not a single spec, right? Because the project had fairly ambitious and first of all, deals with metrics, traces, and logs. It deals with transfer protocols of this data. It deals with, let's say, the definition of spans metrics, all of that as they are emitted and generated in the application. But the goal for the project from the beginning was to standardize how we essentially collect and transmit this data. It would never have the goal. We had the explicit explicitly we didn't want to be a back end for this data, right? We just wanted to standardize in democratize how this data is being collected. From all the applications. And I don't think there was an effort like this before. It was open tracing, of course, that tries to standardize the tracing aspect, and it was open sensors that tried to do the same for tracing and metrics, but generally speaking, we didn't feel like there was another project that was trying to standardize this. That's why we because that's why the project was created in the first place. And that's why we put our effort behind it. And this standardization is very, very important because once you define the specs, then first of all, anybody can be a little implementation. The result provides an implementation, but then the data, let's say, is fairly well described, and structured at the source. So then you can build, let's say, analytics on top, or a backend, open-source proprietary, that can actually be a lot more powerful because the data is truly structured at the source. Unlike let's say traditionally how logs look like, which were free form. And was very, very difficult to build something more than a simple, let's say, text search and full extraction on top, right? Here, let's say we have a lot of metadata, but come from the source, but help us connect all these data together. Gotcha. And was there any connection between the emergence of the OpenTelemetry project and the growth of Kubernetes? Of course, I think the correlation that for the most.

Coming up next