All writings · tech · Observability

Observability 101

This article is all you need to understand the importance of observability, why we need it, it's history, current standards (Open Telemetry) and a brief overview of how it all fits together. This is for people who are starting their journey into Observability.

What is Observability

To fully understand observability and it's importance, let's try to understand things step by step, starting from understanding "system".

System is a collective term, given to the combination of everything you have to manage and handle while running your applications. It includes the infrastructure, networking, business logic, Production/ Dev/QA environment etc. This is where you applications run, and your customers access them.

Observable refers to a system's ability to expose its internal state in a way that allows engineers to understand and troubleshoot it effectively. A system is observable when all the crucial information about it's working, which includes the following, but not limited to:

are exposed, collected, processed and made sense of.


In the context of Observability, this crucial information is called Telemetry Data.

While a system being observable means that it exposes Telemetry Data, Observability is the broader practice of collecting, analyzing, and using this data to gain deep insights into system behavior.

Where the Telemetry data comes from

The telemetry data comes from the applications and the infrastructure running our systems. Your application must be properly Instrumented, i.e. there should be written logic, that makes sure telemetry data is properly collected, structured and exported from the system.

Observability goes beyond just monitoring predefined telemetry data. It enables engineers to ask any question about the system’s state. Even those they did not anticipate when designing it. This flexibility is especially critical in complex, distributed architectures like microservices and cloud-native applications, where unexpected issues arise frequently.

Brief history of Observability

Initially there were no Global Standards for instrumenting systems to process telemetry data. As systems grew in complexity, different teams and organisations implemented their own observability solutions. Focusing on different telemetry data depending on their in-house needs. This lead to:


Open Telemetry

To solve these issues, the industry moved towards a standardized approach, leading to the creation of OpenTelemetry (OTel). OpenTelemetry provides a common framework for generating, collecting, and exporting telemetry data.

It standardized three major sections in a vendor neutral way:

  1. API, SDK and Data:

They divided the telemetry data in three major sections and collectively called them Signals:

  1. Instrumentation: Provided two standard ways to instrument a system:
  1. Collector: A single, standard, vendor-agnostic place (binary) to receive, process and export the telemetry data. This collector is configurable and can fit all of your system's needs.

Using the above standards, the telemetry data can be collected in a standard way, so any tool, that uses OTel standards, can use that data to draw insights, This is where SigNoz shines.

Collector

Collector is not a requirement with OTel, the telemetry signals can directly be sent to a backend, read this to know more.

It is the most crucial part of the OTel framework. It is a binary, that is configurable using a YAML file, through which we can modify every aspect of a telemetry data pipeline i.e. from collecting the data, processing it and all the way to exporting it to a backend of your choice (signoz, prometheus, etc.).

It provides a lot of features out of the box, which includes but not limited to:

In the next article, we will try to understand the working of the Open Telemetry Collector. That will help us better understand observability and how to make our own systems observable.

Next -> Otel Collector


Originally published at prateeksingh.life/writings/tech/observability/observability-101. Save to Instapaper.