Prometheus, Grafana, and OpenTelemetry: Building a Unified Observability Stack for Modern Applications

As senior engineers, we understand that simply deploying an application isn’t enough. In today’s complex, distributed systems landscape, understanding “what’s going on” inside our applications is paramount. This isn’t just about knowing if a service is “up” or “down”; it’s about gaining deep insights into its behavior, performance, and health. This is where the powerful trio of Prometheus, Grafana, and OpenTelemetry comes into play, offering a robust, unified observability stack.

In this comprehensive guide, we’ll dive deep into each of these technologies, explore their individual strengths, and most importantly, demonstrate how to integrate them to create a seamless observability experience. Whether you’re managing microservices, serverless functions, or monolithic applications, this stack provides the eyes and ears you need to keep your systems healthy and performant.

Monitoring vs. Observability: A Crucial Distinction

Before we dissect our tools, let’s clarify two often-interchanged terms:

• Monitoring: Traditionally, monitoring involves collecting a predefined set of metrics and logs to track known-unknowns. You set up alerts for specific thresholds (e.g., CPU usage > 80%) and dashboards to visualize expected behavior. It answers the question, “Is the system working as expected?” Monitoring is reactive; you know what you’re looking for.

• Observability: Observability, on the other hand, is a property of a system that allows you to infer its internal state by examining its external outputs (metrics, traces, logs). It helps you understand unknown-unknowns. When an unexpected issue arises, an observable system provides enough rich data to ask arbitrary questions and debug the problem effectively, without needing to deploy new code. It answers the question, “Why is the system behaving this way?” Observability is proactive and exploratory.

While monitoring focuses on specific symptoms, observability aims to understand the root causes. Our chosen stack—Prometheus, Grafana, and OpenTelemetry—is designed to move us beyond mere monitoring towards true observability, providing the telemetry data and the tools to make sense of it.

The Pillars of Observability: Prometheus, Grafana, OpenTelemetry

Each component in our stack plays a distinct, yet complementary, role:

• Prometheus: A powerful time-series database and alerting system, primarily focused on collecting and storing metrics.

• Grafana: The visualization maestro, transforming raw data from various sources (including Prometheus) into intuitive dashboards and managing alerts.

• OpenTelemetry: The vendor-neutral standard for instrumenting applications to generate high-quality telemetry data (metrics, traces, and logs) and export it to various backends.

Let’s explore each in detail.

Prometheus: The Metrics Powerhouse

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. It has since become a cornerstone of the cloud-native monitoring landscape, particularly popular in Kubernetes environments. Its core strength lies in its robust data model and its pull-based metric collection approach.

Prometheus Architecture

The Prometheus ecosystem comprises several components:

• Prometheus Server: The brain of the operation. It scrapes (pulls) metrics from configured targets, stores them in its time-series database, and runs alerting rules.

• Exporters: Lightweight agents that expose metrics from third-party systems or applications in a Prometheus-readable format. Examples include Node Exporter (for host-level metrics), cAdvisor (for container metrics), and various application-specific exporters (e.g., database exporters, message queue exporters).

• Client Libraries: For instrumenting custom applications to expose metrics directly. These libraries are available for many languages (Go, Java, Python, Ruby, Node.js, etc.).

• Pushgateway: A specialized component for handling metrics from ephemeral or short-lived jobs that cannot be scraped directly by the Prometheus server. Jobs push their metrics to the Pushgateway, which then holds them until Prometheus scrapes it.

• Alertmanager: Handles alerts sent by the Prometheus server. It deduplicates, groups, and routes them to the correct receiver (email, PagerDuty, Slack, OpsGenie, etc.). It also supports silencing and inhibition.

Diagram (in words):

[Application 1 (with Prometheus Client Lib) --exposes /metrics-->]
[Application 2 (with Prometheus Client Lib) --exposes /metrics-->]
[OS/Host (with Node Exporter) --exposes /metrics-->]
[Database (with DB Exporter) --exposes /metrics-->]
[Ephemeral Job --pushes--> Pushgateway --exposes /metrics-->]

                                        |
                                        V
                                 [Prometheus Server]
                                    |         ^
                                    V         |
                             [Time-Series Database]
                                    |
                                    V
                           [Alerting Rule Engine]
                                    |
                                    V
                              [Alertmanager] --routes alerts to--> [Slack/Email/PagerDuty]
                                    |
                                    V
                              [Grafana (Data Source)]

Prometheus Data Model and PromQL

Prometheus stores all data as time series: streams of timestamped values belonging to the same metric and the same set of labeled dimensions. Each time series is uniquely identified by its metric name and a set of key-value pairs called labels.

Example time series:

http_requests_total{method="POST", handler="/api/users", status="200"} 144320 @1678886400
http_requests_total{method="GET", handler="/api/products", status="200"} 987654 @1678886400

PromQL (Prometheus Query Language) is Prometheus’s powerful functional query language. It allows you to select and aggregate time series data in real-time. It’s incredibly flexible for slicing, dicing, and performing mathematical operations on your metrics.

PromQL Examples:

Get the total HTTP requests in the last 5 minutes:

sum(rate(http_requests_total[5m]))

Calculate the 99th percentile of HTTP request durations over the last hour:

histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[1h])) by (le, job))

Prometheus Configuration (prometheus.yml)

A basic Prometheus configuration defines scrape targets and rules:

global:
  scrape_interval: 15s # How frequently to scrape targets
  evaluation_interval: 15s # How frequently to evaluate rules

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['alertmanager:9093']

# Rule files specifies where PromQL-based alerting and recording rules are defined.
rule_files:
  - "alert.rules.yml"

# A list of scrape configurations.
scrape_configs:
  # Scrape Prometheus itself
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

  # Scrape Node Exporter
  - job_name: 'node_exporter'
    static_configs:
    - targets: ['node-exporter:9100']

  # Scrape a sample application instrumented with Prometheus client library
  - job_name: 'my_app'
    static_configs:
    - targets: ['my-app-service:8080']
      labels:
        environment: production
        service_name: my-web-app

Grafana: The Visualization Maestro

Grafana is an open-source platform for monitoring and observability. It allows you to query, visualize, alert on, and explore your metrics, logs, and traces no matter where they are stored. It’s renowned for its beautiful, flexible dashboards and wide array of data source integrations.

Key Features of Grafana

• Dashboards: Build interactive and dynamic dashboards using various panels (graphs, tables, heatmaps, single stats, etc.).

• Data Sources: Connects to virtually any data source, including Prometheus, InfluxDB, PostgreSQL, MySQL, Elasticsearch, Loki, Tempo, and many more.

• Alerting: Define alert rules directly within Grafana dashboards, which can then notify various channels (email, Slack, PagerDuty, etc.). While Prometheus Alertmanager handles routing for Prometheus-native alerts, Grafana’s alerting allows broader data source coverage.

• Templating and Variables: Create dynamic dashboards that can be filtered by service, environment, host, and more using template variables, reducing dashboard sprawl.

• Annotations: Mark events on graphs, such as deployments or incidents, to correlate them with performance changes.

• Explore: A powerful ad-hoc query interface for investigating issues, allowing you to seamlessly switch between metrics, logs, and traces.

Building a Grafana Dashboard for Application Health

Imagine you want to monitor a web application. A typical Grafana dashboard for this might include:

• Request Rate: A graph showing sum(rate(http_requests_total[1m])) by (status_code) to see traffic volume and error rates.

• Latency: A graph showing histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) for 95th percentile request duration.

• Error Rate: A single stat showing sum(rate(http_requests_total{status_code=~"5.."}[1m])) / sum(rate(http_requests_total[1m])) * 100 to quickly see the percentage of 5xx errors.

• System Resources: Panels showing CPU, memory, and disk usage from Node Exporter.

• Active Users/Sessions: If your application exposes such a metric.

Grafana simplifies the creation of these visualizations by providing a user-friendly UI to build queries and select visualization types. You pick Prometheus as a data source, enter your PromQL query, and Grafana does the rest.

Diagram (in words):

[Developer/SRE] <--- Interacts with ---> [Grafana UI (Web Browser)]
                                 ^
                                 |
                          [Grafana Server]
                          |             |
                          V             V
                 [Prometheus DB]   [Other Data Sources (Loki, Tempo, etc.)]
                          ^             ^
                          |             |
                         (Queries)     (Queries)

OpenTelemetry: The Observability Standard

OpenTelemetry (OTel) is a Cloud Native Computing Foundation (CNCF) project that provides a set of APIs, SDKs, and tools to instrument, generate, collect, and export telemetry data (metrics, traces, and logs) in a vendor-agnostic way. Its primary goal is to standardize how applications generate observability data, decoupling instrumentation from the backend analysis tools.

Why OpenTelemetry?

Before OTel, if you wanted to use a different tracing backend (e.g., switch from Jaeger to Zipkin or a commercial solution), you often had to re-instrument your entire application. This was a costly and time-consuming process. OTel solves this “vendor lock-in” problem by providing a single, unified standard for telemetry data.

Key Concepts

• Traces: Represent the end-to-end journey of a request or transaction through a distributed system. A trace is composed of one or more spans.

  • Span: Represents a single operation within a trace (e.g., a function call, an HTTP request to another service, a database query). Spans have a name, start/end timestamps, attributes (key-value metadata), and can have parent-child relationships, forming a directed acyclic graph.

  • Context Propagation: The mechanism by which trace context (trace ID, span ID) is passed between services, allowing the reconstruction of the full trace across multiple microservices.

• Metrics: Numerical measurements representing characteristics of a system at a given point in time. OpenTelemetry supports various metric types that align well with Prometheus:

  • Counters: Monotonically increasing values (e.g., total requests).

  • Gauges: Values that can go up or down (e.g., current CPU usage, number of active connections).

  • Histograms: Samples observations (e.g., request durations) and groups them into configurable buckets, allowing calculation of percentiles.

  • Summaries: Similar to histograms but calculate streaming quantiles on the client side.

• Logs: Timestamped text records, often unstructured, used to describe discrete events that happened within a system. While OTel’s logging API is still evolving compared to traces and metrics, the goal is to associate logs with trace and span IDs for better correlation.

OpenTelemetry Architecture

The OTel ecosystem typically involves:

• API & SDK: The API defines how instrumentation libraries interact with OTel. The SDK implements the API and provides the necessary components (processors, exporters) to process and send telemetry data.

• Instrumentation Libraries: Language-specific libraries that automatically instrument popular frameworks and libraries (e.g., HTTP servers, database clients) to generate telemetry data without manual code changes.

• OpenTelemetry Collector: A vendor-agnostic proxy that receives, processes, and exports telemetry data. It’s highly configurable and can perform tasks like batching, filtering, transforming, and routing data to multiple backends.

• Exporters: Components that send telemetry data from the SDK or Collector to various backends (e.g., Prometheus exporter, Jaeger exporter, Loki exporter, OTLP exporter for OpenTelemetry Protocol compatible backends).

Diagram (in words):

[Application 1 (OTel SDK/Agent/Auto-Instrumentation)] --OTLP-->
[Application 2 (OTel SDK/Agent/Auto-Instrumentation)] --OTLP-->
[Application N (OTel SDK/Agent/Auto-Instrumentation)] --OTLP-->

                                  |
                                  V
                           [OpenTelemetry Collector]
                           (Receivers, Processors, Exporters)
                                  |
                                  +-----------------------+-----------------------+
                                  V                       V                       V
                        [Prometheus (Metrics)]  [Jaeger/Tempo (Traces)]   [Loki (Logs)]
                                  |                       |                       |
                                  V                       V                       V
                          [Grafana (Dashboards)]  [Grafana (Trace UI)]    [Grafana (Explore/Logs)]

Code Example: Basic Python Application with OpenTelemetry

Let’s instrument a simple Python Flask application to emit metrics and traces using OpenTelemetry.

First, install necessary packages:

pip install Flask opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-flask opentelemetry-instrumentation-requests

Then, instrument your Flask app:

from flask import Flask, request
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
import time
import requests

# Resource defines attributes of the service
resource = Resource.create({
    "service.name": "my-flask-app",
    "service.version": "1.0.0",
    "deployment.environment": "development"
})

# --- Tracing Setup ---
trace_provider = TracerProvider(resource=resource)
# Exporter sends spans to the OpenTelemetry Collector
otlp_trace_exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
trace_provider.add_span_processor(BatchSpanProcessor(otlp_trace_exporter))
trace.set_tracer_provider(trace_provider)

# --- Metrics Setup ---
metric_reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint="localhost:4317", insecure=True)
)
metric_provider = MeterProvider(metric_readers=[metric_reader], resource=resource)
# You can also register views for histograms, but for simplicity, we omit here
# views=[HistogramView(instrument_name="http.server.duration", aggregation=BucketAggregation([0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 10]))]
# For basic counter/gauge, default aggregation is fine.
metric_provider.set_meter_provider(metric_provider)

# Get a tracer and meter
tracer = trace.get_tracer(__name__)
meter = metric_provider.get_meter(__name__)

# Create custom metrics
request_counter = meter.create_counter(
    name="my_flask_app_requests_total",
    description="Total number of requests to my Flask app",
    unit="1"
)
request_duration_histogram = meter.create_histogram(
    name="my_flask_app_request_duration_seconds",
    description="Duration of requests to my Flask app",
    unit="s"
)

app = Flask(__name__)

# Auto-instrument Flask and requests libraries
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

@app.route("/")
def hello_world():
    request_counter.add(1, {"endpoint": "/"}) # Increment custom counter

    with tracer.start_as_current_span("hello_world_endpoint") as span:
        span.set_attribute("http.method", request.method)
        span.set_attribute("http.target", request.path)

        # Simulate some work
        time.sleep(0.05)

        # Call another internal service (will be traced if instrumented)
        try:
            response = requests.get("http://localhost:5001/internal") # Assume another service running
            span.set_attribute("internal_service.status_code", response.status_code)
        except requests.exceptions.ConnectionError:
            span.set_attribute("internal_service.error", "Connection refused")

        duration = time.time() - span.start_time / 1_000_000_000 # convert ns to seconds
        request_duration_histogram.record(duration, {"endpoint": "/"}) # Record custom histogram

        return "
Hello, World!
"

@app.route("/internal")
def internal_endpoint():
    with tracer.start_as_current_span("internal_endpoint"):
        time.sleep(0.02) # Simulate some internal work
        return "Internal service data"

if __name__ == "__main__":
    app.run(port=5000, debug=True)

This application, when run, will send traces and metrics to an OpenTelemetry Collector running on `localhost:4317`. The Collector then forwards this data to Prometheus (for metrics) and a tracing backend like Jaeger or Tempo (for traces).

Integrating the Trio: A Unified Observability Stack

The real power of Prometheus, Grafana, and OpenTelemetry emerges when they are integrated into a cohesive observability stack. This integration allows you to leverage the strengths of each tool, providing a holistic view of your system’s health and performance.

The Big Picture Architecture

Let’s visualize the end-to-end flow of telemetry data:

+----------------------------------+   OpenTelemetry  +--------------------------------+
|     Your Application(s)          |   Protocol (OTLP)  |     OpenTelemetry Collector    |
| - OTel SDK (Python, Java, Go...) | -----------------> | - Receives OTLP (gRPC/HTTP)  |
| - Auto-instrumentation           |                    | - Processes (batch, filter)    |
|   (Flask, Spring Boot, Gin...)   |                    | - Exports telemetry data       |
| - Generates:                     |                    +--------------------------------+
|   - Traces (Spans, Context)      |                            |        |         |
|   - Metrics (Counters, Histograms)|                            |        |         |
|   - Logs (future/experimental)   |                            V        V         V
+----------------------------------+                OTLP   Prometheus   OTLP     OTLP
                                                    (Metrics) Exporter  (Traces) (Logs)
                                                        |        |        |        |
                                                        V        V        V        V
+----------------------------+  Scrapes                +----------------+ +--------------+ +--------------+
|   Prometheus Server        | <--------------------- | OTel Collector | | Jaeger/Tempo | | Loki (Logs)  |
| - Scrapes OTel Collector   |                        | (Prometheus    | | (Trace DB)   | | (Log DB)     |
|   (Prometheus receiver)    |                        | Exporter)      | +--------------+ +--------------+
| - Stores OTel metrics      |                        +----------------+
| - Evaluates Alert Rules    |
+----------------------------+
       |             |
       V             V
[Grafana (Prometheus Data Source)]   [Alertmanager] --> [Slack/Email/PagerDuty]
       |
       V
[Grafana Dashboards (Metrics, Logs, Traces)]
[Grafana Explore (Metrics, Logs, Traces)]
[Grafana Alerting (from any data source)]

Explanation of the Flow:

1. Instrumentation: Your applications are instrumented using OpenTelemetry SDKs and auto-instrumentation libraries. This code generates metrics (e.g., request counts, durations), traces (e.g., function calls, HTTP requests), and potentially logs.

2. OTLP Export: The OpenTelemetry SDKs in your applications are configured to export this telemetry data using the OpenTelemetry Protocol (OTLP) to a centralized OpenTelemetry Collector.

3. OpenTelemetry Collector: This acts as a universal agent/gateway. It receives data via OTLP, processes it (e.g., adds common attributes, filters sensitive data, batches data), and then exports it to different backends based on its configuration.

4. Metrics to Prometheus: For metrics, the OTel Collector uses a Prometheus exporter (or Prometheus receiver, allowing Prometheus to scrape the collector directly). Prometheus then scrapes these metrics, stores them, and uses them for alerting.

5. Traces to Jaeger/Tempo: For traces, the OTel Collector exports them to a dedicated tracing backend like Jaeger (