← Back to Blog

Observability in Microservices: Prometheus, Grafana, and Loki

By Stanley Hayford ·

Observability in Microservices: Prometheus, Grafana, and Loki

The Three Pillars of Observability

  1. Metrics: Numeric measurements over time (Prometheus)
  2. Logs: Detailed event records (Loki)
  3. Traces: Request flow across services (Tempo/Jaeger)

Prometheus Metrics

Instrument your Go services with Prometheus client library:

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "Duration of HTTP requests",
        Buckets: prometheus.DefBuckets,
    },
    []string{"method", "path", "status"},
)

func init() {
    prometheus.MustRegister(requestDuration)
}

Grafana Dashboards

Grafana connects to Prometheus as a data source and provides rich visualization. Create dashboards for:

  • Request rate (QPS)
  • Error rate (5xx responses)
  • Latency percentiles (p50, p95, p99)
  • Resource utilization (CPU, memory)

Loki for Logs

Loki is like Prometheus but for logs. It indexes labels, not log content, making it extremely efficient.

Alerting

Set up alerts for critical metrics:

  • Error rate > 1% over 5 minutes
  • P99 latency > 500ms
  • Pod restart count increasing
  • Disk usage > 80%

The key to good observability is not collecting everything — it's collecting the right things and making them actionable.

Tags: Observability, Prometheus, Grafana, DevOps