Setting Up Observability with OpenTelemetry in Go Apps on AWS EKS

By /

Listen to this Post

Featured Image
Observability is crucial for modern applications, especially in microservices architectures. OpenTelemetry has emerged as the leading standard for implementing observability, including distributed tracing. Below, we explore how to set up tracing for Go applications running on AWS EKS.

Understanding Tracing in OpenTelemetry

A trace represents the end-to-end flow of a request through your system, composed of multiple spans (individual operations or steps). In microservices, each service typically contributes a span to the trace.

Key Components

  • Trace ID: Unique identifier for the entire request flow.
  • Span ID: Identifies individual operations within the trace.
  • Context Propagation: Ensures trace continuity across services.

Instrumenting a Go Application with OpenTelemetry

Here’s how to instrument a Go app for tracing:

1. Install OpenTelemetry SDK

go get go.opentelemetry.io/otel \
go.opentelemetry.io/otel/trace \
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
go.opentelemetry.io/otel/sdk/resource \
go.opentelemetry.io/otel/sdk/trace \
go.opentelemetry.io/otel/propagation

2. Initialize Tracer Provider

package main

import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
"google.golang.org/grpc"
)

func initTracer() (sdktrace.TracerProvider, error) {
exporter, err := otlptracegrpc.New(
context.Background(),
otlptracegrpc.WithInsecure(),
otlptracegrpc.WithEndpoint("otel-collector:4317"),
)
if err != nil {
return nil, err
}

tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter),
sdktrace.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceNameKey.String("go-app"),
)),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{},
propagation.Baggage{},
))
return tp, nil
}

3. Create Spans in Your Code

func main() {
tp, err := initTracer()
if err != nil {
log.Fatal(err)
}
defer func() {
if err := tp.Shutdown(context.Background()); err != nil {
log.Fatal(err)
}
}()

ctx := context.Background()
tr := otel.Tracer("example-tracer")

ctx, span := tr.Start(ctx, "main-operation")
defer span.End()

// Your application logic here
}

4. Deploy OpenTelemetry Collector on EKS

Deploy the OpenTelemetry Collector as a sidecar or DaemonSet in your EKS cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
spec:
replicas: 1
selector:
matchLabels:
app: otel-collector
template:
metadata:
labels:
app: otel-collector
spec:
containers:
- name: otel-collector
image: otel/opentelemetry-collector
args: ["--config=/etc/otel-config.yaml"]
ports:
- containerPort: 4317

5. Configure AWS Distro for OpenTelemetry (ADOT)

If using AWS-managed OpenTelemetry:

aws eks create-addon --cluster-name my-cluster --addon-name adot --addon-version v0.60.0-eksbuild.1

You Should Know

  • Viewing Traces: Use AWS X-Ray or Jaeger for visualization.
  • Auto-Instrumentation: Libraries like `otelgin` for Gin or `otelhttp` simplify middleware integration.
  • Sampling: Configure sampling to reduce overhead: 
    sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.5)))
  • Kubernetes Annotations: Use pod annotations for automatic instrumentation in EKS.

What Undercode Say

Observability is not optional in distributed systems. OpenTelemetry simplifies tracing, but proper instrumentation requires:
– Context Propagation: Ensure headers (traceparent) are passed between services.
– Resource Attributes: Tag traces with service.name, k8s.pod.name.
– Performance Overhead: Balance sampling rates to avoid excessive data.

Essential Linux & AWS Commands

 Check OpenTelemetry Collector logs
kubectl logs -l app=otel-collector -n monitoring

List traces in AWS X-Ray
aws xray get-trace-summaries --start-time $(date -v-1d +%s) --end-time $(date +%s)

Enable debug logging in Go OTEL
export OTEL_LOG_LEVEL=debug

Verify EKS cluster add-ons
aws eks describe-addon --cluster-name my-cluster --addon-name adot

Expected Output

A fully instrumented Go application emitting traces to AWS X-Ray or Jaeger, with correlated spans across microservices.

Prediction

As cloud-native apps grow, OpenTelemetry will dominate observability, replacing vendor-specific agents. Expect deeper AWS integrations, including AI-driven anomaly detection in traces.

Relevant URL: OpenTelemetry Go Documentation

References:

Reported By: Darryl Ruggles – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅

Join Our Cyber World:

💬 Whatsapp | 💬 Telegram

Related Posts: