Observability — Tracing

Observability — Tracing

Project-agnostic. Part of the Observability suite. Covers: OpenTelemetry tracing, Micrometer-OTEL bridge, automatic + custom spans, propagation across services, MDC correlation, NewRelic OTEL exporter, NewRelic Java agent.

Replace com.example.platform / platform-service placeholders with your project’s actual package and service name when applying patterns.

Companion files:

• observability.md — index + three-pillars overview.
• obs-metrics.md — Micrometer + Prometheus + decorator metrics.
• obs-logging.md — structured logging with trace correlation.
• obs-recipes.md — step-by-step + dashboards + alerting + checklist.

6. OpenTelemetry (OTEL) Tracing

How Micrometer Tracing Bridge Works

The project uses Micrometer Tracing Bridge to OpenTelemetry (micrometer-tracing-bridge-otel). This means:

1. Micrometer’s ObservationRegistry creates observations (spans).
2. The bridge translates Micrometer observations into OTEL spans.
3. The OTEL exporter (opentelemetry-exporter-otlp) sends spans to an OTEL Collector.
4. Spring Boot auto-configures everything — no manual span creation needed for HTTP, JDBC, Kafka.

What Is Automatically Traced

Spring Boot with Micrometer Tracing auto-instruments:

• HTTP requests (inbound via Spring MVC, outbound via RestClient/WebClient)
• JDBC queries (via DataSource proxy)
• Kafka produce/consume (via Spring Kafka instrumentation)
• Redis operations (via Spring Data Redis)
• Scheduled tasks (via @Scheduled instrumentation)

No additional code is needed for these. The tracing bridge handles it transparently.

OTEL Collector Configuration

Add this to application.yml (bootstrap module):

1
management:
2
  tracing:
3
    enabled: true
4
    sampling:
5
      probability: ${MANAGEMENT_TRACING_SAMPLING_PROBABILITY:1.0}
6
  otlp:
7
    tracing:
8
      endpoint: ${MANAGEMENT_OTLP_TRACING_ENDPOINT:http://localhost:4318/v1/traces}

management.otlp.tracing.endpoint is the canonical Spring Boot 3.x property bound by the micrometer-tracing-bridge-otel autoconfiguration (default http://localhost:4318/v1/traces). spring.application.name is exported as the service.name resource attribute automatically — do NOT use the OTEL-SDK otel.exporter.otlp.* / otel.resource.attributes properties here, as they are read by the OTEL SDK autoconfigure path, not by the Spring Boot bridge.

Sampling Configuration

Override via environment variable: MANAGEMENT_TRACING_SAMPLING_PROBABILITY=0.1

W3C Trace Context — Canonical Reference

The W3C Trace Context spec (w3.org/TR/trace-context/) defines two HTTP headers — traceparent and tracestate — that propagate a distributed trace across services, brokers, and gateways. Spring Boot 3+ (Micrometer Tracing) defaults to W3C; B3 (Zipkin legacy) remains available for backward compatibility.

traceparent format

1
traceparent: 00-<trace-id>-<parent-id>-<flags>
2
              ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3
              |  32 hex chars (16 bytes) — trace-id, identifies the whole trace
4
              |  16 hex chars (8 bytes)  — parent-id (span-id of the caller)
5
              |  02 hex chars (1 byte)   — flags, e.g. "01" = sampled
6
              02 hex chars version, currently always "00"

Example:

1
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Regex for validation (use exactly this — KrakenD reference plugin uses the same):

1
^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$

tracestate

Vendor-specific extensions, opaque to most services. Propagate as-is; never parse or reorder. Example:

1
tracestate: vendorA=opaqueValue,vendorB=otherValue

Propagation matrix

Micrometer Tracing automatically handles propagation when io.micrometer:micrometer-tracing-bridge-otel is on the classpath.

1
management:
2
  tracing:
3
    propagation:
4
      type: W3C        # default in Spring Boot 3+; alternative: B3 (Zipkin legacy)

Manual propagation (when auto-handling is unavailable)

For scheduled jobs, batch entry points, CLI tools, or when starting a span from an external trigger:

1
@Autowired Tracer tracer;
2
@Autowired Propagator propagator;
3

4
void runJob(String externalTraceparent) {
5
    Propagator.Getter<String> getter = (carrier, key) ->
6
        "traceparent".equalsIgnoreCase(key) ? carrier : null;
7
    Span span = propagator.extract(externalTraceparent, getter).start();
8
    try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
9
        // work — outbound calls within this scope are auto-propagated
10
    } finally {
11
        span.end();
12
    }
13
}

MDC integration

Micrometer auto-populates MDC with traceId and spanId once the W3C context is extracted:

1
[%X{traceId:-},%X{spanId:-}]

See the MDC Correlation section below for the full Logback pattern.

B3 fallback (Zipkin legacy)

When integrating with a Zipkin ecosystem:

1
management:
2
  tracing:
3
    propagation:
4
      type: B3

B3 uses different headers (X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId, X-B3-Sampled). Coexistence with W3C is configurable but adds complexity — prefer W3C-only unless integrating with legacy Zipkin services.

Sampling head vs tail

• Head-based (default): the entry point (gateway or service) decides at trace start; the flag in traceparent (01 = sampled, 00 = not) carries the decision to all downstream services.
• Tail-based: the OTEL collector samples after collecting the full trace, based on duration / error / attributes. Requires more storage; useful for catching error traces while keeping volume down.

Configure sampling probability per env:

Override via env var: MANAGEMENT_TRACING_SAMPLING_PROBABILITY=0.1.

Gateway propagation contract

The API gateway is the first hop where W3C context is created or accepted. See api-gateway.md §13 for the canonical gateway-side behavior — including the legacy Trace-Id fallback that bridges older clients to W3C.

Hard rules

• Services MUST extract Traceparent on every inbound request. If absent — log a warning and continue (do not fail). Generate a new trace.
• Outbound calls MUST carry Traceparent. Auto-propagation is non-negotiable; if it breaks, treat as a P2 bug.
• Logs MUST include traceId (via MDC) on every log line — see obs-logging.md.
• Business-level correlation: services that publish events via the transactional outbox ALSO stamp traceId into the EventMessage.Header.traceId field — survives DLQ redrive and broker-header loss.
• Never log full Traceparent body — log only the traceId portion (indices 3..34 inclusive of the string).

MDC Correlation (trace-id, span-id in logs)

Micrometer Tracing automatically populates SLF4J MDC with:

This project already uses these in the logging pattern (see application.yml):

1
logging:
2
  pattern:
3
    console: "%d{HH:mm:ss.SSS} [%thread] [trace-id=%X{traceId}] %-5level %logger{5} - %msg%n"
4
    file: "%d{HH:mm:ss.SSS} [%thread] [trace-id=%X{traceId}] %-5level %logger{5} - %msg%n"

This means every log line includes the trace-id, allowing you to correlate logs with traces in your observability backend.

Custom Spans (when needed)

For rare cases where you need a custom span (avoid unless necessary):

1
// Kotlin
2
import io.micrometer.observation.Observation
3
import io.micrometer.observation.ObservationRegistry
4

5
class SomeAdapter(private val observationRegistry: ObservationRegistry) {
6
    fun someOperation() {
7
        val observation = Observation.createNotStarted("custom.operation", observationRegistry)
8
            .lowCardinalityKeyValue("entity", "user")
9
            .start()
10
        try {
11
            // ... operation logic
12
        } catch (ex: Exception) {
13
            observation.error(ex)
14
            throw ex
15
        } finally {
16
            observation.stop()
17
        }
18
    }
19
}

1
// Java
2
import io.micrometer.observation.Observation;
3
import io.micrometer.observation.ObservationRegistry;
4

5
public class SomeAdapter {
6
    private final ObservationRegistry observationRegistry;
7

8
    public SomeAdapter(ObservationRegistry observationRegistry) {
9
        this.observationRegistry = observationRegistry;
10
    }
11

12
    public void someOperation() {
13
        Observation observation = Observation.createNotStarted("custom.operation", observationRegistry)
14
            .lowCardinalityKeyValue("entity", "user")
15
            .start();
16
        try {
17
            // ... operation logic
18
        } catch (Exception ex) {
19
            observation.error(ex);
20
            throw ex;
21
        } finally {
22
            observation.stop();
23
        }
24
    }
25
}

7. NewRelic with OTEL

Option A: NewRelic OTEL Exporter (Recommended)

NewRelic natively supports OTLP ingestion. This is the simplest approach — just point the OTEL exporter to NewRelic’s OTLP endpoint.

1
management:
2
  otlp:
3
    tracing:
4
      endpoint: ${MANAGEMENT_OTLP_TRACING_ENDPOINT:https://otlp.nr-data.net:4318/v1/traces}
5
      headers:
6
        api-key: ${NEW_RELIC_LICENSE_KEY}

management.otlp.tracing.endpoint and management.otlp.tracing.headers.* are the Spring Boot 3.x properties bound by the micrometer-tracing-bridge-otel bridge; spring.application.name is exported as service.name automatically. The OTEL-SDK otel.exporter.otlp.* properties are NOT read on this path.

Environment variables for production:

export NEW_RELIC_LICENSE_KEY=your-license-key-here
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net:4318
export OTEL_EXPORTER_OTLP_HEADERS="api-key=${NEW_RELIC_LICENSE_KEY}"

Advantages:

• No additional agent or sidecar needed.
• Uses the same OTEL exporter already in the project.
• Traces, metrics, and logs all flow through a single pipeline.
• Full compatibility with NewRelic APM, distributed tracing, and alerting.

NewRelic Endpoints by Region:

Option B: NewRelic Java Agent + OTEL (Alternative)

If you need full APM features (thread profiling, JVM metrics, code-level visibility), use the NewRelic Java Agent alongside OTEL:

1. Download the NewRelic Java Agent.
2. Add to JVM startup flags:

java -javaagent:/path/to/newrelic.jar \
     -Dnewrelic.config.app_name=platform-service \
     -Dnewrelic.config.license_key=${NEW_RELIC_LICENSE_KEY} \
     -jar platform-service.jar

3. Configure newrelic.yml:

1
common: &default_settings
2
  app_name: platform-service
3
  license_key: ${NEW_RELIC_LICENSE_KEY}
4
  distributed_tracing:
5
    enabled: true
6
  application_logging:
7
    enabled: true
8
    forwarding:
9
      enabled: true
10
      max_samples_stored: 10000
11
    metrics:
12
      enabled: true
13
    local_decorating:
14
      enabled: true

When to choose Option B:

• You need thread-level profiling and JVM internals.
• You want NewRelic’s auto-instrumentation for libraries not covered by Micrometer.
• Your organization already uses the NewRelic agent across other services.

NRQL Query Examples

Query custom metrics from this project in NewRelic:

1
-- Use case latency percentiles (p50, p95, p99)
2
SELECT percentile(duration, 50, 95, 99)
3
FROM Metric
4
WHERE metricName = 'usecase.operation'
5
  AND entity = 'user'
6
  AND operation = 'create_user'
7
SINCE 1 hour ago
8
TIMESERIES
9

10
-- Error rate by use case
11
SELECT count(*)
12
FROM Metric
13
WHERE metricName = 'usecase.operation.errors'
14
FACET entity, operation
15
SINCE 1 hour ago
16
TIMESERIES
17

18
-- Throughput by entity
19
SELECT rate(count(*), 1 minute)
20
FROM Metric
21
WHERE metricName = 'usecase.operation.calls'
22
FACET entity
23
SINCE 1 hour ago
24
TIMESERIES
25

26
-- Distributed trace search by trace-id
27
SELECT *
28
FROM Span
29
WHERE trace.id = 'your-trace-id-here'
30
SINCE 1 day ago
31

32
-- Slowest operations
33
SELECT max(duration)
34
FROM Span
35
WHERE service.name = 'platform-service'
36
FACET name
37
SINCE 1 hour ago
38
LIMIT 20