In this guide (12 items)

Observability — Recipes & Dashboards

Observability — Recipes, Dashboards & Checklist

Project-agnostic. Part of the Observability suite. Covers: step-by-step recipe for adding observability to a new feature (port → use case → metrics adapter → chain registration → verify), Prometheus/Grafana dashboard patterns, key metrics to monitor, alerting rules, observability verification checklist.

Replace com.example.platform / platform-service placeholders with your project’s actual package and service name when applying patterns.

Companion files:

observability.md — index + three-pillars overview.
obs-metrics.md — metrics fundamentals.
obs-tracing.md — tracing.
obs-logging.md — logging.

9. Adding Observability to New Features (Step-by-Step)

When adding a new use case (for example, “DeleteUser”), follow these steps to include proper observability.

Step 1: Create the Inbound Port (application module)

1
package com.example.platform.port.inbound
2

3
import com.example.platform.port.ChainablePort
4
import java.util.UUID
5

6
fun interface DeleteUserPort : ChainablePort<DeleteUserPort> {
7
    fun execute(id: UUID)
8
}

1
// Java equivalent
2
package com.example.platform.port.inbound;
3

4
import com.example.platform.port.ChainablePort;
5
import java.util.UUID;
6

7
public interface DeleteUserPort extends ChainablePort<DeleteUserPort> {
8
    void execute(UUID id);
9
}

Step 2: Create the Use Case (application module)

1
package com.example.platform.usecase.user
2

3
import com.example.platform.port.inbound.DeleteUserPort
4
import com.example.platform.port.outbound.repository.UserRepositoryPort
5
import org.springframework.core.annotation.Order
6
import java.util.UUID
7

8
@Order(100)
9
class DeleteUserUseCase(
10
    private val userRepository: UserRepositoryPort,
11
) : DeleteUserPort {
12

13
    override fun execute(id: UUID) {
14
        userRepository.deleteById(id)
15
    }
16
}

1
// Java equivalent
2
package com.example.platform.usecase.user;
3

4
import com.example.platform.port.inbound.DeleteUserPort;
5
import com.example.platform.port.outbound.repository.UserRepositoryPort;
6
import org.springframework.core.annotation.Order;
7
import java.util.UUID;
8

9
@Order(100)
10
public class DeleteUserUseCase implements DeleteUserPort {
11

12
    private final UserRepositoryPort userRepository;
13

14
    public DeleteUserUseCase(UserRepositoryPort userRepository) {
15
        this.userRepository = userRepository;
16
    }
17

18
    @Override
19
    public void execute(UUID id) {
20
        userRepository.deleteById(id);
21
    }
22
}

Step 3: Create the Metrics Adapter (metrics-publisher module)

1
package com.example.platform.infrastructure.driven.adapters.metrics
2

3
import com.example.platform.infrastructure.driven.adapters.metrics.support.MetricRecorded
4
import com.example.platform.port.inbound.DeleteUserPort
5
import io.micrometer.core.instrument.MeterRegistry
6
import org.springframework.stereotype.Component
7
import java.util.UUID
8

9
@Component
10
class DeleteUserMetricsAdapter(
11
    meterRegistry: MeterRegistry,
12
) : MetricRecorded(meterRegistry),
13
    DeleteUserPort {
14

15
    private lateinit var delegate: DeleteUserPort
16

17
    override fun execute(id: UUID) =
18
        record("user", "delete_user") { delegate.execute(id) }
19

20
    override fun setDelegate(delegate: DeleteUserPort) {
21
        this.delegate = delegate
22
    }
23
}

1
// Java equivalent
2
package com.example.platform.infrastructure.driven.adapters.metrics;
3

4
import com.example.platform.infrastructure.driven.adapters.metrics.support.MetricRecorded;
5
import com.example.platform.port.inbound.DeleteUserPort;
6
import io.micrometer.core.instrument.MeterRegistry;
7
import org.springframework.stereotype.Component;
8
import java.util.UUID;
9

10
@Component
11
public class DeleteUserMetricsAdapter extends MetricRecorded implements DeleteUserPort {
12

13
    private DeleteUserPort delegate;
14

15
    public DeleteUserMetricsAdapter(MeterRegistry meterRegistry) {
16
        super(meterRegistry);
17
    }
18

19
    public void setDelegate(DeleteUserPort delegate) {
20
        this.delegate = delegate;
21
    }
22

23
    @Override
24
    public void execute(UUID id) {
25
        record("user", "delete_user", () -> {
26
            delegate.execute(id);
27
            return null;
28
        });
29
    }
30
}

Key points for the metrics adapter:

Annotated with @Component so Spring discovers it.
Extends MetricRecorded(meterRegistry) which provides record() and sets @Order(10).
Implements the same port interface (DeleteUserPort).
Uses setDelegate() from ChainablePort<T> (inherited via the port interface).
The record("user", "delete_user") call matches the naming convention: entity + operation.

Step 4: Register in the Chain (bootstrap module)

Add the new port to UseCaseConfig.kt:

1
@Bean
2
@Primary
3
fun deleteUserPort(): DeleteUserPort =
4
    buildChain(DeleteUserPort::class.java)

Step 5: Verify

After starting the application, confirm:

The /actuator/metrics/usecase.operation endpoint shows the new metric.

Hit the endpoint and verify counters increment:

curl http://localhost:8181/platform-service/actuator/metrics/usecase.operation?tag=operation:delete_user

Confirm Prometheus format at /actuator/prometheus includes:

1
usecase_operation_seconds_count{entity="user",operation="delete_user",...}
2
usecase_operation_calls_total{entity="user",operation="delete_user",success="true",...}

10. Dashboard and Alerting Recommendations

Prometheus/Grafana Dashboard Patterns

Use Case Performance Dashboard

Create a Grafana dashboard with these panels:

Panel 1: Latency Percentiles (p50, p95, p99)

1
# p50
2
histogram_quantile(0.50, rate(usecase_operation_seconds_bucket{application="platform-service"}[5m]))
3

4
# p95
5
histogram_quantile(0.95, rate(usecase_operation_seconds_bucket{application="platform-service"}[5m]))
6

7
# p99
8
histogram_quantile(0.99, rate(usecase_operation_seconds_bucket{application="platform-service"}[5m]))

Group by: entity, operation

Panel 2: Error Rate

1
# Error rate percentage per operation
2
rate(usecase_operation_calls_total{success="false",application="platform-service"}[5m])
3
/
4
rate(usecase_operation_calls_total{application="platform-service"}[5m])
5
* 100

Panel 3: Throughput (requests per minute)

1
rate(usecase_operation_calls_total{application="platform-service"}[5m]) * 60

Group by: entity, operation

Panel 4: Error Count by Type

1
increase(usecase_operation_errors_total{application="platform-service"}[5m])

Group by: entity, operation

Key Metrics to Monitor

Metric	Query	Purpose
Latency p50	`histogram_quantile(0.50, rate(usecase_operation_seconds_bucket[5m]))`	Median response time
Latency p95	`histogram_quantile(0.95, rate(usecase_operation_seconds_bucket[5m]))`	Tail latency
Latency p99	`histogram_quantile(0.99, rate(usecase_operation_seconds_bucket[5m]))`	Worst-case latency
Error rate	`rate(usecase_operation_calls_total{success="false"}[5m])`	Errors per second
Throughput	`rate(usecase_operation_calls_total[5m])`	Requests per second
Error ratio	`errors / total * 100`	Percentage of failing requests

Alert Thresholds

Alert	Condition	Severity	Action
High Latency (p95)	`p95 > 2s for 5 minutes`	Warning	Investigate slow operations
Critical Latency (p99)	`p99 > 5s for 5 minutes`	Critical	Page on-call
Elevated Error Rate	`error_rate > 5% for 5 minutes`	Warning	Check logs and traces
Critical Error Rate	`error_rate > 20% for 2 minutes`	Critical	Immediate investigation
Throughput Drop	`throughput < 50% of avg(1h) for 10 minutes`	Warning	Check upstream dependencies
Zero Throughput	`throughput == 0 for 5 minutes`	Critical	Service may be down
Health Check Failure	`up{job="platform-service"} == 0`	Critical	Service is not responding

Prometheus Alerting Rules Example

1
groups:
2
  - name: platform-service
3
    rules:
4
      - alert: HighLatencyP95
5
        expr: histogram_quantile(0.95, rate(usecase_operation_seconds_bucket{application="platform-service"}[5m])) > 2
6
        for: 5m
7
        labels:
8
          severity: warning
9
        annotations:
10
          summary: "High p95 latency on {{ $labels.entity }}.{{ $labels.operation }}"
11
          description: "p95 latency is {{ $value }}s (threshold: 2s)"
12

13
      - alert: CriticalErrorRate
14
        expr: |
15
          rate(usecase_operation_calls_total{success="false",application="platform-service"}[5m])
16
          / rate(usecase_operation_calls_total{application="platform-service"}[5m])
17
          > 0.20
18
        for: 2m
19
        labels:
20
          severity: critical
21
        annotations:
22
          summary: "Critical error rate on {{ $labels.entity }}.{{ $labels.operation }}"
23
          description: "Error rate is {{ $value | humanizePercentage }} (threshold: 20%)"
24

25
      - alert: ServiceDown
26
        expr: up{job="platform-service"} == 0
27
        for: 1m
28
        labels:
29
          severity: critical
30
        annotations:
31
          summary: "platform-service is down"
32
          description: "The service has been unreachable for more than 1 minute"

Observability Verification Checklist

Use this checklist when adding or reviewing observability for any feature: