Skip to main content

Trust Domain Server Metrics

This guide covers metrics collection, configuration, and monitoring for Trust Domain Servers. Servers expose Prometheus-compatible metrics for SVID operations, attestations, gRPC performance, and resource utilization.

Enabling Metrics​

Trust Domain Servers expose metrics on a configurable port (default: 9090) via a /metrics endpoint.

Enable metrics in your Helm chart values file:

trust-domain-values.yaml
telemetry:
  enabled: true
  collectors:
    grpc:
      emitLatencyMetrics: true  # Optional: enables gRPC latency histograms (produces ~500 additional metrics series per instance)
  metricsAPI:
    port: 9090

Apply the configuration:

helm upgrade --install <trust-domain-name> \
  oci://ghcr.io/spirl/charts/spirl-server \
  --values trust-domain-values.yaml

See the Metrics Reference for the complete list of available metrics.

Verifying Metrics Endpoint​

Test that metrics are accessible:

# Locate a server pod (replace <namespace> with your trust domain deployment namespace)
kubectl -n <namespace> get po -l app.kubernetes.io/name=spirl-server

# Port-forward to a server pod
kubectl port-forward -n <namespace> spirl-server-0 9090:9090

# In a separate shell, query the metrics endpoint
curl http://localhost:9090/metrics

Example output:

# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.000582792
go_gc_duration_seconds{quantile="0.25"} 0.00085675
...

Key Metrics to Monitor​

gRPC Performance​

  • grpc_server_handled_total - Total gRPC requests completed
  • grpc_server_handling_seconds - Request latency histogram
  • grpc_server_started_total - Total gRPC streams started

Resource Utilization​

  • go_memstats_alloc_bytes - Current memory allocation
  • go_goroutines - Number of goroutines
  • process_cpu_seconds_total - CPU time

Kubernetes Runtime​

See Kubernetes Metrics for guidance on monitoring the Kubernetes runtime for issues.

HPA Prometheus metrics​

Monitor HPA decisions with Prometheus (requires kube-state-metrics):

# Current replica count (actual running pods)
kube_horizontalpodautoscaler_status_current_replicas{namespace="<namespace>"}

# Desired replica count (what HPA wants)
kube_horizontalpodautoscaler_status_desired_replicas{namespace="<namespace>"}

# Maximum replica limit
kube_horizontalpodautoscaler_spec_max_replicas{namespace="<namespace>"}

# Check if HPA is scaling (current != desired)
kube_horizontalpodautoscaler_status_desired_replicas{namespace="<namespace>"}
!=
kube_horizontalpodautoscaler_status_current_replicas{namespace="<namespace>"}

# Alert: HPA at max capacity (cannot scale further)
kube_horizontalpodautoscaler_status_current_replicas{namespace="<namespace>"}
>=
kube_horizontalpodautoscaler_spec_max_replicas{namespace="<namespace>"}

# HPA target metrics (e.g., CPU utilization target)
kube_horizontalpodautoscaler_status_target_metric{namespace="<namespace>"}
Out of autoscaling capacity

If kube_horizontalpodautoscaler_status_current_replicas equals kube_horizontalpodautoscaler_spec_max_replicas, the HPA has reached its maximum scaling limit and cannot add more replicas even if load continues to increase.

For HPA configuration and setup, see Trust Domain Server High Availability — Horizontal Pod Autoscaling.

Troubleshooting Server Metrics​

Metrics Endpoint Not Accessible​

Test the endpoint directly:

kubectl port-forward -n <namespace> <pod-name> 9090:9090
curl http://localhost:9090/metrics

Next Steps​