High Availability Deployments
The Defakto platform is designed to support the availability needs of the most stringent environments. This guide discusses the design and configuration of the Defakto servers and agents to support a high availability deployment.
Trust Domain Servers
Deploying Trust Domain Servers with high availability is crucial to ensuring workloads in the trust-domain can continue to get and renew their SVIDs.
Trust Domain Deployment Architecture and Resilience
Trust Domain servers receive configuration from the Defakto Control Plane and store it locally in the Kubernetes cluster. Multiple replicas of the Trust Domain Server running on the same Kubernetes cluster access the single local configuration. This design makes it possible to scale and restart Trust Domain Servers independently of the Defakto Control Plane. However, if disconnected, note that any configuration changes will not propagate and metrics for the Trust Domain and its agents will not update in the console until the connection is re-established.
Trust Domain Server replicas are stateless. There is no leader election or inter-replica coordination. This architecture allows for any replica to serve any SVID issuance request in the trust domain. Therefore, increasing the number of Trust Domain Server replicas running in a Kubernetes cluster will increase the capacity of the agents to issue SVIDs. To distribute agent connections across replicas, a load balancer should be deployed in front of the Trust Domain Server replicas. See Deploy Trust Domain Servers for load balancer setup.
Single-cluster High Availability configuration
Two high availability settings are enabled out of the box.
Pod Disruption Budget (PDB) (trustDomainDeployment.podDisruptionBudget.enabled: true, trustDomainDeployment.maxUnavailable: 1): The chart creates a PDB that limits voluntary disruptions — node drains, cluster upgrades, rolling updates — to one pod at a time. This ensures Kubernetes will not evict a second pod until the first is back, preventing voluntary disruptions from cascading. The maxUnavailable: 1 default means at most one pod is voluntarily disrupted at a time; the minimum available replica count during voluntary disruptions is replicaCount - 1.
Graceful shutdown (trustDomainDeployment.deployment.drainDuration: 10s): When a pod receives a termination signal, it waits 10 seconds before closing its listening servers and beginning the shutdown process. This is to give enough time for any load balancers to re-sync with Kubernetes that the pod is no longer active, and that we don't reject any connections if a load balancer is slow to redirect traffic.
Increasing the replica count
The chart defaults to a single replica (replicaCount: 1), which provides no redundancy. For production, set this to at least 3:
trustDomainDeployment:
deployment:
replicaCount: 3
Three replicas are recommended rather than two because of how the Pod Disruption Budget interacts with failures. With maxUnavailable: 1, the minimum available replica count is replicaCount - 1. At two replicas, a simultaneous node failure and a voluntary disruption leaves you with zero. At three replicas, the same scenario leaves one replica serving traffic.
When deploying Trust Domain Servers to production, estimate how many agents will connect to those servers to ensure the load will be performant. spirl-perf is a tool that simulates load from Defakto agents and outputs metrics on the latency of the requests. See Running performance tests on the Trust Domain Server for more details.
Spreading replicas across nodes (multi-node clusters)
By default, the Helm chart sets no affinity rules, so the Kubernetes scheduler may place all replicas on the same node. If your cluster has multiple nodes, configure pod anti-affinity to ensure each replica lands on a different node:
trustDomainDeployment:
deployment:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: spirl-server
topologyKey: kubernetes.io/hostname
This rule prevents a single node failure from taking down all replicas simultaneously. It requires your cluster to have at least as many schedulable nodes as replicas. If not enough nodes are available, pods will remain Pending.
Spreading replicas across availability zones
For clusters that span multiple availability zones, add a topology spread constraint to distribute replicas evenly across zones:
trustDomainDeployment:
deployment:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/name: spirl-server
This requires your nodes to carry the topology.kubernetes.io/zone label. Most managed Kubernetes services (EKS, GKE, AKS) apply this automatically. When combined with the node anti-affinity rule above, no two replicas share a node and replicas are distributed evenly across zones.
Multi-regional active-active
Trust Domain Servers can also be scaled across multiple regions in an active-active configuration. Each region runs its own independent set of replicas, and all regions serve live traffic simultaneously. Each region is called a Trust Domain Deployment and must be registered with the Defakto control plane. Two options are recommended below to ensure agents failover to remote regions automatically.
Agent endpoint configuration
Agents can be configured with a prioritized list of endpoints to connect to. The first entry is the primary (typically the agent's local region) and the agent will prefer connecting to this region. When an agent cannot reach the primary endpoint, it works through the fallback list until it finds a reachable server. Every 30 minutes, it attempts to reconnect to the primary endpoint so agents return to their local region automatically once it recovers, without requiring a restart.
agent:
endpoint:
endpoints:
- "us-east.agent.example.com:443" # Primary: tried first
- "eu-west.agent.example.com:443"
- "ap-southeast.agent.example.com:443"
Global load balancer
Alternatively, place a global load balancer (such as AWS Global Accelerator, Google Cloud Global Load Balancer, or Azure Front Door) in front of your regional Trust Domain Server deployments. Configure the load balancer with health checks and regional failover so that agents connecting to a single global endpoint are routed to the nearest healthy region.
Both approaches can be used together by targeting the global load balancer and falling over to regional endpoints.
Resource management and autoscaling
Resource requests and limits
By default, the Helm chart does not configure resource requests or limits. Setting them explicitly is recommended for production deployments. Without them, the Kubernetes scheduler cannot make good placement decisions and pods run as BestEffort QoS, meaning they are the first to be killed under node memory pressure.
Resource requirements depend on your specific usage patterns, including attestation frequency, SVID rotation rates, number of agents, and API request volume. Start with conservative estimates and adjust based on observed metrics. Ensure sufficient headroom to support failovers.
trustDomainDeployment:
deployment:
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "1000m"
memory: "512Mi"
For tested starting-point values and guidance on adjusting based on observed metrics, see Resource Sizing.
Horizontal Pod Autoscaling
If your agent count varies significantly over time, enable the HPA to scale replica count automatically based on CPU utilization.
Trust Domain Servers are written in Go, which uses efficient garbage collection and memory management. CPU is a more reliable scaling signal than memory for Go applications, because memory usage tends to remain relatively stable regardless of load.
trustDomainDeployment:
deployment:
hpa:
enabled: true
minReplicas: 3
maxReplicas: 7
targetCPUUtilizationPercentage: 70
Set minReplicas to at least 3 to preserve the availability guarantee during scale-down.
Failover headroom: In a multi-regional deployment, a region going offline shifts its agent connections to the surviving regions. If surviving replicas are already near the CPU target, they will be overloaded until the HPA adds new pods which can take a minute or more. To absorb a regional failover without degraded performance, lower the CPU target (targetCPUUtilizationPercentage) to keep spare capacity on each running replica. For example, if you have two regions of equal size and need to survive a full regional failure, set the target to roughly 50% so each region can handle double its normal load. If some degradation during failover is acceptable, a higher target (such as the default 70%) reduces cost in steady state while still allowing the HPA to catch up.
HPA requirements
For HPA to function, you must:
-
Set resource requests — HPA calculates utilization as a percentage of requested resources:
resources:requests:cpu: "250m" -
Deploy metrics-server — HPA requires the metrics-server to retrieve pod resource usage:
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/helm upgrade --install metrics-server metrics-server/metrics-server -
Verify metrics availability:
kubectl top pods -n <namespace>
Monitoring HPA behavior
# View HPA status
kubectl get hpa -n <namespace>
# View detailed HPA status
kubectl describe hpa <hpa-name> -n <namespace>
# Monitor HPA events
kubectl get events -n <namespace> --field-selector involvedObject.kind=HorizontalPodAutoscaler
Example output:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
spirl-server Deployment/spirl-server 45%/70% 3 7 3 5d
The TARGETS column shows current vs target utilization (45% current, 70% target).
For Prometheus queries to alert on HPA scaling and capacity, see Server Metrics — HPA Prometheus metrics.
Best practices
- Start conservatively — Begin with the starting-point values from Resource Sizing and adjust based on observed metrics.
- Monitor utilization — Track CPU and memory usage against requests and limits. Check for CPU throttling, which can increase request latency.
- Avoid memory-based HPA — For Go applications, CPU is a more reliable scaling signal.
- Set a reasonable
maxReplicas— Prevent runaway scaling that exhausts cluster capacity.
Key manager considerations
All replicas in a Trust Domain Deployment share the same signing keys. Every replica must be able to read from the same key manager backend.
Default CRD backend
By default, signing keys are stored encrypted in a Kubernetes Custom Resource Definition (CRD). The Trust Domain Server reads this CRD via the Kubernetes API server on startup: if the API server is unavailable, a replica that needs to load its signing key cannot start. Replicas that are already running continue to sign SVIDs normally, because the key is held in memory and does not require the API server after initial load.
An Kubernetes API server outage has two additional effects:
- Configuration updates from the Defakto Control Plane stop propagating to replicas. Changes made in the console will not take effect until connectivity is restored.
- The cluster cannot scale or schedule new Trust Domain Server pods.
For production deployments using the default CRD backend, ensure your API server is highly available. Most managed Kubernetes services (EKS, GKE, AKS) handle this automatically.
External key managers
AWS KMS, Azure Key Vault, and GCP Cloud KMS remove the Kubernetes API server dependency for key loading. They are managed services with their own availability guarantees, independent of your Kubernetes control plane. For multi-regional deployments, ensure that replicas in each region have network connectivity and the necessary IAM or RBAC permissions to reach the key manager endpoint.
See Key Manager Configuration for setup instructions.
Verifying your setup
After deploying with the configuration above, confirm the following.
Pod distribution:
kubectl get pods -n <namespace> -o wide
Each pod should land on a different node. If using topology spread constraints, verify that pods are distributed across zones. Check the topology.kubernetes.io/zone label on the nodes each pod was scheduled to.
PDB status:
kubectl get pdb -n <namespace>
The ALLOWED DISRUPTIONS column should show at least 1 when all replicas are healthy.
Failover behavior:
Delete one pod and confirm that agents continue renewing SVIDs without errors:
kubectl delete pod <pod-name> -n <namespace>
Watch agent logs for reconnection events. Agents should connect to a surviving replica within seconds and resume SVID renewal without interruption.
Agents
Agents run as a Kubernetes DaemonSet, so there is one pod per node. Because workloads depend on the agent for SVID issuance, agent restarts and upgrades are handled gracefully to avoid any disruptions to workloads.
Rolling upgrades
By default the agent DaemonSet is configured to do rolling upgrades with maxUnavailable: 0 and maxSurge: 10%:
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 10%
With this updateStrategy, Kubernetes starts the new agent pod on the node before terminating the old one. The old pod remains active and serving workloads while the new pod initializes and passes its readiness probe. Only then is the old pod terminated.
When agent.hostNetwork: true is set, surge cannot be used. Two agent pods on the same node would conflict on the host network. In this mode, the DaemonSet falls back to a sequential restart so that the old pod is terminated before the new one starts. Workloads on that node will lose access to the Workload API during the transition.
Avoid hostNetwork: true in production unless your platform requires it.
What happens when an agent restarts
The agent serves the SPIFFE Workload API over a Unix domain socket. By default, upgrades use a surge rolling strategy, where Kubernetes starts the new agent pod before terminating the old one. The socket handoff between them is designed to minimize the reconnect window:
- The new agent starts, loads its configuration and signing keys, and passes its readiness probe. At this point it is ready to serve workloads.
- The new agent unlinks the socket path from the filesystem by deleting the filesystem handle. However, the file descriptor and client connections on the old agent's socket remain intact but it is no longer reachable by path.
- The new agent binds a new socket at the same path so that new workload connections immediately go to the new agent.
- The old agent receives SIGTERM once the new agent is ready and begins graceful shutdown. It cancels all in-flight gRPC contexts but waits up to 10 seconds for handlers to drain before hard-stopping.
Workloads with open streaming RPCs (FetchX509SVID, FetchX509Bundles, and FetchJWTBundles) will receive a stream cancellation (gRPC status UNAVAILABLE) and be closed. Workloads are expected to retry when requesting SVIDs and trust bundles. Most SPIFFE SDKs handle this automatically.
Because the new agent is already running and listening at the socket path before the old agent begins draining, reconnecting workloads find a live socket immediately rather than hitting "connection refused". The agent also proactively refreshes X.509 SVIDs at the midpoint of their validity period. If the agent is unavailable for longer than the remaining SVID validity period, workloads will have stale certificates. In practice, the default one-hour SVID TTL means a brief agent restart during a rolling upgrade poses no risk of expiry.
Minimizing disruption
-
Use the default update strategy. Do not set
hostNetwork: trueunless required by your platform. Surge upgrades give the new agent time to become ready before any existing connections are disrupted. -
Size SVID TTL with headroom. Keep the SVID TTL long enough that a brief agent outage (tens of seconds) does not cause expiry. The default one-hour TTL is sufficient for normal operations, but if the system must be resilient against longer outages, set the TTL accordingly.
-
Write retry-tolerant workloads. Workloads consuming the Workload API directly should implement exponential backoff and reconnect on
UNAVAILABLEerrors. Most SPIFFE SDKs handle this automatically. -
Monitor agent availability. Track
kube_daemonset_status_number_readyfrom kube-state-metrics to alert when agents are unavailable longer than expected:kube_daemonset_status_number_ready{daemonset="spirl-agent"}< kube_daemonset_status_desired_number_scheduled{daemonset="spirl-agent"}