Signing Key Management

Introduction

Every SVID that Defakto issues, whether an X.509 certificate or a JWT, is cryptographically signed. That signature is what allows a receiving workload to trust the identity claim. The key that produces those signatures is therefore the most security-sensitive material in your deployment. How that key is generated, stored, protected, and eventually rotated determines the overall security posture of your workload identity infrastructure.

Defakto gives you a range of options. The default setup is secure and requires no external dependencies. As your security requirements grow, you can move key material progressively further from the Defakto server - first by protecting storage with an external key manager, and eventually by keeping signing keys inside a key manager entirely so they never touch server memory. You can also configure Defakto to obtain its signing authority from your organization's existing certificate authority, so that the X.509 SVIDs it issues chain up to a root your infrastructure already trusts.

This document explains each of these options: what they protect, what they trade off, and how they work. It is written for technical audiences who are familiar with the concepts of SVIDs, X.509, and JWTs, but who may not have a deep background in PKI mechanics. We will spend some time on those mechanics where they matter, because getting the mental model right makes the rest of the choices straightforward.

What Signing Keys Actually Do

Before discussing where signing keys live, it helps to be precise about what they are and how they are used.

The Defakto server maintains a signing key pair: a private key and a corresponding public key. When a workload requests an SVID, the server uses the private signing key to produce a cryptographic signature. That signature is what makes the SVID verifiable — any party that trusts the corresponding public key can confirm that Defakto issued the credential and that it has not been tampered with.

X.509 SVIDs and JWT SVIDs use the same underlying signing key, but the credentials themselves look quite different.

What an X.509 SVID actually contains

An X.509 SVID is not a single certificate. It is a small bundle of three things delivered together to the requesting workload:

• A certificate chain - typically two certificates. The first is the workload certificate itself, which contains the workload's SPIFFE ID and is valid for a short time. The second is the Defakto CA certificate, which was signed by the Defakto signing key and acts as the trust anchor that connects the workload certificate to something verifiable.
• A private key that belongs to the workload certificate - not the Defakto signing key. This is a fresh key pair generated by the Defakto server specifically for this workload, for this issuance. The workload uses this private key to authenticate itself in mTLS connections.

This distinction matters: the signing key that lives on the Defakto server, and the per-workload private key that gets delivered to a workload, are entirely separate keys. The Defakto signing key signs the certificate; the per-workload private key is what the workload actually uses day-to-day.

What a JWT SVID contains

A JWT SVID is simpler: it is a signed token. The Defakto server signs the token's payload using its signing key, and verifying parties confirm the signature using the corresponding public key from the Defakto trust bundle. There is no certificate chain, no per-workload private key, and no concept of a root CA. JWTs are self-contained signed assertions, and that is why everything discussed later in this document about upstream certificate authorities applies only to X.509 SVIDs - JWTs simply do not have certificate chains to root anywhere.

The Default Setup: Keys in Kubernetes

Out of the box, the Defakto server generates its own signing key pair and stores it in a Kubernetes Custom Resource Definition (CRD). The signing key material is never stored in plaintext. Before being written to the CRD, it is encrypted using a separate key called the Key Encrypting Key, or KEK. The KEK itself is stored in a Kubernetes Secret in the same namespace as your Defakto deployment.

The relationship between these two pieces of storage is worth understanding clearly, because it is the foundation for all the more advanced options.

How the KEK secret is created and protected

The KEK secret is created automatically, either by a one-time initialization job that runs as part of the Helm installation, or by the Defakto server itself on first startup if the job has not yet run. Either way, the secret is created in the same Kubernetes namespace as your Defakto deployment, whatever namespace you specified when installing the Helm chart.

Access to the KEK secret is governed by Kubernetes Role-Based Access Control (RBAC). The chart creates a dedicated Kubernetes service account for the Defakto server pods, and a Role that grants only that service account permission to read and update the specific named secret that holds the KEK. Nothing else in the cluster - no other service account, no workload in a different namespace - has access to it.

This means the security of the default setup rests on two things: the integrity of your Kubernetes RBAC configuration, and the protection of the nodes that run your Defakto server pods. If both of those are in good shape, the default setup offers a solid security baseline for most organizations.

Key rotation

Signing keys are rotated automatically on a schedule. During rotation, a new signing key is generated, encrypted with the KEK, and stored in the CRD alongside the old key. The old key is kept temporarily so that SVIDs issued under it remain valid until they expire. Once all SVIDs signed by the old key have naturally expired, the old key is retired. At no point during rotation does the Defakto server need to restart or interrupt SVID issuance.

The tradeoff to understand

In the default configuration, the KEK lives inside the Kubernetes cluster. This is convenient and operationally simple, but it means that the full security boundary is your cluster. An attacker who gains sufficient access to your Kubernetes control plane - specifically, the ability to read secrets in the deployment namespace - could in principle retrieve the KEK and use it to decrypt the signing key from the CRD.

For many organizations, this risk is acceptable given good cluster hygiene. For those with stricter requirements, the options described in the following sections move the trust boundary outside the cluster.

Moving the KEK to an External Key Manager

The first upgrade from the default setup is to replace the Kubernetes Secret with an external key management service — such as AWS KMS, or equivalent services from other major cloud providers. This option is available today.

The concept here is sometimes called "bring your own key" or envelope encryption, and it works as follows. Rather than storing the KEK as plaintext in a Kubernetes Secret, the Defakto server calls out to the key management service when it needs to encrypt or decrypt the signing key. The key management service holds the KEK internally and never exposes it. The server sends the signing key material to the KMS to be encrypted, receives back an opaque encrypted blob, and stores that blob in the CRD. To decrypt, the server sends the blob to the KMS and receives back the plaintext signing key.

The critical property: the KEK itself never leaves the key management service. Even the Defakto server never sees it.

What this buys you

The trust boundary shifts to the KMS provider. An attacker who can read Kubernetes secrets in your deployment namespace will find only an encrypted blob - useless without access to the key management service. The KEK itself is protected by the key management service's own access controls, audit logging, and hardware security guarantees (which vary by provider but are generally very strong for cloud KMS offerings).

The tradeoff

This option introduces a dependency on the external key management service at two points in time: startup, when the Defakto server decrypts the signing key to load it into memory, and key rotation, when a newly generated signing key must be encrypted before storage. In steady state - meaning when no rotation is happening and the server is already running - SVID issuance requires no calls to the key management service. The decrypted signing key lives in server memory and is used directly.

The practical implication is if the key management service is unreachable, a running Defakto server continues issuing SVIDs normally. But a server that needs to start up, or a rotation event that is triggered, will wait until the service is reachable. Plan for high availability of the key management service accordingly.

Signing Keys Inside the Key Manager

The next level of protection eliminates the window in which the signing key exists in Defakto server memory at all. Instead of the server decrypting the signing key and holding it in memory, the signing key is generated and stored inside the key management service, and every signing operation is a call to that service.

Defakto offers built-in integrations with key management services such as AWS KMS, and you can also connect any compatible hardware security module or key management system using the Key Manager Extension interface — a webhook-based protocol that Defakto calls for each key operation.

How signing works in this model

When a workload requests an X.509 SVID, the Defakto server constructs the certificate structure, produces a digest of the data to be signed, and sends that digest to the key manager with a request to sign it. The key manager returns a signature. The server then assembles the final certificate using that signature. At no point does the signing key travel to the server.

For JWT SVIDs, the same pattern applies: the server assembles the token payload, sends a digest to the key manager, receives a signature, and constructs the final JWT.

What this buys you

The signing key never exists outside the key management service, not even briefly in server memory. This provides the strongest available protection for signing key material and is the appropriate choice for environments with strict key custody requirements.

The tradeoff

Every SVID issuance now involves a call to the key management service. This has three practical implications you should plan for. First, the key management service must be highly available - any outage directly interrupts SVID issuance for your workloads. Second, it must handle the volume of signing requests your environment generates. Third, the per-call cost to a KMS service varies, and at scale this becomes an operating cost to factor in. Cloud KMS services such as AWS KMS generally meet the bar on all three dimensions, but if you are considering a custom key manager integration, these are the questions to answer before committing.

Understanding Certificate Chains and Trust Roots

Everything discussed so far - the default setup, the KEK in an external key manager, and signing keys inside a key manager - shares one characteristic: the X.509 SVIDs that Defakto issues are rooted in a certificate that Defakto generated itself. This section explains what that means, why it sometimes matters, and what you can do about it.

What "rooted in" means

An X.509 certificate does not stand alone. To verify it, a relying party needs to trace a chain of signatures from the certificate back to a root certificate authority that it trusts. Each certificate in the chain is signed by the one above it, and the chain terminates at a self-signed root certificate - one that signs itself.

A simple picture: your workload presents its SVID certificate. The verifying party asks: who signed this certificate? The answer is the Defakto CA certificate. Who signed the Defakto CA certificate? In the default setup, Defakto did - it signed its own CA certificate when it started up. That self-signed certificate is the trust anchor. For another party to trust your workload's SVID, they need to have the Defakto CA certificate in their trust store.

The diagram below shows this chain for a typical X.509 SVID.

When self-issued roots become a challenge

In the default configuration, the Defakto CA certificate is self-signed, meaning it is the root of its own chain. This is perfectly valid cryptographically, but it means any party that wants to verify SVIDs issued by Defakto must trust this specific certificate. In practice, that trust is distributed automatically to agents in your SPIFFE trust domain, so workloads within your Defakto deployment handle this without any manual configuration.

The situation becomes more complicated in two common scenarios.

The first is compliance. Some organizations have a requirement that all X.509 certificates issued within the company must chain up to a company-managed root CA - one whose key material is controlled by the security team, audited, and subject to the organization's certificate policy. A self-signed Defakto CA certificate does not satisfy this requirement.

The second is migration. If you have existing servers that terminate mTLS and validate client certificates against a specific trust store - one containing your company's CA but not the Defakto CA - then workloads presenting Defakto-issued SVIDs will fail validation. Getting SVIDs to chain up to the company CA eliminates this problem without requiring you to update trust stores on every existing server.

Upstream Authorities

Defakto supports plugging in an Upstream Authority - an external certificate authority that signs the Defakto CA certificate, creating a chain that roots in something your organization already controls. Built-in integrations are available for certificate services such as AWS Private CA, and you can connect any compatible CA using the Upstream Authority Extension interface.

It is important to clear up a misconception that comes up frequently: using an upstream authority does not mean the external CA is issuing private keys to workloads, and it does not mean the Defakto server needs the root CA's private key. The mechanics are more precise than that, and understanding them prevents a lot of confusion.

How the Defakto server gets a signing certificate from an upstream CA

The process begins with the Defakto server, not the upstream CA. When Defakto needs to establish or rotate its signing authority, it generates a new key pair locally. It then creates a Certificate Signing Request - a CSR - which is a standardized message that says, in effect: "Here is my public key, and here is what I want the certificate to say about me. Please sign it."

Before sending it, the Defakto server signs the CSR with its own newly generated private key. This proves to the upstream CA that the requester genuinely holds the private key corresponding to the public key in the request.

The Defakto server sends this CSR to the upstream CA. The upstream CA validates the request and, if it approves, signs the public key with its own private key. The result is a signed CA certificate for Defakto - one that chains up to the upstream CA's root. The upstream CA never generates a private key for Defakto, and it never needs to. Defakto's private signing key was generated by Defakto and stays there.

The diagram below shows this flow from beginning to end, through to the SVID that a workload ultimately receives.

The diagram below shows how this changes the certificate chain that a workload receives. With an upstream authority configured, the Defakto CA certificate becomes an intermediate rather than a trust anchor, and the chain gains a third level rooting in the upstream CA.

Key management and upstream authorities are independent choices

A point that often causes confusion: where the signing key lives and whether an upstream authority is configured are separate, independent choices. You can use an upstream authority regardless of which key management option you have chosen. The two axes do not interact.

The table below lists the available combinations:

Key management	Upstream authority	What it means
Default (Kubernetes)	None (self-signed)	Signing key in CRD, protected by KEK in Kubernetes Secret. SVIDs chain to Defakto self-signed root.
Default (Kubernetes)	Upstream CA configured	Same key storage, but the Defakto CA certificate is signed by your upstream CA. SVIDs chain to your company root.
KEK in external KMS	None (self-signed)	KEK held in cloud KMS, signing key in CRD as encrypted blob. SVIDs chain to Defakto self-signed root.
KEK in external KMS	Upstream CA configured	Strongest available today. KEK in cloud KMS, and SVIDs chain to your company root.
Signing keys in key manager	None (self-signed)	Signing key never in server memory. SVIDs chain to Defakto self-signed root.
Signing keys in key manager	Upstream CA configured	Maximum protection for key material, with SVIDs chaining to company root.

The diagram below shows what this looks like in practice when both are configured together — signing keys held in an external key manager, and SVIDs chaining to an upstream CA root.

As the diagram makes clear, the key manager and the upstream CA operate in separate lanes. They are configured independently and have no knowledge of each other.

Choosing the Right Configuration

The following questions can help orient the decision.

Do you have a compliance requirement that all certificates chain to a company-managed root CA? If yes, you will need an upstream authority configured, regardless of your key management choice.

Do you have existing servers that validate mTLS clients against a specific trust store? If those trust stores contain your company CA but not the Defakto CA, configuring an upstream authority may let you avoid updating trust stores everywhere.

Is your primary concern the protection of signing key material? If the default Kubernetes-based setup does not meet your key custody requirements, consider moving the KEK to a cloud KMS service as a first step. If your security posture demands that signing keys never reside in server memory, the key manager integration is the right direction.

How much operational complexity can you absorb? Each step up in key management introduces a dependency on an external service. The default setup has none; KEK in a cloud KMS introduces a dependency at startup and rotation time; signing keys in a key manager introduce a dependency on every SVID issuance. Each of these dependencies requires the external service to be highly available and correctly configured.

For most organizations starting out, the default configuration with careful Kubernetes RBAC hygiene is a solid baseline. Moving the KEK to a cloud KMS is a natural next step when you want stronger assurance that key material cannot be accessed from within the cluster. Upstream authorities address a different concern entirely - certificate chain policy - and can be adopted independently of either key management choice.

Summary

Defakto's signing key architecture gives you control over where key material lives and where certificate trust is rooted, without coupling those two concerns together. The default setup is operationally simple and secure for most environments. External key managers offer progressively stronger custody guarantees as your requirements demand. Upstream authorities integrate your SVID infrastructure into your existing PKI without disrupting how SVID issuance works internally.