Skip to main content

Signing Key Rotation

This guide covers on-demand signing key rotation using the spirlctl trust-domain deployment keyset command family, including when to use each command and how to sequence them correctly.

Understanding Signing Keysets​

A signing keyset bundles the X.509 and JWT signing keys for a single trust domain deployment. At any point a deployment holds one or more keysets, each in one of three states:

  • PREPARED — in the trust bundle and trusted for verification, but not actively signing new SVIDs.
  • ACTIVE — the keyset currently signing all newly issued SVIDs. Exactly one keyset per deployment is ACTIVE at a time.
  • TAINTED — marked for retirement. Agents proactively renew X.509 SVIDs signed by a tainted keyset. JWT SVIDs continue to be served until their normal lifetime expires.
info

For background on signing keys themselves see the Signing Key Rotation guide.

Listing Keysets​

Inspect the current state of all signing keysets for a deployment:

spirlctl trust-domain deployment keyset list \
  --trust-domain example.com \
  --deployment-name default

Example output:

Deployment: default
Signing Keys:
  Type   Keyset ID      Issued At                      Expires At                     State
  X.509  ks_def5678uvw  2025-12-30 00:00:00 +0000 UTC  2026-03-30 00:00:00 +0000 UTC  Prepared
  JWT    ks_def5678uvw  2025-12-30 00:00:00 +0000 UTC  2026-03-30 00:00:00 +0000 UTC  Prepared
  X.509  ks_abc1234xyz  2025-10-15 09:23:41 +0000 UTC  2026-01-13 09:23:41 +0000 UTC  Active
  JWT    ks_abc1234xyz  2025-10-15 09:23:41 +0000 UTC  2026-01-13 09:23:41 +0000 UTC  Active

This command is read-only and safe to run at any time. Each keyset appears as two rows — one for its X.509 signing key and one for its JWT signing key. Use --output json (or -o json) for machine-consumable output.

Preparing a New Keyset​

Generate a new keyset and inject it into the trust bundle:

spirlctl trust-domain deployment keyset prepare \
  --trust-domain example.com \
  --deployment-name default

Example output:

Keyset ks_def5678uvw prepared.

The new keyset is now PREPARED — included in the trust bundle and trusted for verification, but not yet signing. Note the keyset ID from the output; you will need it for the next step.

Activating a Keyset​

Promote a PREPARED keyset to ACTIVE. The previously active keyset steps down to PREPARED:

spirlctl trust-domain deployment keyset activate ks_def5678uvw \
  --trust-domain example.com \
  --deployment-name default

Example output:

Keyset ks_def5678uvw activated.
warning

Only a PREPARED keyset can be activated. Attempting to activate a keyset in any other state returns an error.

Tainting a Keyset​

Mark a keyset as TAINTED to signal that it should be retired. The keyset remains in the trust bundle so existing SVIDs continue to verify, but agents proactively renew any X.509 SVIDs it has signed:

spirlctl trust-domain deployment keyset taint ks_abc1234xyz \
  --trust-domain example.com \
  --deployment-name default

Example output:

Keyset ks_abc1234xyz tainted.
warning

Only a PREPARED keyset can be tainted. The currently ACTIVE keyset cannot be tainted directly — the signer rejects the request with FailedPrecondition. To retire the active keyset, first activate a successor (the active keyset drops to PREPARED), then taint it.

Propagation to agents is not immediate. After the control plane publishes the updated bundle, agents pick it up on their next bundle refresh, then apply up to 60 seconds of per-stream jitter before renewing X.509-SVIDs. In typical fleets, allow 2 to 5 minutes after taint before concluding renewal is complete. JWT SVIDs do not proactively renew; they continue to be served until their normal lifetime expires (24 hours by default).

Removing a Keyset​

Evict a TAINTED keyset from the trust bundle. Any SVIDs still signed by it will no longer verify:

spirlctl trust-domain deployment keyset remove ks_abc1234xyz \
  --trust-domain example.com \
  --deployment-name default

Example output:

Keyset ks_abc1234xyz removed.
danger

Removing a keyset is permanent and cannot be undone. To introduce a new keyset after removal, start over from prepare. Before removing a TAINTED keyset, allow enough time for X.509 renewal to complete across your fleet to avoid verification failures.

Any keyset that is not currently ACTIVE can be removed.

Concurrency and Timeouts​

All mutation commands (prepare, activate, taint, remove) are blocking with a 120-second server-side timeout. The command does not return until the keyset mutation and bundle publication are complete, or the timeout elapses. Commands do not retry automatically: If a command fails, run list to observe the resulting state before deciding whether to retry.

Two concurrency rules apply:

  1. Two on-demand operations on the same deployment cannot run concurrently. The second command fails immediately. Wait for the first to complete before retrying.

  2. An on-demand operation and a scheduled automatic rotation can race. If both attempt to mutate the keyset, whichever commits second fails with an error. Run list to observe the resulting state, then decide whether to retry.

Concurrent operations on different deployments, even within the same trust domain, are independent and never conflict.

Troubleshooting​

"Command timed out" / connection interrupted​

Cause: The 120-second server-side timeout elapsed, or the connection was interrupted before the server returned. The server-side operation is also bounded by its own execution timeout, so a timed-out command is not left running indefinitely in the background.

Solution: Run list to check the current state — the operation may have already completed. If the change is not reflected the command is safe to re-run.

"The request failed due to an internal error, please try again."​

This generic error covers two distinct situations that are not distinguishable from the error message alone.

Concurrent operation conflict: Two on-demand commands targeting the same deployment cannot run at the same time. If a second command is submitted while the first is still in flight, it fails with this error.

Key ring version mismatch: An automatic rotation and your on-demand command raced and both attempted to mutate the keyset. The second commit failed.

Solution for both: Run list to see the current keyset states. If the intended mutation already happened, no further action is needed. If it did not, wait a moment for any in-flight operation to settle and retry.

"Error: cannot taint active key set" / "Error: cannot remove active key set" / "Error: cannot activate tainted key set"​

Cause: You attempted a state transition not permitted by the lifecycle.

Solution: Run list to check the keyset's current state, then use the appropriate command:

CommandRequires the keyset to be in
activatePREPARED
taintPREPARED (activate a successor first if the keyset is currently ACTIVE)
removePREPARED or TAINTED (any non-ACTIVE keyset)

Agents are not renewing after taint​

Allow 2 to 5 minutes in typical fleets before concluding something is wrong. If renewal does not occur:

  • Confirm the agent version supports taint-aware renewal. Older agents ignore the tainted-keyset list in the bundle.
  • Check agent logs for a bundle refresh event after the taint.
  • JWT-SVIDs are not proactively renewed when the keyset is tainted. They will renew at their expiry-driven rates.

A workload failed verification after remove​

Cause: The removed keyset is evicted from the bundle, so any SVID signed by it fails verification. This is the intended outcome of remove.

Solution: Affected workloads recover once they fetch a fresh SVID signed by the current active keyset. To prevent this, always allow X.509 renewal enough time to propagate before running remove.

Best Practices​

Rotation sequencing​

  1. Inspect state before you start — run list to confirm the current keyset states before beginning any rotation.
  2. Follow the sequence — complete each step before advancing: prepare → activate → taint → wait → remove. Skipping the wait between taint and remove risks workloads holding unverifiable SVIDs.
  3. Allow taint propagation time — give agents at least 2 to 5 minutes after taint before running remove. For large fleets or deployments with JWT SVID consumers, allow longer.

Command Reference​

CommandDescription
trust-domain deployment keyset listList signing keysets and their states
trust-domain deployment keyset prepareGenerate a new PREPARED keyset
trust-domain deployment keyset activatePromote a PREPARED keyset to ACTIVE
trust-domain deployment keyset taintMark a PREPARED keyset as TAINTED
trust-domain deployment keyset removeEvict a TAINTED keyset from the bundle

Flags​

FlagApplies toDescription
--trust-domainAll commandsTrust domain name (e.g. example.com)
--deployment-nameAll commandsDeployment name within the trust domain (e.g. default)
--output / -olist onlyOutput format: pretty (default) for human-readable output, json for structured JSON.

Next Steps​