Integrating Defakto with Istio

Defakto Agents also implement the Envoy SDS API over the same socket as the SPIFFE Workload API and can be used directly with Envoy or with Istio. Istio has strict requirements on the format of the SPIFFE ID (spiffe://trust-domain/ns/namespace-name/sa/service-account-name), the location and name of the SDS API socket (/run/secrets/workload-spiffe-uds/socket), and the name of the volume the socket is mounted from (workload-socket). Meeting those requirements takes two things, set in different places. The platform the cluster is registered with in Defakto sets the SPIFFE ID format. The agent's socket settings set the socket's location, name and volume.

Register the cluster with the Istio platform

The SPIFFE ID format is configured when the cluster is registered with Defakto. Register the cluster as istio, or eks-istio on Amazon EKS to get the SPIFFE ID required by Istio (e.g. spiffe://trust-domain/ns/namespace-name/sa/service-account-name) by default. The other Kubernetes platforms add the cluster name before /ns/, and Istio rejects those SPIFFE IDs.

Register the cluster the way you manage the rest of Defakto: spirlctl cluster add or spirlctl cluster register with --platform istio, the Terraform spirl_cluster resource with platform = "istio", or the Console. On Amazon EKS, use eks-istio.

For how Defakto chooses the SPIFFE ID format, and how to change it, see SPIFFE ID templates.

Istio clusters cannot belong to a realm

Defakto rejects a cluster registered as istio or eks-istio that belongs to a realm. Creating the cluster, or a new cluster version, fails with istio platforms do not support realms currently, even if the cluster's path template does not include the realm.

Configure the socket for Istio

With spirlctl

spirlctl cluster add with --platform istio generates manifests that are already configured for Istio:

spirlctl cluster add team-awesome --trust-domain example.com --platform istio

With the Helm chart

If you install the spirl-system chart from your own values file, add socketLayout: ISTIO to that file:

socketLayout: ISTIO

The preset places the socket where Istio expects to find it, and gives the controller the matching volume name and mount path. The preset works with the sidecar template below without changes. Most deployments should use the preset.

To set the paths yourself, use socket instead. This example spells out exactly what the preset applies:

socket:
  spiffeSocketPath: /run/spirl/sockets/socket
  volumeName: workload-socket
  volumeMountPath: /run/secrets/workload-spiffe-uds

• spiffeSocketPath: Where the agent creates the socket. Must name a file directly in /run/spirl/sockets, the directory the agent mounts
• volumeName: The name of the CSI volume the controller adds to workloads
• volumeMountPath: Where that volume is mounted inside a workload

The two paths describe opposite ends of one socket. spiffeSocketPath is on the agent's side, and volumeMountPath is on the workload's side. Setting spiffeSocketPath to the workload's path fails at render time:

socket.spiffeSocketPath must name a file directly in /run/spirl/sockets, the directory the agent mounts; got "/run/secrets/workload-spiffe-uds/socket"

The socket's file name is not a separate setting. The chart takes it from the end of spiffeSocketPath. For Istio the file name must be socket, because that is the file Istio looks for.

Set socketLayout or socket, not both. A values file with both fails schema validation. If you change volumeName or volumeMountPath, change the sidecar template to match.

Existing values files

Values files that set platform: ISTIO or platform: EKS_ISTIO keep working unchanged, including the socketFileName setting in the istio and eksIstio blocks. Replacing platform: ISTIO with socketLayout: ISTIO renders the same deployment when the istio block keeps the chart's defaults. To match platform: EKS_ISTIO, also set skipKubeletVerification: false. If you changed settings in the block, move each one to its top-level key instead, using socket rather than socketLayout for socket settings. A block setting stops applying once platform no longer selects the block. See Kubernetes for the other agent settings.

To move explicit socket settings out of a block and into socket, rename them:

istio or eksIstio block	socket
spiffeSocketPath	spiffeSocketPath
socketVolumeName	volumeName
socketVolumeMountPath	volumeMountPath
socketFileName	No equivalent. The chart takes the file name from the end of spiffeSocketPath

Using Istio Operator Injection Webhook Template

Istio Operator has a feature called Istio Sidecar injection custom template. This feature allows you to use your own template to inject Istio Sidecar into your Pods. Defakto provides a template that you can use to inject Istio Sidecar into your Pods.

Please create file named istio-spirl.yaml with the following content:

caution

Please make sure that trustDomain in the template matches the trust domain you specified when you installed Defakto on Kubernetes.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  profile: default
  meshConfig:
    trustDomain: example.org
  values:
    global:
    # This is used to customize the sidecar template
    sidecarInjectorWebhook:
      templates:
        spirl: |
          spec:
            containers:
            - name: istio-proxy
              volumeMounts:
              - name: workload-socket
                mountPath: /run/secrets/workload-spiffe-uds
                readOnly: true
            volumes:
              - name: workload-socket
                csi:
                  driver: "csi.spiffe.io"
                  readOnly: true
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        label:
          istio: ingressgateway
          spiffe.io/spire-managed-identity: "true"
        k8s:
          overlays:
            - apiVersion: apps/v1
              kind: Deployment
              name: istio-ingressgateway
              patches:
                - path: spec.template.spec.volumes.[name:workload-socket]
                  value:
                    name: workload-socket
                    csi:
                      driver: "csi.spiffe.io"
                      readOnly: true
                - path: spec.template.spec.containers.[name:istio-proxy].volumeMounts.[name:workload-socket]
                  value:
                    name: workload-socket
                    mountPath: "/run/secrets/workload-spiffe-uds"
                    readOnly: true
                - path: spec.template.spec.initContainers
                  value:
                    - name: wait-for-spire-socket
                      image: busybox:1.28
                      volumeMounts:
                        - name: workload-socket
                          mountPath: /run/secrets/workload-spiffe-uds
                          readOnly: true
                      env:
                        - name: CHECK_FILE
                          value: /run/secrets/workload-spiffe-uds/socket
                      command:
                        - sh
                        - "-c"
                        - |-
                          echo "$(date -Iseconds)" Waiting for: ${CHECK_FILE}
                          while [[ ! -e ${CHECK_FILE} ]] ; do
                            echo "$(date -Iseconds)" File does not exist: ${CHECK_FILE}
                            sleep 15
                          done
                          ls -l ${CHECK_FILE}

This template will do all the necessary configuration to make Istio work with Defakto SDS API.

Install Istio

Install Istio as usual:

istioctl install -f istio-spirl.yaml

Verify Istio Installation

Make sure that your deployment or namespace has Istio injection enabled (i.e. has the istio-injection: enabled label). This is to tell Istio to inject the Istio sidecar into your Pods.

Here is an example how to enable Istio injection for a namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: bookinfo
  labels:
    istio-injection: enabled

However, Istio will inject the sidecar using default template in this case. To use the Defakto template you need to add the inject.istio.io/templates: "sidecar,spirl" annotation to your Pod template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v3
  namespace: bookinfo
  labels:
    app: reviews
    version: v3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v3
  template:
    metadata:
      labels:
        app: reviews
        version: v3
        k8s.spirl.com/spiffe-csi: enabled
      annotations:
        inject.istio.io/templates: "sidecar,spirl"

After you restart your Pods you should see Istio Sidecar injected into your Pods:

kubectl get pods -n bookinfo
NAME                             READY   STATUS    RESTARTS   AGE
details-v1-6b7b4f9b7c-4q9q2      2/2     Running   0          2m

That means that Istio is injecting an Envoy sidecar into your Pods. View the Istio logs to make sure that Istio is using Defakto SDS API:

kubectl -n bookinfo logs details-v1-74f4cbc748-7vjfm --container istio-proxy |grep SDS
2023-08-18T00:23:14.858580Z info Workload SDS socket found. Istio SDS Server won't be started

If you don't see the line Workload SDS socket found. Istio SDS Server won't be started it means something is not working properly. Please check Istio full logs and Defakto Agent logs for errors.