Chapter 7. SPIRE integration for mesh security

Procedure

1. Install the Operator by following the steps in Installing the Zero Trust Workload Identity Manager (which is OpenShift Container Platform documentation).

   You can install the Zero Trust Workload Identity Manager by using either the web console or CLI.

2. Configure the Operator to work in CreateOnly mode by running the following command:

   $ oc -n zero-trust-workload-identity-manager patch subscription \
     openshift-zero-trust-workload-identity-manager \
     --type='merge' -p '{"spec":{"config":{"env":[{"name":"CREATE_ONLY_MODE","value":"true"}]}}}'

   CreateOnly mode prevents the Operator from automatically overwriting your SPIRE configurations during Istio integration.

3. Define required environment variables:

   $ export TRUST_DOMAIN=<apps.mycluster.example.com>
   $ export ZTWIM_NS=zero-trust-workload-identity-manager
   $ export JWT_ISSUER="https://oidc-discovery.$(oc get ingresses.config/cluster -o jsonpath={.spec.domain})"

   where:

   TRUST_DOMAIN

   The SPIFFE trust domain for the service mesh.

   ZTWIM_NS

   The namespace where the Zero Trust Workload Identity Manager is deployed.

   JWT_ISSUER

   The OpenID Connect (OIDC) discovery endpoint URL for the SPIRE OIDC Discovery Provider. SPIRE exposes SPIFFE identities as OIDC-compatible JSON Web Tokens (JWTs) through this endpoint, enabling Istio to validate workload identities with standard OIDC authentication protocols.

4. Deploy the Operator components according to the OpenShift Container Platform documentation Deploying Zero Trust Workload Identity Manager operands. The components (operands) are:

   • ZeroTrustWorkloadIdentityManager CR
   • SPIRE server
   • SPIRE agent
   • SPIFFE Container Storage Interface (CSI) driver
   • SPIRE OpenID Connect (OIDC) discovery provider

Procedure

1. Define environment variables for the Istio configuration, as shown in the following example:

   $ export ZTWIM_NS=zero-trust-workload-identity-manager
   $ export TRUST_DOMAIN=<apps.mycluster.example.com>
   $ export JWT_ISSUER="https://oidc-discovery.$(oc get ingresses.config/cluster -o jsonpath={.spec.domain})"
   $ export OSSM_NS=istio-system
   $ export OSSM_CNI=istio-cni
   $ export VERIFY_NS=verify-ossm-ztwim
   $ export EXTRA_ROOT_CA="$(oc get secret oidc-serving-cert \
                            -n ${ZTWIM_NS} -o json | \
                            jq -r '.data."tls.crt"' | \
                            base64 -d | \
                            sed 's/^/        /')"

   where:

   ZTWIM_NS

   The namespace where the Zero Trust Workload Identity Manager is deployed. Use the value from your Zero Trust Workload Identity Manager configuration.

   TRUST_DOMAIN

   The SPIFFE trust domain for the service mesh, which is the same domain specified in your Zero Trust Workload Identity Manager configuration.

   JWT_ISSUER

   The OpenID Connect (OIDC) discovery endpoint URL for the SPIRE OIDC Discovery Provider. SPIRE exposes SPIFFE identities as OIDC-compatible JSON Web Tokens (JWTs) through this endpoint, enabling Istio to validate workload identities by using standard OIDC authentication protocols. This URL must match the URL specified in your Zero Trust Workload Identity Manager configuration.

   OSSM_NS

   The namespace where you deploy the Istio control plane.

   OSSM_CNI

   The namespace where you deployed the Istio CNI component.

   VERIFY_NS

   The namespace for deploying verification workloads.

   EXTRA_ROOT_CA

   The root CA certificate from the SPIRE OIDC Discovery Provider. This certificate enables Istio’s pilot component to trust and verify JWTs issued by the SPIRE OIDC provider.

2. Create the Istio CR according to the following example:

   $ oc new-project "${OSSM_NS}" 2>/dev/null
   # Create Istiod
   $ cat <<EOF | oc apply -f -
   apiVersion: sailoperator.io/v1
   kind: Istio
   metadata:
     name: default
   spec:
     namespace: istio-system
     updateStrategy:
       type: InPlace
     values:
       pilot:
         jwksResolverExtraRootCA: |
   ${EXTRA_ROOT_CA}
         env:
           PILOT_JWT_ENABLE_REMOTE_JWKS: "true"
       meshConfig:
         trustDomain: $TRUST_DOMAIN
         defaultConfig:
           proxyMetadata:
             WORKLOAD_IDENTITY_SOCKET_FILE: "spire-agent.sock"
       sidecarInjectorWebhook:
         templates:
           spire: |
             spec:
               initContainers:
               - name: istio-proxy
                 volumeMounts:
                 - name: workload-socket
                   mountPath: /run/secrets/workload-spiffe-uds
                   readOnly: true
               volumes:
                 - name: workload-socket
                   csi:
                     driver: "csi.spiffe.io"
                     readOnly: true
           spireGateway: |
             spec:
               containers:
               - name: istio-proxy
                 volumeMounts:
                 - name: workload-socket
                   mountPath: /run/secrets/workload-spiffe-uds
                   readOnly: true
               volumes:
                 - name: workload-socket
                   csi:
                     driver: "csi.spiffe.io"
                     readOnly: true
   EOF
   # Wait until it is successfully installed
   until oc get deployment istiod -n "${OSSM_NS}" &> /dev/null; do sleep 3; done
   oc wait --for=condition=Available deployment/istiod -n "${OSSM_NS}" --timeout=300s

   where:

   PILOT_JWT_ENABLE_REMOTE_JWKS

   Enables Istio’s pilot component to fetch JSON Web Key Sets (JWKS) from remote OIDC endpoints. Pilot uses these keys to validate JWTs issued by the SPIRE OIDC Discovery Provider.

   WORKLOAD_IDENTITY_SOCKET_FILE

   Specifies the UNIX domain socket filename that sidecar proxies use to communicate with the SPIRE agent and obtain SPIFFE identities.

   sidecarInjectorWebhook.templates.spire

   Custom injection template for regular workload sidecars. It mounts the SPIFFE CSI driver volume at /run/secrets/workload-spiffe-uds. This directory has the UNIX domain socket file that sidecars use to communicate with the SPIRE agent and obtain SPIFFE identities.

   sidecarInjectorWebhook.templates.spireGateway

   Custom injection template for ingress and egress gateway pods. It mounts the SPIFFE Container Storage Interface (CSI) driver volume at /run/secrets/workload-spiffe-uds. This directory has the UNIX domain socket file that gateways use to communicate with the SPIRE agent and obtain SPIFFE identities.

   csi.spiffe.io

   The SPIFFE CSI driver that provides workloads with access to the SPIRE agent’s UNIX domain socket for obtaining SPIFFE identities.

1. Create a test namespace with sidecar injection enabled:

   $ oc new-project "${VERIFY_NS}"
   $ oc label namespace "${VERIFY_NS}" istio-injection=enabled

2. Deploy the httpbin test application and wait for it to become available:

   $ cat <<EOF | oc apply -f -
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: httpbin
     namespace: ${VERIFY_NS}
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: httpbin
         version: v1
     template:
       metadata:
         annotations:
           inject.istio.io/templates: "sidecar,spire"
           spiffe.io/audience: "sky-computing-demo"
         labels:
           app: httpbin
           version: v1
       spec:
         containers:
         - image: docker.io/mccutchen/go-httpbin:v2.15.0
           imagePullPolicy: IfNotPresent
           name: httpbin
           ports:
           - containerPort: 8080
   EOF
   $ oc wait --for=condition=Available deployment/httpbin -n "${VERIFY_NS}" --timeout=300s

3. Verify that the workload received a SPIFFE identity from SPIRE:

   $ HTTPBIN_POD=$(oc get pod -l app=httpbin -n "${VERIFY_NS}" -o jsonpath="{.items[0].metadata.name}")
   $ istioctl proxy-config secret "$HTTPBIN_POD" -n "${VERIFY_NS}" -o json \
     | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' \
     | base64 --decode > chain.pem
   $ openssl x509 -in chain.pem -text | grep SPIRE

   The following output confirms that SPIRE issued the certificate:

   Issuer: C=US, O=Sky Computing Corporation, CN=SPIRE Server CA/serialNumber=...
   Subject: C=US, O=SPIRE

   Note

   SPIRE issues two types of identities: X.509 SPIFFE Verifiable Identity Documents (SVIDs) for mTLS communication, and JWT-SVIDs for HTTP-based authentication. This verification confirms X.509 SVID functionality only.

Procedure

1. Define environment variables for the test namespace and SPIFFE audience:

   $ export TPJ=test-ossm-with-ztwim
   $ export SPIFFE_AUDIENCE="sky-computing-demo"

   where:

   TPJ

   Name of the test namespace where you deploy the sample workloads.

   SPIFFE_AUDIENCE

   Intended recipient of the SPIFFE identity. When workloads request identities from SPIRE, the audience value specifies which services or systems can validate and accept those identities. This restriction helps prevent unauthorized services from misusing identities.

2. Create a test namespace with sidecar injection enabled:

   $ oc create namespace "${TPJ}"
   $ oc label namespace "${TPJ}" istio-injection=enabled

3. Deploy an httpbin server application:

   $ cat <<EOF | oc apply -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: httpbin
     namespace: ${TPJ}
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: httpbin
     namespace: ${TPJ}
     labels:
       app: httpbin
       service: httpbin
   spec:
     ports:
     - name: http-ex-spiffe
       port: 443
       targetPort: 8080
     - name: http
       port: 80
       targetPort: 8080
     selector:
       app: httpbin
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: httpbin
     namespace: ${TPJ}
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: httpbin
         version: v1
     template:
       metadata:
         annotations:
           inject.istio.io/templates: "sidecar,spire"
           spiffe.io/audience: "${SPIFFE_AUDIENCE}"
         labels:
           app: httpbin
           version: v1
       spec:
         serviceAccountName: httpbin
         containers:
         - image: docker.io/mccutchen/go-httpbin:v2.15.0
           imagePullPolicy: IfNotPresent
           name: httpbin
           ports:
           - containerPort: 8080
   EOF
   $ oc wait --for=condition=Available deployment/httpbin -n "${TPJ}" --timeout=300s

4. Deploy a curl client application:

   $ cat <<EOF | oc apply -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: curl
     namespace: ${TPJ}
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: curl
     namespace: ${TPJ}
     labels:
       app: curl
       service: curl
   spec:
     ports:
     - port: 80
       name: http
     selector:
       app: curl
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: curl
     namespace: ${TPJ}
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: curl
     template:
       metadata:
         annotations:
           inject.istio.io/templates: "sidecar,spire"
           spiffe.io/audience: "${SPIFFE_AUDIENCE}"
         labels:
           app: curl
       spec:
         terminationGracePeriodSeconds: 0
         serviceAccountName: curl
         containers:
         - name: curl
           image: curlimages/curl:8.16.0
           command:
           - /bin/sh
           - -c
           - sleep inf
           imagePullPolicy: IfNotPresent
   EOF
   $ oc wait --for=condition=Available deployment/curl -n "${TPJ}" --timeout=300s

5. Test connectivity without mTLS enforcement:

   $ CURL_POD=$(oc get pod -l app=curl -n ${TPJ} -o jsonpath="{.items[0].metadata.name}")
   $ oc exec $CURL_POD -n ${TPJ} -it -- curl -s -o /dev/null -w "%{http_code}" http://httpbin

   The command returns HTTP status code 200, which confirms basic connectivity in permissive mode.

6. Enable strict mTLS and configure destination rules:

   $ cat <<EOF | oc apply -f -
   apiVersion: security.istio.io/v1beta1
   kind: PeerAuthentication
   metadata:
     name: default
     namespace: ${TPJ}
   spec:
     mtls:
       mode: STRICT
   ---
   apiVersion: networking.istio.io/v1
   kind: DestinationRule
   metadata:
     name: curl
     namespace: ${TPJ}
   spec:
     host: curl
     trafficPolicy:
       tls:
         mode: ISTIO_MUTUAL
   ---
   apiVersion: networking.istio.io/v1
   kind: DestinationRule
   metadata:
     name: httpbin
     namespace: ${TPJ}
   spec:
     host: httpbin
     trafficPolicy:
       tls:
         mode: ISTIO_MUTUAL
   EOF

Procedure

1. Grant the necessary security context permissions for the gateway pod:

   $ oc adm policy add-scc-to-user anyuid system:serviceaccount:istio-system:istio-gateway

2. Add the Istio Helm repository:

   $ helm repo add istio https://istio-release.storage.googleapis.com/charts
   $ helm repo update

3. Install the ingress gateway and enable it to communicate with the SPIRE agent:

   $ helm install istio-gateway -n "${OSSM_NS}" \
     istio/gateway --set-json \
     'podAnnotations={"inject.istio.io/templates":"gateway,spireGateway"}'
   $ oc wait --for=condition=Available deployment/istio-gateway -n "${OSSM_NS}" --timeout=300s

   The pod annotation inject.istio.io/templates: gateway,spireGateway mounts the SPIRE agent socket into the gateway pod. The gateway uses this socket to request and receive SPIFFE identities from SPIRE.

4. Create the Gateway resource to define ingress routing rules:

   $ cat <<EOF | oc apply -f -
   apiVersion: networking.istio.io/v1
   kind: Gateway
   metadata:
     name: httpbin-gateway
     namespace: ${TPJ}
   spec:
     selector:
       istio: gateway
     servers:
       - port:
           number: 80
           name: http
           protocol: HTTP
         hosts:
           - "*"
   EOF

5. Configure a VirtualService to route traffic from the gateway to the backend service:

   $ cat <<EOF | oc apply -f -
   apiVersion: networking.istio.io/v1
   kind: VirtualService
   metadata:
     name: httpbin
     namespace: ${TPJ}
   spec:
     hosts:
       - "*"
     gateways:
       - httpbin-gateway
     http:
       - route:
         - destination:
             host: httpbin.${TPJ}.svc.cluster.local
             port:
               number: 80
   EOF

   You do not need to configure destination rules because you created them in the previous example procedure.