Chapter 4. Use X.509 cryptographic identity verification

Procedure

1. On your local host, generate your certificate authority (CA) by running the following command:

   $ openssl req -x509 -sha512 -nodes \
     -days 365 \
     -newkey rsa:4096 \
     -subj "/CN=Test CA/O=Example Org/C=US" \
     -addext basicConstraints=CA:TRUE \
     -addext keyUsage=digitalSignature,keyCertSign \
     -keyout /tmp/ca.key \
     -out /tmp/ca.crt

2. On your local host, create X.509 v3 extensions for your client certificate by running the following command:

   $ cat > /tmp/x509v3.ext << EOF
   authorityKeyIdentifier=keyid,issuer
   basicConstraints=CA:FALSE
   keyUsage=digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
   extendedKeyUsage=clientAuth
   EOF

3. Generate the client certificate on your local host by running the following commands:

   1. Generate your Private Key. This is the secret file that stays on the client. Run the following command:

      $ openssl genrsa -out /tmp/client.key 4096

   2. Create a Certificate Signing Request (CSR). The following command uses your private key and identity information to create a request for a certificate:

      $ openssl req -new \
        -subj "/CN=test-client/O=Example Org/C=US" \
        -key /tmp/client.key \
        -out /tmp/client.csr

   3. Sign the certificate. This command takes your request, looks at your internal CA, and issues the final Public Certificate, the client.crt:

      $ openssl x509 -req -sha512 \
        -days 365 \
        -CA /tmp/ca.crt \
        -CAkey /tmp/ca.key \
        -CAcreateserial \
        -extfile /tmp/x509v3.ext \
        -in /tmp/client.csr \
        -out /tmp/client.crt

4. If you do not have a CA that you use to issue and sign client certificates, create the self-signed CA certificate in your cluster by running the following commands:

   1. Create the ConfigMap for Gateway TLS validation by running the following command:

      $ oc create configmap gateway-client-ca \
        -n <gateway_system> \
        --from-file=ca.crt=/tmp/ca.crt

      • Replace <gateway_system> with the namespace where your Gateway object is applied.
      • The ConfigMap must be in the same namespace as the Gateway object.

   2. Create the Authorino validation Secret by running the following command:

      $ oc create secret generic authorino-ca-validation \
        -n kuadrant-system \
        --from-file=ca.crt=/tmp/ca.crt

      The Secret must be in the same namespace as Authorino.

   3. Label the Secret so that Authorino can find its trusted anchors by running the following command:

      $ oc label secret authorino-ca-validation \
        -n kuadrant-system \
        authorino.kuadrant.io/managed-by=authorino \
        app.kubernetes.io/name=trusted-client

Procedure

1. Create the cert-manager Issuer custom resource (CR) by using the following example:

   apiVersion: cert-manager.io/v1
   kind: Issuer
   metadata:
     name: gateway-cert-issuer
     namespace: <gateway_system>
   spec:
     selfSigned: {}

   • Replace <gateway_system> with the namespace of your Gateway object.

2. Apply the cert-manager Issuer object by running the following command:

   $ oc apply - f gateway-cert-issuer.yaml

3. Create the TLSPolicy object by using the following example:

   apiVersion: kuadrant.io/v1
   kind: TLSPolicy
   metadata:
     name: mtls-gateway-tls
     namespace: <gateway_system>
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: Gateway
       name: mtls-gateway
       sectionName: https
     issuerRef:
       group: cert-manager.io
       kind: Issuer
       name: gateway-cert-issuer

   • Replace the value of the metadata.namespace parameter with the namespace where your Gateway object is applied.

4. Apply the TLSPolicy object by running the following command:

   $ oc apply -f mtls-gateway-tls.yaml

5. On your local host, create an infrastructure customization ConfigMap CR to inject the client CA volume mounts into the managed Istio proxy pods by using the following example:

   Example ConfigMap CR for use with an X.509 Gateway object

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: mtls-gateway-config
     namespace: <gateway_system>
   data:
     deployment: |
       spec:
         template:
           spec:
             volumes:
             - name: client-ca
               configMap:
                 name: gateway-client-ca
             containers:
             - name: istio-proxy
               volumeMounts:
               - name: client-ca
                 mountPath: /etc/certs
                 readOnly: true

   • Replace the value of the metadata.namespace parameter with the namespace where your Gateway object is applied.

6. Apply the infrastructure customization ConfigMap CR by running the following command:

   $ oc apply -f mtls-gateway-config.yaml

7. Create the mTLS Gateway object by using the following example:

   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: mtls-gateway
     namespace: <gateway_system>
   spec:
     gatewayClassName: openshift-default
     infrastructure:
       parametersRef:
         group: ""
         kind: ConfigMap
         name: mtls-gateway-config
     listeners:
     - name: https
       protocol: HTTPS
       port: 443
       hostname: "<gateway_hostname>"
       allowedRoutes:
         namespaces:
           from: All
       tls:
         mode: Terminate
         certificateRefs:
         - name: <gateway_name>-<listener_name>-tls
           kind: Secret

   • Replace the value of the metadata.namespace parameter with the namespace where your Gateway object is applied.
   • Replace the value of the spec.listeners.hostname parameter with the name of your Gateway object hostname.
   • Replace spec.listeners.tls.certificateRefs.name with the name of your Gateway object and the name of your mTLS listener in the <gateway_name>-<listener_name>-tls format.

8. Apply the mTLS Gateway CR by running the following command:

   $ oc apply -f mtls-gateway.yaml

9. Create an EnvoyFilter CR to configure mutual TLS validation by using the following example:

   Example mutual TLS EnvoyFilter CR

   apiVersion: networking.istio.io/v1alpha3
   kind: EnvoyFilter
   metadata:
     name: mtls-validation
     namespace: <gateway_system>
   spec:
     targetRefs:
     - group: gateway.networking.k8s.io
       kind: Gateway
       name: mtls-gateway
     configPatches:
     - applyTo: FILTER_CHAIN
       match:
         context: GATEWAY
         listener:
           portNumber: 443
       patch:
         operation: MERGE
         value:
           transport_socket:
             name: envoy.transport_sockets.tls
             typed_config:
               "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
               commonTlsContext:
                 validationContext:
                   trustedCa:
                     filename: /etc/certs/ca.crt
               requireClientCertificate: true

   • Replace the value of the metadata.namespace parameter with the namespace where your Gateway object is applied.

10. Apply the EnvoyFilter CR by running the following command:

    $ oc apply -f mtls-envoyfilter.yaml

Procedure

1. Deploy the sample application by using the following resources:

   Example httpbin application

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: httpbin
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: httpbin
     labels:
       app: httpbin
       service: httpbin
   spec:
     ports:
     - name: http
       port: 80
     selector:
       app: httpbin
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: httpbin
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: httpbin
         version: v1
     template:
       metadata:
         labels:
           app: httpbin
           version: v1
       spec:
         serviceAccountName: httpbin
         securityContext:
           runAsNonRoot: true
           seccompProfile:
             type: RuntimeDefault
         containers:
         - name: httpbin
           image: quay.io/kuadrant/httpbin@sha256:2865a9ac6596340135fc3cd82cbc543c4050124afb10361e36f727abe4f71900
           imagePullPolicy: IfNotPresent
           securityContext:
             runAsUser: 1000
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
             capabilities:
               drop:
               - ALL
           ports:
           - containerPort: 80
           volumeMounts:
           - name: tmp
             mountPath: /tmp
         volumes:
         - name: tmp
           emptyDir: {}

2. Apply the application objects by running the following command:

   $ oc apply -f httpbin.yaml

Procedure

1. Create your HTTPRoute CR by using the following example:

   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: <httpbin_route>
     namespace: <default>
     labels:
       app: <httpbin>
   spec:
     parentRefs:
     - name: mtls-gateway
       namespace: <gateway_system>
     rules:
     - matches:
       - path:
           type: PathPrefix
           value: /
       backendRefs:
       - name: httpbin
         port: 80

   • Replace the metadata.name parameter value with the name of your application route.
   • Replace the metadata.namespace parameter with the namespace where your Gateway object is applied.
   • Replace the metadata.labels.app value with the name of the application.
   • Replace the spec.parentRefs.namespace value with the namespace where your Gateway object is applied.

2. Apply your HTTPRoute custom resource by running the following command:

   $ oc apply -f <app_http_route.yaml>

   Replace <app_http_route.yaml> with the name of your HTTPRoute YAML.

Procedure

1. Create your X.509 AuthPolicy CR by using the following example:

   Example X.509 AuthPolicy CR

   apiVersion: kuadrant.io/v1
   kind: AuthPolicy
   metadata:
     name: x509-auth-policy
     namespace: default
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: httpbin-route
     rules:
       authentication:
         "x509-client-cert":
           x509:
             source:
               xfccHeader: "x-forwarded-client-cert"
             selector:
               matchLabels:
                 app.kubernetes.io/name: trusted-client
       authorization:
         "certificate-attributes":
           patternMatching:
             patterns:
             - predicate: "size(auth.identity.Organization) > 0 && auth.identity.Organization[0] == 'Example Org'"
       response:
         success:
           headers:
             "x-client-common-name":
               plain:
                 expression: auth.identity.CommonName
             "x-client-org":
               plain:
                 expression: auth.identity.Organization[0]

   • The spec.rules.authentication."x509-client-cert".x509 parameter causes the policy to extract the certificate from the X-Forwarded-Client-Cert standard (XFCC) header. The Gateway object populates this header after validating the client certificate.
   • The spec.rules.authentication."x509-client-cert".x509.source.xfccHeader parameter value is the name of the XFCC header. This is the default header name.
   • Set the spec.rules.response.success.headers."x-client-common-name".plain.expression parameter value to ensure that the policy extracts the common name (CN) from the certificate.
   • Set the spec.rules.response.success.headers"x-client-org".plain.expression parameter value to ensure that the policy extracts the organization (O) from the certificate.

2. Apply your AuthPolicy custom resource by running the following command:

   $ oc apply -f <app_authpolicy.yaml>

   Replace <app_authpolicy.yaml> with the name of your AuthPolicy YAML.

Procedure

1. Retrieve the Gateway object hostname by running the following command:

   $ export GATEWAY_HOSTNAME=$(oc get gateway mtls-gateway -n <gateway_system> -o jsonpath='{.status.addresses[0].value}')

   Replace <gateway_system> with the namespace where your Gateway object is applied.

2. Test with a valid certificate by running the following command:

   $ curl -ik https://httpbin.${GATEWAY_HOSTNAME}/get \
     --cert /tmp/client.crt \
     --key /tmp/client.key

   Example output

   HTTP 200

3. Verify that the TLS handshake fails when there is no certificate by running the following command:

   $ curl -ik https://httpbin.${GATEWAY_HOSTNAME}/get

   Example output

   curl: (56) OpenSSL SSL_read: OpenSSL/3.6.2: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

4. Verify that the TLS handshake fails when there is an untrusted certificate by running the following commands:

   1. Generate an untrusted certificate by running the following command:

      $ openssl req -x509 -sha512 -nodes \
        -days 365 \
        -newkey rsa:4096 \
        -subj "/CN=Untrusted CA/O=Untrusted Org/C=US" \
        -keyout /tmp/untrusted-ca.key \
        -out /tmp/untrusted-ca.crt

   2. Generate an untrusted private key by running the following command:

      $ openssl genrsa -out /tmp/untrusted-client.key 4096

   3. Create a Certificate Signing Request (CSR) for the untrusted client by running the following command:

      $ openssl req -new \
        -subj "/CN=untrusted-client/O=Untrusted Org/C=US" \
        -key /tmp/untrusted-client.key \
        -out /tmp/untrusted-client.csr

   4. Sign the request using your untrusted authority to issue the client certificate by running the following command:

      $ openssl x509 -req -sha512 \
        -days 365 \
        -CA /tmp/untrusted-ca.crt \
        -CAkey /tmp/untrusted-ca.key \
        -CAcreateserial \
        -extfile /tmp/x509v3.ext \
        -in /tmp/untrusted-client.csr \
        -out /tmp/untrusted-client.crt

   5. Test the untrusted certificate by running the following command:

      $ curl -ik https://httpbin.${GATEWAY_HOSTNAME}/get \
        --cert /tmp/untrusted-client.crt \
        --key /tmp/untrusted-client.key

      Example output

      curl: (56) OpenSSL SSL_read: OpenSSL/3.6.2: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0

Procedure

1. Retrieve the gateway hostname by running the following command:

   $ export GATEWAY_HOSTNAME=$(oc get gateway mtls-gateway -n <gateway_system> -o jsonpath='{.status.addresses[0].value}')

   Replace <gateway_system> with the namespace where your Gateway object is applied.

2. Verify a connection failure when there is a valid certificate, but wrong attributes by running the following commands:

   1. Generate a certificate with a different organization by running the following command:

      $ openssl genrsa -out /tmp/unauthorized-client.key 4096

   2. Create a Certificate Signing Request (CSR) with unauthorized attributes by running the following command:

      $ openssl req -new \
        -subj "/CN=unauthorized-client/O=Unauthorized Org/C=US" \
        -key /tmp/unauthorized-client.key \
        -out /tmp/unauthorized-client.csr

   3. Sign the request using your trusted certificate authority by running the following command:

      $ openssl x509 -req -sha512 \
        -days 365 \
        -CA /tmp/ca.crt \
        -CAkey /tmp/ca.key \
        -CAcreateserial \
        -extfile /tmp/x509v3.ext \
        -in /tmp/unauthorized-client.csr \
        -out /tmp/unauthorized-client.crt

   4. Test the unauthorized certificate by running the following command:

      $ curl -ik https://httpbin.${GATEWAY_HOSTNAME}/get \
        --cert /tmp/unauthorized-client.crt \
        --key /tmp/unauthorized-client.key

      Example output

      HTTP 403

Procedure

1. If the TLS handshake succeeds without client certificate, the Envoy filter might not be properly applied. Troubleshoot this situation by taking the following steps:

   1. Verify that the EnvoyFilter exists by running the following command:

      $ oc get envoyfilter mtls-validation -n <gateway_system>

      Replace <gateway_system> with the namespace where your Gateway object is applied.

   2. Check that the targetRef in the Envoy filter points to the Gateway object by running the following command:

      $ oc get envoyfilter mtls-validation -n <gateway_system> -o yaml | grep -A 3 targetRefs

      Replace <gateway_system> with the namespace where your Gateway object is applied.

   3. Check that the Envoy configuration has the correct configuration by running the following command:

      $ oc exec -n <gateway_system> \
        $(oc get pod -n <gateway_system> -l gateway.networking.k8s.io/gateway-name=mtls-gateway -o jsonpath='{.items[0].metadata.name}') \
        -- curl -s localhost:15000/config_dump?include_eds | grep -A 20 validation_context

      Replace <gateway_system> with the namespace where your Gateway object is applied.

2. If the Gateway object logs show failed to load trusted CA, the certificate authority (CA) might not be mounted. Use the following steps to check the CA:

   1. Verify that the ConfigMap exists by running the following command:

      $ oc get configmap gateway-client-ca -n <gateway_system>

      Replace <gateway_system> with the namespace where your Gateway object is applied.

   2. Check that the volume mount in the Gateway pod by running the following command:

      $ oc describe pod -n <gateway_system> -l gateway.networking.k8s.io/gateway-name=mtls-gateway | grep -A 5 Mounts

      Replace <gateway_system> with the namespace where your Gateway object is applied.

   3. Verify that the CA file exists in the Gateway pod by running the following command:

      $ oc exec -n <gateway_system> \
        $(oc get pod -n <gateway_system> -l gateway.networking.k8s.io/gateway-name=mtls-gateway -o jsonpath='{.items[0].metadata.name}') \
        -- cat /etc/certs/ca.crt

      Replace <gateway_system> with the namespace where your Gateway object is applied.

3. If Authorino rejects requests with a certificate not found error, the XFCC header might not be set correctly. Check the configuration and logs to help resolve this issue as follows:

   1. Verify the forward_client_cert_details configuration by running the following command:

      $ oc exec -n <gateway_system> \
        $(oc get pod -n <gateway_system> -l gateway.networking.k8s.io/gateway-name=mtls-gateway -o jsonpath='{.items[0].metadata.name}') \
        -- curl -s localhost:15000/config_dump?include_eds | grep -i forward_client_cert

      Replace <gateway_system> with the namespace where your Gateway object is applied.

   2. Check the Envoy access logs for the XFCC header by running the following command:

      $ oc logs -n <gateway_system> -l gateway.networking.k8s.io/gateway-name=mtls-gateway | grep XFCC

      Replace <gateway_system> with the namespace where your Gateway object is applied.