×

The AWS Secrets and Configuration Provider (ASCP) provides a way to expose AWS Secrets as Kubernetes storage volumes. With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on ROSA.

This is made even easier and more secure through the use of AWS STS and Kubernetes PodIdentity.

Prerequisites

  • A ROSA cluster deployed with STS

  • Helm 3

  • aws CLI

  • oc CLI

  • jq

Preparing Environment

  1. Validate that your cluster has STS:

    $ oc get authentication.config.openshift.io cluster -o json \
      | jq .spec.serviceAccountIssuer

    You should see something like the following, if not you should not proceed, instead look to the Red Hat documentation on creating an STS cluster.

    "https://xxxxx.cloudfront.net/xxxxx"
  2. Set SecurityContextConstraints to allow the CSI driver to run:

    $ oc new-project csi-secrets-store
    $ oc adm policy add-scc-to-user privileged \
        system:serviceaccount:csi-secrets-store:secrets-store-csi-driver
    $ oc adm policy add-scc-to-user privileged \
        system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws
  3. Create some environment variables to refer to later:

    $ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
    $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster \
       -o jsonpath='{.spec.serviceAccountIssuer}' | sed  's|^https://||')
    $ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
    $ export AWS_PAGER=""

Deploy the AWS Secrets and Configuration Provider

  1. Use Helm to register the secrets store CSI driver:

    $ helm repo add secrets-store-csi-driver \
        https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
  2. Update your Helm repositories:

    $ helm repo update
  3. Install the secrets store CSI driver:

    $ helm upgrade --install -n csi-secrets-store \
        csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver
  4. Deploy the AWS provider:

    $ oc -n csi-secrets-store apply -f \
        https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml
  5. Check that both Daemonsets are running:

    $ oc -n csi-secrets-store get ds \
        csi-secrets-store-provider-aws \
        csi-secrets-store-driver-secrets-store-csi-driver
  6. Label the Secrets Store CSI Driver to allow use with the restricted pod security profile:

    $ oc label csidriver.storage.k8s.io/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted

Creating a Secret and IAM Access Policies

  1. Create a secret in Secrets Manager:

    $ SECRET_ARN=$(aws --region "$REGION" secretsmanager create-secret \
        --name MySecret --secret-string \
        '{"username":"shadowman", "password":"hunter2"}' \
        --query ARN --output text)
    $ echo $SECRET_ARN
  2. Create an IAM Access Policy document:

    $ cat << EOF > policy.json
    {
       "Version": "2012-10-17",
       "Statement": [{
          "Effect": "Allow",
          "Action": [
            "secretsmanager:GetSecretValue",
            "secretsmanager:DescribeSecret"
          ],
          "Resource": ["$SECRET_ARN"]
          }]
    }
    EOF
  3. Create an IAM Access Policy:

    $ POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
    --output text iam create-policy \
    --policy-name openshift-access-to-mysecret-policy \
    --policy-document file://policy.json)
    $ echo $POLICY_ARN
  4. Create an IAM Role trust policy document:

    The trust policy is locked down to the default service account of a namespace you will create later.

    $ cat <<EOF > trust-policy.json
    {
       "Version": "2012-10-17",
       "Statement": [
       {
       "Effect": "Allow",
       "Condition": {
         "StringEquals" : {
           "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:my-application:default"]
          }
        },
        "Principal": {
           "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
        },
        "Action": "sts:AssumeRoleWithWebIdentity"
        }
        ]
    }
    EOF
  5. Create an IAM role:

    $ ROLE_ARN=$(aws iam create-role --role-name openshift-access-to-mysecret \
    --assume-role-policy-document file://trust-policy.json \
    --query Role.Arn --output text)
    $ echo $ROLE_ARN
  6. Attach the role to the policy:

    $ aws iam attach-role-policy --role-name openshift-access-to-mysecret \
        --policy-arn $POLICY_ARN

Create an Application to use this secret

  1. Create an OpenShift project:

    $ oc new-project my-application
  2. Annotate the default service account to use the STS Role:

    $ oc annotate -n my-application serviceaccount default \
        eks.amazonaws.com/role-arn=$ROLE_ARN
  3. Create a secret provider class to access our secret:

    $ cat << EOF | oc apply -f -
    apiVersion: secrets-store.csi.x-k8s.io/v1
    kind: SecretProviderClass
    metadata:
      name: my-application-aws-secrets
    spec:
      provider: aws
      parameters:
        objects: |
          - objectName: "MySecret"
            objectType: "secretsmanager"
    EOF
  4. Create a Deployment using our secret:

    $ cat << EOF | oc apply -f -
    apiVersion: v1
    kind: Pod
    metadata:
      name: my-application
      labels:
        app: my-application
    spec:
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "my-application-aws-secrets"
      containers:
      - name: my-application-deployment
        image: k8s.gcr.io/e2e-test-images/busybox:1.29
        command:
          - "/bin/sleep"
          - "10000"
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
    EOF
  5. Verify the Pod has the secret mounted:

    $ oc exec -it my-application -- cat /mnt/secrets-store/MySecret

Clean up

  1. Delete the application:

    $ oc delete project my-application
  2. Delete the secrets store csi driver:

    $ helm delete -n csi-secrets-store csi-secrets-store-driver
  3. Delete Security Context Constraints:

    $ oc adm policy remove-scc-from-user privileged \
        system:serviceaccount:csi-secrets-store:secrets-store-csi-driver
    $ oc adm policy remove-scc-from-user privileged \
        system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws
  4. Delete the AWS provider:

    $ oc -n csi-secrets-store delete -f \
    https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml
  5. Delete AWS Roles and Policies:

    $ aws iam detach-role-policy --role-name openshift-access-to-mysecret \
        --policy-arn $POLICY_ARN
    $ aws iam delete-role --role-name openshift-access-to-mysecret
    $ aws iam delete-policy --policy-arn $POLICY_ARN
  6. Delete the Secrets Manager secret:

    $ aws secretsmanager --region $REGION delete-secret --secret-id $SECRET_ARN