Rotate the Rekor signer key

You can proactively rotate Rekor's signer key by using the sharding feature to freeze the log tree, and create a new log tree with a new signer key.

Before you begin

For OpenShift:

  • Installation of the RHTAS operator running on Red Hat OpenShift Container Platform.
  • A running Securesign instance.
  • A workstation with the oc, openssl, and cosign binaries installed.

For Red Hat Enterprise Linux:

  • Installation of RHTAS running on Red Hat Enterprise Linux managed by Ansible.
  • A workstation with the rsync, openssl, and cosign binaries installed.
  • A SSH connection to the managed node, with root-level privileges on the managed node.

About this task

This procedure walks you through expiring your old Rekor signer key, and replacing it with a new signer key for Red Hat Trusted Artifact Signer (RHTAS) to use. When expiring your old Rekor signer key you can still verify artifacts signed by the old key.

Important
This procedure requires downtime to the Rekor service.

Procedure

  1. For OpenShift: Download the rekor-cli binary from the OpenShift cluster to your workstation.
    1. Login to the OpenShift web console. From the home page, click the ? icon, click Command line tools, go to the rekor-cli download section, and click the link for your platform.
    2. Open a terminal on your workstation, decompress the binary .gz file, and set the execute bit: 
      $ gunzip rekor-cli-amd64.gz
$ chmod +x rekor-cli-amd64
    3. Move and rename the binary to a location within your $PATH environment: 
      $ sudo mv rekor-cli-amd64 /usr/local/bin/rekor-cli
  2. For Red Hat Enterprise Linux: Download the rekor-cli binary from the Red Hat developers portal.
    Note
    The binaries are available at Red Hat Trusted Artifact Signer download page.
    1. From the download page, go to the rekor-cli download section, and click the link for your platform.
    2. Open a terminal on your workstation, decompress the binary .gz file, and set the execute bit: 
      $ gunzip rekor-cli-amd64.gz
$ chmod +x rekor-cli-amd64
    3. Move and rename the binary to a location within your $PATH environment: 
      $ sudo mv rekor-cli-amd64 /usr/local/bin/rekor-cli
  3. For OpenShift: Download the tufcli binary from the OpenShift cluster to your workstation.
    1. From the home page, click the ? icon, click Command line tools, go to the tufcli download section, and click the link for your platform.
  4. For Red Hat Enterprise Linux: Download the tufcli binary from the Red Hat developers portal.
    Note
    The binaries are available at Red Hat Trusted Artifact Signer download page.
    1. From the download page, go to the tufcli download section, and click the link for your platform.
  5. Open a terminal on your workstation, extract the .tar.gz archive, and set the execution bit: 
    $ tar -xzf tufcli_linux_amd64.tar.gz
$ chmod +x tufcli
  6. Move the binary to a location within your $PATH environment: 
    $ sudo mv tufcli /usr/local/bin/tufcli
  7. For OpenShift: Log in to OpenShift from the command line: 
    oc login --token=TOKEN --server=SERVER_URL_AND_PORT
     
    $ oc login --token=sha256~ZvFDBvoIYAbVECixS4-WmkN4RfnNd8Neh3y1WuiFPXC --server=https://example.com:6443
    Note
    You can find your login token and URL for use on the command line from the OpenShift web console. Log in to the OpenShift web console. Click your user name, and click Copy login command. Offer your user name and password again, if asked, and click Display Token to view the command.
  8. For OpenShift: Switch to the RHTAS project: 
    $ oc project trusted-artifact-signer
  9. For OpenShift: Make a backup of the current Rekor configuration and keys: 
    $ export SERVER_CONFIG_NAME=$(oc get rekor -o jsonpath='{.items[0].status.serverConfigRef.name}')
$ oc get secret $SERVER_CONFIG_NAME -o jsonpath="{.data.private}" | base64 --decode > private.pem
$ oc get secret $SERVER_CONFIG_NAME -o jsonpath="{.data.public}" | base64 --decode > public.pem
  10. For OpenShift: Capture the current tree identifier: 
    $ export OLD_TREE_ID=$(oc get rekor -o jsonpath='{.items[0].status.treeID}')
  11. For OpenShift: Set the log tree to the DRAINING state: 
    $ oc run updatetree --image registry.redhat.io/rhtas/updatetree-rhel9:1.1.0 --restart=Never --attach=true --rm=true -q -- --admin_server=trillian-logserver.trusted-artifact-signer.svc:8091 --tree_id=${OLD_TREE_ID} --tree_state=DRAINING --tls_cert_file=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

    While draining, the tree log will not accept any new entries. Watch and wait for the queue to empty.

    Important
    You must wait for the queues to be empty before proceeding to the next step. If leaves are still integrating while draining, then freezing the log tree during this process can cause the log path to exceed the maximum merge delay (MMD) threshold.
  12. For OpenShift: Once the queue has been fully drained, freeze the log: 
    $ oc run updatetree --image registry.redhat.io/rhtas/updatetree-rhel9:1.1.0 --restart=Never --attach=true --rm=true -q -- --admin_server=trillian-logserver.trusted-artifact-signer.svc:8091 --tree_id=${OLD_TREE_ID} --tree_state=FROZEN --tls_cert_file=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
  13. For OpenShift: Create a new Merkle tree, and capture the new tree identifier: 
    $ oc run createtree --image registry.redhat.io/rhtas/createtree-rhel9:1.1.0 --restart=Never -q -- -logtostderr=false --admin_server=trillian-logserver.trusted-artifact-signer.svc:8091 --display_name=rekor-tree --tls_cert_file=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
$ oc wait pod/createtree --for=jsonpath='{.status.phase}'=Succeeded --timeout=5m
$ export NEW_TREE_ID=$(oc logs pod/createtree | tail -n 1 | tr -d '[:space:]')
$ oc delete pod/createtree
  14. For OpenShift: Generate a new certificate, along with new public and private keys: 
    $ openssl ecparam -genkey -name prime256v1 -noout -out new-rekor.pem
$ openssl ec -in new-rekor.pem -pubout -out new-rekor-public.pem
    Important
    The certificate and new keys must have unique file names.
  15. Configure The Update Framework (TUF) service to use the new Rekor signer key.
    1. For OpenShift: Set up your shell environment: 
      $ export WORK="${HOME}/trustroot-example"
$ export ROOT="${WORK}/root/root.json"
$ export KEYDIR="${WORK}/keys"
$ export INPUT="${WORK}/input"
$ export REKOR_URL="$(oc get rekor -o jsonpath='{.items[0].status.url}')"
$ export TUF_REPO="${WORK}/tuf-repo"
$ export TUF_SERVER_POD="$(oc get pods -l app.kubernetes.io/component=tuf,\!job-name -o jsonpath='{.items[0].metadata.name}')"
$ export TUF_URL="$(oc get tuf -o jsonpath='{.items[0].status.url}')"
    2. For Red Hat Enterprise Linux: Set up your shell environment: 
      $ export BASE_HOSTNAME=BASE_HOSTNAME_OF_RHTAS_SERVICE
$ export WORK="${HOME}/trustroot-example"
$ export ROOT="${WORK}/root/root.json"
$ export KEYDIR="${WORK}/keys"
$ export INPUT="${WORK}/input"
$ export REKOR_URL="https://rekor.${BASE_HOSTNAME}"
$ export TUF_REPO="${WORK}/tuf-repo"
$ export TUF_URL="https://tuf.${BASE_HOSTNAME}"
$ export MANAGED_NODE_IP=IP_OF_ANSIBLE_MANAGED_NODE
$ export MANAGED_NODE_SSH_USER=USER_TO_CONNECT_TO_MANAGED_NODE
$ export REMOTE_KEYS_VOLUME=$(ssh ${MANAGED_NODE_SSH_USER}@${MANAGED_NODE_IP} -t "sudo podman volume mount tuf-signing-keys" | tr -d '[:space:]')
$ export REMOTE_TUF_VOLUME=$(ssh ${MANAGED_NODE_SSH_USER}@${MANAGED_NODE_IP} -t "sudo podman volume mount tuf-repository" | tr -d '[:space:]')
    3. Create a temporary TUF directory structure: 
      $ mkdir -p "${WORK}/root/" "${KEYDIR}" "${INPUT}" "${TUF_REPO}"
    4. For OpenShift: Download the TUF contents to the temporary TUF directory structure: 
      $ oc extract --confirm --to "${KEYDIR}/" secret/tuf-root-keys
$ oc cp "${TUF_SERVER_POD}:/var/www/html" "${TUF_REPO}"
$ cp "${TUF_REPO}/root.json" "${ROOT}"
    5. For Red Hat Enterprise Linux: Download the TUF contents to the temporary TUF directory structure: 
      $ rsync -r --rsync-path="sudo rsync" ${MANAGED_NODE_SSH_USER}@${MANAGED_NODE_IP}:"${REMOTE_KEYS_VOLUME}/" "${KEYDIR}"
$ rsync -r --rsync-path="sudo rsync" ${MANAGED_NODE_SSH_USER}@${MANAGED_NODE_IP}:"${REMOTE_TUF_VOLUME}/" "${TUF_REPO}"
$ cp "${TUF_REPO}/root.json" "${ROOT}"
    6. Expire the old Rekor signer key: 
      $ tufcli rhtas \
  --root "${ROOT}" \
  --key "${KEYDIR}/snapshot.pem" \
  --key "${KEYDIR}/targets.pem" \
  --key "${KEYDIR}/timestamp.pem" \
  --set-rekor-target "public.pem" \
  --rekor-uri "${REKOR_URL}" \
  --rekor-status "Expired" \
  --outdir "${TUF_REPO}" \
  --metadata-url "file://${TUF_REPO}"
    7. Add the new Rekor signer key: 
      $ tufcli rhtas \
  --root "${ROOT}" \
  --key "${KEYDIR}/snapshot.pem" \
  --key "${KEYDIR}/targets.pem" \
  --key "${KEYDIR}/timestamp.pem" \
  --set-rekor-target "new-rekor-public.pem" \
  --rekor-uri "${REKOR_URL}" \
  --outdir "${TUF_REPO}" \
  --metadata-url "file://${TUF_REPO}"
    8. For OpenShift: Upload these changes to the TUF server: 
      $ oc rsync --no-perms "${TUF_REPO}/" "${TUF_SERVER_POD}:/var/www/html"
    9. For Red Hat Enterprise Linux: Create a compressed archive file of the updated TUF repository: 
      $ tar -C "${WORK}" -czvf repository.tar.gz tuf-repo
    10. For Red Hat Enterprise Linux: Update the RHTAS Ansible playbook by adding the new compressed archive file content to the tas_single_node_trust_root variable: 
      tas_single_node_trust_root:
  full_archive: "{{ lookup('file', 'repository.tar.gz') | b64encode }}"
    11. Delete the working directory: 
      $ rm -r $WORK
  16. For OpenShift: Create a new ConfigMap for the new Rekor signer keys: 
    $ oc create configmap rekor-keys --from-file=private=new-rekor.pem --from-file=public=new-rekor-public.pem
  17. For OpenShift: Update the Securesign Rekor configuration: 
    $ read -r -d '' SECURESIGN_PATCH <<EOF
[
{
    "op": "replace",
    "path": "/spec/rekor/treeID",
    "value": ${NEW_TREE_ID}
},
{
    "op": "replace",
    "path": "/spec/rekor/signer/privateKeyRef",
    "value": {"name": "rekor-keys", "key": "private"}
},
{
    "op": "replace",
    "path": "/spec/rekor/signer/publicKeyRef",
    "value": {"name": "rekor-keys", "key": "public"}
}
]
EOF
  18. For OpenShift: Patch the Securesign instance: 
    $ oc patch Securesign securesign-sample --type='json' -p="$SECURESIGN_PATCH"
  19. For OpenShift: Wait for the Rekor server to redeploy: 
    $ oc wait pod -l app.kubernetes.io/name=rekor-server,\!job-name --for=condition=Ready
  20. For OpenShift: Accept the updated Rekor trust material: 
    $ oc annotate rekor securesign-sample rhtas.redhat.com/refresh-trust-material=true --overwrite
    Note
    After you rotate the signer, the Rekor instance stays unready until you annotate the resource to accept the new trust material.
  21. For OpenShift: Wait until the Rekor instance is ready: 
    $ oc wait rekor securesign-sample --for=condition=Ready
  22. For Red Hat Enterprise Linux: Run the RHTAS Ansible Playbook to apply the changes: 
    $ ansible-playbook -i inventory play.yml
  23. Update the cosign configuration with the updated TUF configuration: 
    $ : "${TAS_TUF_ROOT_CHECKSUM:?Set the pinned TAS_TUF_ROOT_CHECKSUM}"
$ cosign initialize --mirror=$TUF_URL --root=$TUF_URL/1.root.json --root-checksum=$TAS_TUF_ROOT_CHECKSUM

    The --root-checksum option ensures that the TUF root metadata is valid and has not been tampered with.

  24. For OpenShift, confirm the new public key: 
    $ oc annotate rekor securesign-sample rhtas.redhat.com/refresh-trust-material=true --overwrite

    The operator processes the annotation and updates the trust material.

Results

For Openshift, verify that the trust material refresh completed successfully: 
$ oc get rekor securesign-sample -o jsonpath='{.status.conditions[?(@.type=="TrustMaterialAvailable")].status}'
This command should return True.

Now, you are ready to sign and verify your artifacts with the new CT log signer key.

What to do next

For Red Hat Enterprise Linux:

Additional resources

  • Running your playbook with Ansible Automation Platform