Maintain and diagnose runtime issues

Effective maintenance and troubleshooting ensure that containerized workloads remain secure and functional. This involves applying consistent security updates through automated builds and utilizing specialized debugging techniques for minimal, hardened environments.

Update Red Hat Hardened Images to receive security patches
Copy link

Regular updates mitigate vulnerabilities by incorporating the latest security fixes. Because Red Hat Hardened Images are immutable, updates require rebuilding the image to layer application code over the refreshed base.

About this task
Copy link

The steps for updating an image depend on your specific image configuration.

Procedure
Copy link

  • If you use the original image without customizations, enter: 
    $ podman pull registry.access.redhat.com/hi/<image>:<tag>
  • If you use the image with a specific digest in a Containerfile:
    1. Update the digest to the one you want to use: 
      FROM registry.access.redhat.com/hi/<image>:<new_digest>
    2. Rebuild the custom image: 
      $ podman build ...
  • If you use the image with a specific version tag in a Containerfile, rebuild the custom image with the --pull=newer option: 
    $ podman build --pull=newer ...

    For example, if your Containerfile uses the :3.14 tag, Podman pulls the latest image with this tag but without switching to a newer version of the software, if available.

Debug minimal Red Hat Hardened Images runtime containers by displaying the container logs
Copy link

To reduce the attack surface, hardened images do not contain a shell and standard utilities. To troubleshoot containers with applications that log to standard error (stderr), you can display the container logs to identify problems.

Before you begin
Copy link

  • If you use RHEL, the podman package is installed.

Procedure
Copy link

  • If you use Podman on RHEL, enter: 
    $ podman container logs <container_name_or_id>
  • If you use OpenShift, enter: 
    $ oc logs <service_or_pod>

Debug minimal Red Hat Hardened Images runtime containers by using a transient debug container
Copy link

To reduce the attack surface, hardened images do not contain a shell and standard utilities. To troubleshoot, you can attach a transient debug container to the running process. This approach provides a method to inspect network traffic, trace system calls, and inspect files.

Before you begin
Copy link

  • If you use RHEL, the podman package is installed.

Procedure
Copy link

  • If you use Podman on RHEL:
    1. Identify the process ID (PID) of the process in the container that you want to debug: 
      $ podman top <debug_container_name>
       
      USER   PID  PPID  %CPU   ELAPSED            TTY  TIME  COMMAND
mysql  1    0     0.000  2h18m24.07081569s  ?    0s    mariadbd
    2. Use a Red Hat Hardened Image builder variant, and attach it to the target container: 
      $ podman run \
    --rm \
    --interactive \
    --tty \
    --name <debug_container_name> \
    --cap-add=SYS_PTRACE \
    --security-opt label=disable \
    --pid container:<target_container_id> \
    --network container:<target_container_id> \
    --user 0 \
    registry.access.redhat.com/hi/core-runtime:latest-builder bash

      This command starts a temporary interactive container that shares the process ID (PID) and network namespaces of your target container. It operates with root permissions and disabled SELinux labels to enable package installation and process inspection within the hardened environment. Using a builder variant ensures that a shell and the dnf package manager are available in the image. When you exit the debug container, Podman automatically removes the debug container.

    3. Inside the temporary container, identify files that are currently opened in the container by the identified PID: 
      # ls -l /proc/<pid>/fd
       
      ...
l-wx------. 1 999 999 64 Mar 12 10:50 1 -> /var/log/mariadb/mariadb.log
...
    4. Display the log file: 
      # cat /proc/<pid>/root/var/log/mariadb/mariadb.log

      You must specify the /proc/<pid>/root/ prefix to access files in the source container.

    5. Leave the temporary container: 
      # exit
    6. Optional: Copy the log file from the container to the current directory on the host: 
      $ podman cp <debug_container_name>:/var/log/mariadb/mariadb.log
  • If you use OpenShift, use Ephemeral Containers. For details, see How do I run top and ps commands when in a container without standard Linux tools.