OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history
×

Deploying OpenShift sandboxed containers workloads

Preparing your cluster for OpenShift sandboxed containers

Before you install OpenShift sandboxed containers, ensure that your OpenShift Container Platform cluster meets the following requirements:

  • Your cluster must be installed on bare metal infrastructure with Red Hat Enterprise Linux CoreOS (RHCOS) workers. Your cluster must use installer-provisioned infrastructure.

    • OpenShift sandboxed containers only supports RHCOS worker nodes. RHEL 7 or RHEL 8 nodes are not supported.

    • Nested virtualization is not supported.

Additional resource requirements for OpenShift sandboxed containers

OpenShift sandboxed containers is a product that brings the ability to run workloads inside a sandboxed runtime, such as Kata Containers, to your OpenShift Container Platform clusters. Each pod is represented by a virtual machine (VM). Each VM runs in a qemu process and hosts a kata-agent process that acts as a supervisor for managing container workloads and processes that are running in those containers. There are two additional processes that add more overhead:

  • containerd-shim-kata-v2 is used to communicate with the pod.

  • virtiofsd handles host file system access on behalf of the guest.

Each VM is configured with a default amount of memory. Additional memory is hot-plugged into the VM for containers that explicitly request memory.

  • If a container runs without a given memory resource, it is able to consume free memory. It will do so until the total memory used by the VM reaches the default allocation. The guest and its I/O buffers also consume memory.

  • If a container is given a specific amount of memory, then that memory is hot-plugged into the VM before the container starts.

  • If a memory limit is specified, then the workload is terminated if it consumes more memory than the limit. If no memory limit is specified, the kernel that is running on the virtual machine might run out of memory. If the kernel runs out of memory it might terminate other processes on the virtual machine.

Default memory sizes

The following table lists some the default values for resource allocation.

Resource Value

Memory allocated by default to a virtual machine

2Gi

Guest Linux kernel memory usage at boot

~110Mi

Memory used by the QEMU process (excluding VM memory)

~30Mi

Memory used by the virtiofsd process (excluding VM I/O buffers)

~10Mi

Memory used by the containerd-shim-kata-v2 process

~20Mi

File buffer cache data after running dnf install on Fedora

~300Mi* [1]

  1. File buffers appear and are accounted for in multiple locations:

    • In the guest where it appears as file buffer cache.

    • In the virtiofsd daemon that maps allowed user-space file I/O operations.

    • In the QEMU process as guest memory.

Total memory usage is properly accounted for by the memory utilization metrics, which only count that memory once.

Pod overhead describes the amount of system resources that a pod on a node uses. You can get the current pod overhead for the kata runtime class by using oc describe runtimeclass kata as shown below.

Example
$ oc describe runtimeclass kata
Example output
Name:         kata
[...]
Metadata:
[...]
Overhead:
  Pod Fixed:
    Cpu:     250m
    Memory:  350Mi
[...]

You can change the pod overhead by changing the spec.overhead field for a RuntimeClass. For instance, if the configuration that you run for your containers consumes more than 350Mi of memory for the QEMU process and guest kernel data, you can alter the RuntimeClass overhead to suit your needs.

The specified default overhead values are supported by Red Hat. Changing default overhead values is not supported and can result in technical issues.
Example
kind: RuntimeClass
apiVersion: node.k8s.io/v1
metadata:
  name: kata
overhead:
  podFixed:
    memory: "500Mi"
    cpu: "500m"

  • The default allocation for virtual machines is 2Gi.

  • The Linux kernel uses approximately 100Mi of memory at boot time.

  • The QEMU process uses approximately 30Mi of memory.

  • The virtiofsd process uses approximately 10Mi of memory.

  • The shim-v2 process uses approximately 20Mi of memory.

When performing any kind of file system I/O in the guest, file buffers are allocated in the guest kernel. The file buffers are also mapped in the QEMU process on the host, as well as on the virtiofsd process. For example, if you use 300Mi of file buffer cache in the guest, both QEMU and virtiofsd appear to use 300Mi additional memory. However, the same memory is being used in all three cases. In other words, the total memory usage is only 300Mi, mapped in three different places. This is correctly accounted for when reporting the memory utilization metrics.

Additional resources

Deploying OpenShift sandboxed containers Operator using the web console

You can install the Operator and view your workloads from the web console.

Installing the OpenShift sandboxed containers Operator using the web console

You can install the OpenShift sandboxed containers Operator from the OpenShift Container Platform web console.

Prerequisites

  • You have OpenShift Container Platform 4.8 installed.

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Open a browser window and log in to the OpenShift Container Platform web console.

  2. From the Administrator perspective, navigate to Operators → OperatorHub.

  3. In the Filter by keyword field, type OpenShift sandboxed containers.

  4. Select the OpenShift sandboxed containers tile.

  5. Read the information about the Operator and click Install.

  6. On the Install Operator page:

    1. Select preview-1.0 from the list of available Update Channel options. This ensures that you install the version of OpenShift sandboxed containers that is compatible with your OpenShift Container Platform version.

    2. For Installed Namespace, ensure that the Operator recommended namespace option is selected. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace, which is automatically created if it does not exist.

      Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

    3. For Approval Strategy, ensure that Automatic, which is the default value, is selected. OpenShift sandboxed containers automatically updates when a new z-stream release is available.

  7. Click Install to make the Operator available to the OpenShift sandboxed containers namespace.

The OpenShift sandboxed containers Operator is now installed on your cluster. You can trigger the Operator by enabling the runtime on your cluster. You can do this by creating the KataConfig custom resource using the OpenShift CLI (oc).

apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig

Viewing OpenShift sandboxed containers workloads from the web console

OpenShift sandboxed containers based workloads look and feel the same as normal workloads when viewed in the web console. The only difference between the two is the runtimeClassName. runtimeClassName is what decides the runtime used for workloads. In this context, the runtime enabled by OpenShift sandboxed containers-based is kata. You can view the runtimeClass that the pods for your workloads use.

Prerequisites

  • You have OpenShift Container Platform 4.8 installed on your cluster.

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Navigate to Administration → Workloads.

  2. Identify the type of workload you want to view details for. For example, Pod, Deployment, DeploymentConfigs objects and so on.

  3. Choose the corresponding workload from the list.

  4. On the Details page, navigate to runtimeClass.

  5. Hover over runtimeClass to view more information. If kata was used as the runtime, the value of the runtimeClass is kata.

Deploying OpenShift sandboxed containers Operator using the CLI

You can install and deploy the Operator and view workloads from the CLI.

Installing the OpenShift sandboxed containers Operator using the CLI

You can install the OpenShift sandboxed containers Operator using the OpenShift Container Platform CLI.

Prerequisites

  • You have OpenShift Container Platform 4.8 installed on your cluster.

  • You have installed the OpenShift CLI (oc).

  • You have access to the cluster as a user with the cluster-admin role.

  • You have subscribed to the OpenShift sandboxed containers catalog.

    Subscribing to the OpenShift sandboxed containers catalog provides openshift-sandboxed-containers-operator namespace access to the OpenShift sandboxed containers Operator.
Procedure

  1. Create a YAML file that contains the following manifest:

    apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sandboxed-containers-operator
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-sandboxed-containers-kataconfig-group
  namespace: openshift-sandboxed-containers-operator
spec:
  targetNamespaces:
    - openshift-sandboxed-containers-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sandboxed-containers-operatorhub
  namespace: openshift-sandboxed-containers-operator
spec:
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  name: sandboxed-containers-operator
  startingCSV: sandboxed-containers-operator.v1.0.0
  channel: "preview-1.0"
  approval: "Automatic"

    Using the preview-1.0 channel ensures that you install the version of OpenShift sandboxed containers that is compatible with your OpenShift Container Platform version.

  2. Create the required Namespace, OperatorGroup, and Subscription objects for OpenShift sandboxed containers:

    $ oc create -f <file name>.yaml

  3. Ensure that the Operator is correctly installed:

    $ oc get csv -n openshift-sandboxed-containers-operator
    Example output
    NAME                             DISPLAY                                  VERSION  REPLACES                    PHASE
openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.0.0    <csv-of-previous-version>   Succeeded

  4. View the available deployments:

    $ oc get deployments -n openshift-sandboxed-containers-operator
    Example output
    NAME                                        READY  UP-TO-DATE   AVAILABLE   AGE
openshift-sandboxed-containers-operator                         1/111       9m48s
Verification

  • Verify that the Operator is up and running, so you can create the KataConfig resource to trigger the installation.

    $ oc get deployments -n openshift-sandboxed-containers-operator
    Example output
    NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
openshift-sandboxed-containers-controller-manager   1/1     1            1           40d
Additional resources

Triggering the installation of the Kata runtime

You must create one KataConfig custom resource (CR) to trigger the OpenShift sandboxed containers Operator to do the following:

  • Install the needed RHCOS extensions, such as QEMU and kata-containers, on your RHCOS node.

  • Ensure that the runtime, CRI-O, is configured with the correct Kata runtime handlers.

  • Create a RuntimeClass custom resource with necessary configurations for additional overhead caused by virtualization and the required additional processes.

Prerequisites

  • You have OpenShift Container Platform 4.8 installed on your cluster.

  • You have installed the OpenShift CLI (oc).

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Create the KataConfig resource:

    $ oc create -f <file name>.yaml
    Example
    apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: cluster-kataconfig

  2. Monitor the installation progress.

    • You can describe the KataConfig installation:

      $ oc describe kataconfig

      • Verify the Completed nodes field in the status.

      • If the value of Completed nodes matches the number of worker nodes, then the installation is completed. The status also contains a list of nodes where the installation is completed.

    • You can check the progress of the installation by watching the KataConfig resource:

      $ watch -n 10 oc describe kataconfig

      Alternatively, you can check the status of the KataConfig resource. This can be done by running oc get KataConfig <name> -oyaml and inspecting the status field in the output.

The Kata runtime is now installed on the cluster and ready for use as a secondary runtime. Verify that you see a newly created RuntimeClass for Kata on your cluster.

OpenShift sandboxed containers installs Kata only as a secondary optional runtime on the cluster and not as the primary runtime.
Verification

  • You can monitor the values of the KataConfig custom resource by running:

    $ watch oc describe KataConfig cluster-kataconfig
Additional resources

Selecting nodes for OpenShift sandboxed containers

You can selectively install the Kata runtime on specific workers.

Prerequisites

  • You have OpenShift Container Platform 4.8 installed on your cluster.

  • You have installed the OpenShift CLI (oc).

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Identify the labels that you want to use for selecting your nodes. For this example, use labels to selects to be chosen as candidates to run on your OpenShift sandboxed containers workloads. If the nodes exist, they are selected.

    1. To apply a label to a node, run the following command:

      $ oc label node <worker_node_name> <label>=<value>

      This labels your worker node with the <label> label that has a value of <value>.

  2. To add a label selector, edit the KataConfig custom resource (CR):

    $ oc edit kataconfig
    Example
      apiVersion: kataconfiguration.openshift.io/v1
  kind: KataConfig
  metadata:
    name: cluster-kataconfig
  spec:
    kataConfigPoolSelector:
      matchLabels:
         custom-kata-machine-pool: 'true'
Verification

  • You can check to see if the nodes in the machine-config-pool object are going through a config update.

    • If you are using the default nodes, you can monitor the machine-config-pool resource by running:

      $ watch oc get mcp worker

    • If you are using selected nodes, you can monitor the machine-config-pool resource by running:

      $ watch oc get mcp kata-oc

  • You can run watch oc describe kataconfig cluster-kataconfig to display information about sandboxed-containers extension failure on a node. The information is gathered from the status of the machine-config-pool object. You can view the information by running:

    $ oc describe mcp <machine-config-pool>

Scheduling OpenShift sandboxed containers workloads

You can schedule your workloads to run on OpenShift sandboxed containers.

Prerequisites

  • You have OpenShift Container Platform 4.8 installed on your cluster.

  • You have installed the OpenShift CLI (oc).

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Add runtimeClassName: kata to any pod-templated resources:

    • Pod objects

    • ReplicaSet objects

    • ReplicationController objects

    • StatefulSet objects

    • Deployment objects

    • DeploymentConfig objects

Example for Pod objects
  apiVersion: v1
  kind: Pod
  metadata:
   name: mypod
  spec:
    runtimeClassName: kata
Example for Deployment objects
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: mypod
    labels:
      app: mypod
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: mypod
    template:
      metadata:
        labels:
          app: mypod
      spec:
        runtimeClassName: kata
        containers:
        - name: mypod
          image: myImage

After the pod-templated resource is created with runtimeClassName: kata, OpenShift Container Platform begins scheduling the workload on OpenShift sandboxed containers enabled nodes. If no selector is used, the default is set to all worker nodes. Your workload runs on OpenShift sandboxed containers.

Viewing OpenShift sandboxed containers workloads from the CLI

You can view the runtimeClass that the pods for your workloads use from the CLI.

Prerequisites

  • You have OpenShift Container Platform 4.8 installed on your cluster.

  • You have installed the OpenShift CLI (oc).

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  • Inspect the runtimeClassName field on the pod to see a pod running on OpenShift sandboxed containers versus a normal container.

    • On the node, each pod has a corresponding qemu process.

Verification

  • You can check the logs of the openshift-sandboxed-containers-operator controller pod to see detailed messages about the steps it is running.

    • You can retrieve the name of the controller pod by running:

      $ oc get pods -n openshift-sandboxed-containers-operator | grep openshift-sandboxed-containers-operator-controller-manager

      This enables you to monitor the logs of the container manager of that pod.