Chapter 8. Manage workloads with Kueue

1. Installs the Red Hat build of Kueue Operator on the cluster, as described in the Red Hat build of Kueue documentation in Additional resources.
2. Activates the Kueue integration by setting the managementState to Unmanaged in the DataScienceCluster custom resource, as described in Configure workload management with Kueue in Additional resources.

3. Chooses a queue management mode:

   • Operator-managed queues: Sets autoCreateQueues to true to have the Operator create default ClusterQueue and LocalQueue resources.
   • Self-managed queues: Keeps autoCreateQueues set to false or omits it, then creates ClusterQueue and LocalQueue resources outside the Operator by using GitOps, kubectl, or the Red Hat build of Kueue Operator directly. For more information, see Manage ClusterQueues and LocalQueues outside the Operator in Additional resources.
4. Configures quotas to optimize resource allocation for user workloads, as described in the Red Hat build of Kueue documentation in Additional resources.

5. Enables Kueue in the dashboard by setting disableKueue to false in the OdhDashboardConfig custom resource, as described in Enable Kueue in the dashboard in Additional resources.

   Note

   When Kueue is enabled in the dashboard, OpenShift AI automatically enables Kueue management for all new projects created from the dashboard.

1. Creates Kueue-enabled hardware profiles so that users can submit workloads from the OpenShift AI dashboard, as described in Working with hardware profiles in Additional resources.

1. For workloads created from the OpenShift AI dashboard, such as workbenches and model servers, selects a Kueue-enabled hardware profile during creation.
2. For workloads created by using a command-line interface or a software development kit (SDK), such as distributed training jobs, adds the kueue.x-k8s.io/queue-name label to the workload’s YAML manifest and sets its value to the target LocalQueue name.
3. After starting or creating a workbench, views the scheduling status on the Workbenches tab. For Kueue-managed workbenches, the status shows the current Kueue state, such as Queued or Starting, and the queue position when available.

Procedure

1. In a terminal window, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

2. Update the odh-dashboard-config custom resource in the OpenShift AI applications namespace. Replace <applications_namespace> with your OpenShift AI applications namespace. The default is redhat-ods-applications.

   $ oc patch odhdashboardconfig odh-dashboard-config \
     -n <applications_namespace> \
     --type merge \
     -p {"spec":{"dashboardConfig":{"disableKueue":false}}}

Verification

1. From the OpenShift AI dashboard, create a new project.

2. Verify that the project namespace is labeled for Kueue management:

   $ oc get ns <project_namespace> -o jsonpath='{.metadata.labels.kueue\.openshift\.io/managed}{"\n"}'

   The output should be true.

3. Confirm that a LocalQueue exists for the project namespace:

   $ oc get localqueues -n <project_namespace>

   The LocalQueue might be created by the dashboard for new projects, by the Operator when autoCreateQueues is true, or by the cluster administrator when managing queues outside the Operator.

4. Create a test workload, for example a Notebook, and verify that it includes the kueue.x-k8s.io/queue-name label.

Procedure

1. In a terminal window, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

2. Verify that autoCreateQueues is set to false or is not present in the DataScienceCluster resource:

   $ oc get datasciencecluster default-dsc -o jsonpath='{.spec.components.kueue}' | jq .

   The autoCreateQueues field is either absent or set to false. If the field is set to true, the Operator manages queue resources automatically and you do not need to create them manually.

3. Create a ClusterQueue resource with quota definitions. The following example creates a cluster queue with CPU, memory, and GPU quotas:

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: ClusterQueue
   metadata:
     name: <cluster_queue_name>
   spec:
     namespaceSelector: {}
     resourceGroups:
     - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
       flavors:
       - name: "<resource_flavor_name>"
         resources:
         - name: "cpu"
           nominalQuota: 9
         - name: "memory"
           nominalQuota: 36Gi
         - name: "nvidia.com/gpu"
           nominalQuota: 5

   where:

   <cluster_queue_name>

   Specifies the name of the cluster queue, for example team-gpu-cluster-queue.

   namespaceSelector: {}

   Specifies that all namespaces can submit workloads to this cluster queue. In multitenant environments, replace this with a label selector to restrict access to specific namespaces.

   <resource_flavor_name>

   Specifies the name of a ResourceFlavor you have already created. For ResourceFlavor configuration, see Configuring a resource flavor.

   nominalQuota

   Specifies the maximum amount of each resource that the cluster queue can allocate.

   Replace the example quota values with values appropriate for your cluster. If you use AMD GPUs, replace nvidia.com/gpu with amd.com/gpu in the example code.

   Apply the configuration:

   $ oc apply -f cluster_queue.yaml

4. Create a LocalQueue resource in each project namespace that points to the cluster queue. The following example creates a local queue in a project namespace:

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: LocalQueue
   metadata:
     name: <local_queue_name>
     namespace: <project_namespace>
   spec:
     clusterQueue: <cluster_queue_name>

   where:

   <local_queue_name>

   Specifies the name of the local queue, for example team-gpu-local-queue.

   <project_namespace>

   Specifies the namespace of the project that uses this queue.

   <cluster_queue_name>

   Specifies the name of the cluster queue you created in the previous step.

   Apply the configuration:

   $ oc apply -f local_queue.yaml

   Repeat this step for each project namespace that requires workload scheduling.

5. Enable Kueue management for each project namespace by applying the kueue.openshift.io/managed=true label:

   $ oc label namespace <project_namespace> kueue.openshift.io/managed=true --overwrite

Verification

1. Verify that the ClusterQueue resource was created and is active:

   $ oc get clusterqueues

2. Verify that the LocalQueue resource exists in the target namespace:

   $ oc get localqueues -n <project_namespace>

3. Verify that the local queue is ready to admit workloads:

   $ oc get localqueues -n <project_namespace> -o jsonpath='{.items[0].status.conditions}' | jq .

Resolution

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Verify that the Kueue pod is running. If necessary, restart the Kueue pod.

4. Review the logs for the Kueue pod to verify that the webhook server is serving, as shown in the following example:

   {"level":"info","ts":"2024-06-24T14:36:24.255137871Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443}

Resolution

1. Check whether a local queue exists in the user’s project, as follows:

   1. In the OpenShift console, select the user’s project from the Project list.
   2. Click Home Search, and from the Resources list, select LocalQueue.
   3. If no local queues are found, create a local queue.
   4. Provide the user with the details of the local queues in their project, and advise them to add a local queue to their cluster configuration.

2. Define a default local queue.

   For information about creating a local queue and defining a default local queue, see Configuring quota management for distributed workloads.

Resolution

1. In the OpenShift console, select the user’s project from the Project list.

   1. Click Search, and from the Resources list, select LocalQueue.

   2. Resolve the problem in one of the following ways:

      • If no local queues are found, create a local queue.
      • If one or more local queues are found, provide the user with the details of the local queues in their project. Advise the user to ensure that they spelled the local queue name correctly in their cluster configuration, and that the namespace value in the cluster configuration matches their project name.

   3. Define a default local queue.

      For information about creating a local queue and defining a default local queue, see Configuring quota management for distributed workloads.

Diagnosis

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Click the user’s pod name to open the pod details page.
4. Click the Events tab, and review the pod events to check whether the image pull completed successfully.