×

Important

Azure Red Hat OpenShift 3.11 will be retired 30 June 2022. Support for creation of new Azure Red Hat OpenShift 3.11 clusters continues through 30 November 2020. Following retirement, remaining Azure Red Hat OpenShift 3.11 clusters will be shut down to prevent security vulnerabilities.

Follow this guide to create an Azure Red Hat OpenShift 4 cluster. If you have specific questions, please contact us


Overview

Using quotas and limit ranges, cluster administrators can set constraints to limit the number of objects or amount of compute resources that are used in your project. This helps cluster administrators better manage and allocate resources across all projects, and ensure that no projects are using more than is appropriate for the cluster size.

The following sections help you understand how to check on your quota and limit range settings, what sorts of things they can constrain, and how you can request or limit compute resources in your own pods and containers.

Quotas

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per project. It can limit the quantity of objects that can be created in a project by type, as well as the total amount of compute resources and storage that may be consumed by resources in that project.

Quotas are set by cluster administrators and are scoped to a given project.

Viewing Quotas

You can view usage statistics related to any hard limits defined in a project’s quota by navigating in the web console to the project’s Quota page.

You can also use the CLI to view quota details:

  1. First, get the list of quotas defined in the project. For example, for a project called demoproject:

    $ oc get quota -n demoproject
    NAME                AGE
    besteffort          11m
    compute-resources   2m
    core-object-counts  29m
  2. Then, describe the quota you are interested in, for example the core-object-counts quota:

    $ oc describe quota core-object-counts -n demoproject
    Name:			core-object-counts
    Namespace:		demoproject
    Resource		Used	Hard
    --------		----	----
    configmaps		3	10
    persistentvolumeclaims	0	4
    replicationcontrollers	3	20
    secrets			9	10
    services		2	10

Full quota definitions can be viewed by running oc get --export on the object. The following show some sample quota definitions:

core-object-counts.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: core-object-counts
spec:
  hard:
    configmaps: "10" (1)
    persistentvolumeclaims: "4" (2)
    replicationcontrollers: "20" (3)
    secrets: "10" (4)
    services: "10" (5)
1 The total number of ConfigMap objects that can exist in the project.
2 The total number of persistent volume claims (PVCs) that can exist in the project.
3 The total number of replication controllers that can exist in the project.
4 The total number of secrets that can exist in the project.
5 The total number of services that can exist in the project.
openshift-object-counts.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: openshift-object-counts
spec:
  hard:
    openshift.io/imagestreams: "10" (1)
1 The total number of image streams that can exist in the project.
compute-resources.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    pods: "4" (1)
    requests.cpu: "1" (2)
    requests.memory: 1Gi (3)
    requests.ephemeral-storage: 2Gi (4)
    limits.cpu: "2" (5)
    limits.memory: 2Gi (6)
    limits.ephemeral-storage: 4Gi (7)
1 The total number of pods in a non-terminal state that can exist in the project.
2 Across all pods in a non-terminal state, the sum of CPU requests cannot exceed 1 core.
3 Across all pods in a non-terminal state, the sum of memory requests cannot exceed 1Gi.
4 Across all pods in a non-terminal state, the sum of ephemeral storage requests cannot exceed 2Gi.
5 Across all pods in a non-terminal state, the sum of CPU limits cannot exceed 2 cores.
6 Across all pods in a non-terminal state, the sum of memory limits cannot exceed 2Gi.
7 Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot exceed 4Gi.
besteffort.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: besteffort
spec:
  hard:
    pods: "1" (1)
  scopes:
  - BestEffort (2)
1 The total number of pods in a non-terminal state with BestEffort quality of service that can exist in the project.
2 Restricts the quota to only matching pods that have BestEffort quality of service for either memory or CPU.
compute-resources-long-running.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources-long-running
spec:
  hard:
    pods: "4" (1)
    limits.cpu: "4" (2)
    limits.memory: "2Gi" (3)
    limits.ephemeral-storage: "4Gi" (4)
  scopes:
  - NotTerminating (5)
1 The total number of pods in a non-terminal state.
2 Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value.
3 Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value.
4 Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot exceed this value.
5 Restricts the quota to only matching pods where spec.activeDeadlineSeconds is set to nil. Build pods will fall under NotTerminating unless the RestartNever policy is applied.
compute-resources-time-bound.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources-time-bound
spec:
  hard:
    pods: "2" (1)
    limits.cpu: "1" (2)
    limits.memory: "1Gi" (3)
    limits.ephemeral-storage: "1Gi" (4)
  scopes:
  - Terminating (5)
1 The total number of pods in a non-terminal state.
2 Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value.
3 Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value.
4 Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot exceed this value.
5 Restricts the quota to only matching pods where spec.activeDeadlineSeconds >=0. For example, this quota would charge for build or deployer pods, but not long running pods like a web server or database.
storage-consumption.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-consumption
spec:
  hard:
    persistentvolumeclaims: "10" (1)
    requests.storage: "50Gi" (2)
    gold.storageclass.storage.k8s.io/requests.storage: "10Gi" (3)
    silver.storageclass.storage.k8s.io/requests.storage: "20Gi" (4)
    silver.storageclass.storage.k8s.io/persistentvolumeclaims: "5" (5)
    bronze.storageclass.storage.k8s.io/requests.storage: "0" (6)
    bronze.storageclass.storage.k8s.io/persistentvolumeclaims: "0" (7)
1 The total number of persistent volume claims in a project
2 Across all persistent volume claims in a project, the sum of storage requested cannot exceed this value.
3 Across all persistent volume claims in a project, the sum of storage requested in the gold storage class cannot exceed this value.
4 Across all persistent volume claims in a project, the sum of storage requested in the silver storage class cannot exceed this value.
5 Across all persistent volume claims in a project, the total number of claims in the silver storage class cannot exceed this value.
6 Across all persistent volume claims in a project, the sum of storage requested in the bronze storage class cannot exceed this value. When this is set to 0, it means bronze storage class cannot request storage.
7 Across all persistent volume claims in a project, the sum of storage requested in the bronze storage class cannot exceed this value. When this is set to 0, it means bronze storage class cannot create claims.

Resources Managed by Quota

The following describes the set of compute resources and object types that may be managed by a quota.

A pod is in a terminal state if status.phase in (Failed, Succeeded) is true.

Table 1. Compute Resources Managed by Quota
Resource Name Description

cpu

The sum of CPU requests across all pods in a non-terminal state cannot exceed this value. cpu and requests.cpu are the same value and can be used interchangeably.

memory

The sum of memory requests across all pods in a non-terminal state cannot exceed this value. memory and requests.memory are the same value and can be used interchangeably.

ephemeral-storage

The sum of local ephemeral storage requests across all pods in a non-terminal state cannot exceed this value. ephemeral-storage and requests.ephemeral-storage are the same value and can be used interchangeably. This resource is available only if you enabled the ephemeral storage technology preview. This feature is disabled by default.

requests.cpu

The sum of CPU requests across all pods in a non-terminal state cannot exceed this value. cpu and requests.cpu are the same value and can be used interchangeably.

requests.memory

The sum of memory requests across all pods in a non-terminal state cannot exceed this value. memory and requests.memory are the same value and can be used interchangeably.

requests.ephemeral-storage

The sum of ephemeral storage requests across all pods in a non-terminal state cannot exceed this value. ephemeral-storage and requests.ephemeral-storage are the same value and can be used interchangeably. This resource is available only if you enabled the ephemeral storage technology preview. This feature is disabled by default.

limits.cpu

The sum of CPU limits across all pods in a non-terminal state cannot exceed this value.

limits.memory

The sum of memory limits across all pods in a non-terminal state cannot exceed this value.

limits.ephemeral-storage

The sum of ephemeral storage limits across all pods in a non-terminal state cannot exceed this value. This resource is available only if you enabled the ephemeral storage technology preview. This feature is disabled by default.

Table 2. Storage Resources Managed by Quota
Resource Name Description

requests.storage

The sum of storage requests across all persistent volume claims in any state cannot exceed this value.

persistentvolumeclaims

The total number of persistent volume claims that can exist in the project.

<storage-class-name>.storageclass.storage.k8s.io/requests.storage

The sum of storage requests across all persistent volume claims in any state that have a matching storage class, cannot exceed this value.

<storage-class-name>.storageclass.storage.k8s.io/persistentvolumeclaims

The total number of persistent volume claims with a matching storage class that can exist in the project.

Table 3. Object Counts Managed by Quota
Resource Name Description

pods

The total number of pods in a non-terminal state that can exist in the project.

replicationcontrollers

The total number of replication controllers that can exist in the project.

resourcequotas

The total number of resource quotas that can exist in the project.

services

The total number of services that can exist in the project.

secrets

The total number of secrets that can exist in the project.

configmaps

The total number of ConfigMap objects that can exist in the project.

persistentvolumeclaims

The total number of persistent volume claims that can exist in the project.

openshift.io/imagestreams

The total number of image streams that can exist in the project.

Quota Scopes

Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches the intersection of enumerated scopes.

Adding a scope to a quota restricts the set of resources to which that quota can apply. Specifying a resource outside of the allowed set results in a validation error.

Scope Description

Terminating

Match pods where spec.activeDeadlineSeconds >= 0.

NotTerminating

Match pods where spec.activeDeadlineSeconds is nil.

BestEffort

Match pods that have best effort quality of service for either cpu or memory.

NotBestEffort

Match pods that do not have best effort quality of service for cpu and memory.

A BestEffort scope restricts a quota to limiting the following resources:

  • pods

A Terminating, NotTerminating, and NotBestEffort scope restricts a quota to tracking the following resources:

  • pods

  • memory

  • requests.memory

  • limits.memory

  • cpu

  • requests.cpu

  • limits.cpu

  • ephemeral-storage

  • requests.ephemeral-storage

  • limits.ephemeral-storage

Ephemeral storage requests and limits apply only if you enabled the ephemeral storage technology preview. This feature is disabled by default.

Quota Enforcement

After a resource quota for a project is first created, the project restricts the ability to create any new resources that may violate a quota constraint until it has calculated updated usage statistics.

After a quota is created and usage statistics are updated, the project accepts the creation of new content. When you create or modify resources, your quota usage is incremented immediately upon the request to create or modify the resource.

When you delete a resource, your quota use is decremented during the next full recalculation of quota statistics for the project. If project modifications exceed a quota usage limit, the server denies the action. An appropriate error message is returned explaining the quota constraint violated, and what your currently observed usage stats are in the system.

Requests Versus Limits

When allocating compute resources, each container may specify a request and a limit value each for CPU, memory, and ephemeral storage. Quotas can restrict any of these values.

If the quota has a value specified for requests.cpu or requests.memory, then it requires that every incoming container make an explicit request for those resources. If the quota has a value specified for limits.cpu or limits.memory, then it requires that every incoming container specify an explicit limit for those resources.

See Compute Resources for more on setting requests and limits in pods and containers.

Limit Ranges

A limit range, defined by a LimitRange object, enumerates compute resource constraints in a project at the pod, container, image, image stream, and persistent volume claim level, and specifies the amount of resources that a pod, container, image, image stream, or persistent volume claim can consume.

All requests to create and modify resources are evaluated against each LimitRange object in the project. If the resource violates any of the enumerated constraints, the resource is rejected. If the resource does not set an explicit value, and if the constraint supports a default value, the default value is applied to the resource.

For CPU and memory limits, if you specify a maximum value but do not specify a minimum limit, the resource can consume more CPU and memory resources than the maximum value.

Limit ranges are set by cluster administrators and are scoped to a given project.

Viewing Limit Ranges

You can view any limit ranges that are defined in a project by navigating in the web console to the Quota page for the project.

You can also use the CLI to view limit range details by performing the following steps:

  1. Get the list of limit range objects that are defined in the project. For example, for a project called demoproject:

    $ oc get limits -n demoproject
    Example Output
    NAME              AGE
    resource-limits   6d
  2. Describe the limit range. For example, for a limit range called resource-limits:

    $ oc describe limits resource-limits -n demoproject
    Example Output
    Name:                           resource-limits
    Namespace:                      demoproject
    Type                            Resource                Min     Max     Default Request Default Limit   Max Limit/Request Ratio
    ----                            --------                ---     ---     --------------- -------------   -----------------------
    Pod                             cpu                     200m    2       -               -               -
    Pod                             memory                  6Mi     1Gi     -               -               -
    Container                       cpu                     100m    2       200m            300m            10
    Container                       memory                  4Mi     1Gi     100Mi           200Mi           -
    openshift.io/Image              storage                 -       1Gi     -               -               -
    openshift.io/ImageStream        openshift.io/image      -       12      -               -               -
    openshift.io/ImageStream        openshift.io/image-tags -       10      -               -               -

Full limit range definitions can be viewed by running oc get --export on the object. The following shows an example limit range definition:

Core Limit Range Object Definition
apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "core-resource-limits" (1)
spec:
  limits:
    - type: "Pod"
      max:
        cpu: "2" (2)
        memory: "1Gi" (3)
      min:
        cpu: "200m" (4)
        memory: "6Mi" (5)
    - type: "Container"
      max:
        cpu: "2" (6)
        memory: "1Gi" (7)
      min:
        cpu: "100m" (8)
        memory: "4Mi" (9)
      default:
        cpu: "300m" (10)
        memory: "200Mi" (11)
      defaultRequest:
        cpu: "200m" (12)
        memory: "100Mi" (13)
      maxLimitRequestRatio:
        cpu: "10" (14)
1 The name of the limit range object.
2 The maximum amount of CPU that a pod can request on a node across all containers.
3 The maximum amount of memory that a pod can request on a node across all containers.
4 The minimum amount of CPU that a pod can request on a node across all containers. If you do not set a min value or you set min to 0, the result is no limit and the pod can consume more than the max CPU value.
5 The minimum amount of memory that a pod can request on a node across all containers. If you do not set a min value or you set min to 0, the result is no limit and the pod can consume more than the max memory value.
6 The maximum amount of CPU that a single container in a pod can request.
7 The maximum amount of memory that a single container in a pod can request.
8 The minimum amount of CPU that a single container in a pod can request. If you do not set a min value or you set min to 0, the result is no limit and the pod can consume more than the max CPU value.
9 The minimum amount of memory that a single container in a pod can request. If you do not set a min value or you set min to 0, the result is no limit and the pod can consume more than the max memory value.
10 The default CPU limit for a container if you do not specify a limit in the pod specification.
11 The default memory limit for a container if you do not specify a limit in the pod specification.
12 The default CPU request for a container if you do not specify a request in the pod specification.
13 The default memory request for a container if you do not specify a request in the pod specification.
14 The maximum limit-to-request ratio for a container.

For more information on how CPU and memory are measured, see Compute Resources.

Container Limits

Supported Resources:

  • CPU

  • Memory

Supported Constraints:

Per container, the following must hold true if specified:

Table 4. Container
Constraint Behavior

Min

Min[resource] less than or equal to container.resources.requests[resource] (required) less than or equal to container/resources.limits[resource] (optional)

If the configuration defines a min CPU, the request value must be greater than the CPU value. If you do not set a min value or you set min to 0, the result is no limit and the pod can consume more of the resource than the max value.

Max

container.resources.limits[resource] (required) less than or equal to Max[resource]

If the configuration defines a max CPU, you do not need to define a CPU request value. However, you must set a limit that satisfies the maximum CPU constraint that is specified in the limit range.

MaxLimitRequestRatio

MaxLimitRequestRatio[resource] less than or equal to (container.resources.limits[resource] / container.resources.requests[resource])

If the limit range defines a maxLimitRequestRatio constraint, any new containers must have both a request and a limit value. Additionally, Azure Red Hat OpenShift calculates a limit-to-request ratio by dividing the limit by the request. The result should be an integer greater than 1.

For example, if a container has cpu: 500 in the limit value, and cpu: 100 in the request value, the limit-to-request ratio for cpu is 5. This ratio must be less than or equal to the maxLimitRequestRatio.

Supported Defaults:

Default[resource]

Defaults container.resources.limit[resource] to specified value if none.

Default Requests[resource]

Defaults container.resources.requests[resource] to specified value if none.

Pod Limits

Supported Resources:

  • CPU

  • Memory

Supported Constraints:

Across all containers in a pod, the following must hold true:

Table 5. Pod
Constraint Enforced Behavior

Min

Min[resource] less than or equal to container.resources.requests[resource] (required) less than or equal to container.resources.limits[resource]. If you do not set a min value or you set min to 0, the result is no limit and the pod can consume more of the resource than the max value.

Max

container.resources.limits[resource] (required) less than or equal to Max[resource].

MaxLimitRequestRatio

MaxLimitRequestRatio[resource] less than or equal to (container.resources.limits[resource] / container.resources.requests[resource]).

Compute Resources

Each container running on a node consumes compute resources, which are measurable quantities that can be requested, allocated, and consumed.

When authoring a pod configuration file, you can optionally specify how much CPU, memory (RAM), and local ephemeral storage (if your administrator enabled the ephemeral storage technology preview) each container needs in order to better schedule pods in the cluster and ensure satisfactory performance.

CPU is measured in units called millicores. Each node in a cluster inspects the operating system to determine the amount of CPU cores on the node, then multiplies that value by 1000 to express its total capacity. For example, if a node has 2 cores, the node’s CPU capacity would be represented as 2000m. If you wanted to use 1/10 of a single core, it would be represented as 100m.

Memory and ephemeral storage are measured in bytes. In addition, it may be used with SI suffixes (E, P, T, G, M, K) or their power-of-two-equivalents (Ei, Pi, Ti, Gi, Mi, Ki).

apiVersion: v1
kind: Pod
spec:
  containers:
  - image: openshift/hello-openshift
    name: hello-openshift
    resources:
      requests:
        cpu: 100m (1)
        memory: 200Mi (2)
        ephemeral-storage: 1Gi (3)
      limits:
        cpu: 200m (4)
        memory: 400Mi (5)
        ephemeral-storage: 2Gi (6)
1 The container requests 100m CPU.
2 The container requests 200Mi memory.
3 The container requests 1Gi ephemeral storage. Your administrator must enable the ephemeral storage technology preview to specify this parameter value.
4 The container limits 200m CPU.
5 The container limits 400Mi memory.
6 The container limits 2Gi ephemeral storage. Your administrator must enable the ephemeral storage technology preview to specify this parameter value.

CPU Requests

Each container in a pod can specify the amount of CPU it requests on a node. The scheduler uses CPU requests to find a node with an appropriate fit for a container.

The CPU request represents a minimum amount of CPU that your container may consume, but if there is no contention for CPU, it can use all available CPU on the node. If there is CPU contention on the node, CPU requests provide a relative weight across all containers on the system for how much CPU time the container may use.

On the node, CPU requests map to Kernel CFS shares to enforce this behavior.

Viewing Compute Resources

To view compute resources for a pod:

$ oc describe pod ruby-hello-world-tfjxt
Name:       ruby-hello-world-tfjxt
Namespace:      default
Image(s):     ruby-hello-world
Node:       /
Labels:       run=ruby-hello-world
Status:       Pending
Reason:
Message:
IP:
Replication Controllers:  ruby-hello-world (1/1 replicas created)
Containers:
  ruby-hello-world:
    Container ID:
    Image ID:
    Image:    ruby-hello-world
    QoS Tier:
      cpu:  Burstable
      memory: Burstable
    Limits:
      cpu:  200m
      memory: 400Mi
      ephemeral-storage: 1Gi
    Requests:
      cpu:    100m
      memory:   200Mi
      ephemeral-storage: 2Gi
    State:    Waiting
    Ready:    False
    Restart Count:  0
    Environment Variables:

CPU Limits

Each container in a pod can specify the amount of CPU it is limited to use on a node. CPU limits control the maximum amount of CPU that your container may use independent of contention on the node. If a container attempts to exceed the specified limit, the system will throttle the container. This allows the container to have a consistent level of service independent of the number of pods scheduled to the node.

Memory Requests

By default, a container is able to consume as much memory on the node as possible. In order to improve placement of pods in the cluster, specify the amount of memory required for a container to run. The scheduler will then take available node memory capacity into account prior to binding your pod to a node. A container is still able to consume as much memory on the node as possible even when specifying a request.

Ephemeral Storage Requests

This topic applies only if your administrator enabled the ephemeral storage technology preview.

By default, a container is able to consume as much local ephemeral storage on the node as is available. In order to improve placement of pods in the cluster, specify the amount of required local ephemeral storage for a container to run. The scheduler will then take available node local storage capacity into account prior to binding your pod to a node. A container is still able to consume as much local ephemeral storage on the node as possible even when specifying a request.

Memory Limits

If you specify a memory limit, you can constrain the amount of memory the container can use. For example, if you specify a limit of 200Mi, a container will be limited to using that amount of memory on the node. If the container exceeds the specified memory limit, it will be terminated and potentially restarted dependent upon the container restart policy.

Ephemeral Storage Limits

This topic applies only if your administrator enabled the ephemeral storage technology preview.

If you specify an ephemeral storage limit, you can constrain the amount of ephemeral storage the container can use. For example, if you specify a limit of 2Gi, a container will be limited to using that amount of ephemeral storage on the node. If the container exceeds the specified memory limit, it will be terminated and potentially restarted dependent upon the container restart policy.

Quality of Service Tiers

When created, a compute resource is classified with a quality of service (QoS). There are three tiers, and each is based on the request and limit value specified for each resource:

Quality of Service Description

BestEffort

Provided when a request and limit are not specified.

Burstable

Provided when a request is specified that is less than an optionally specified limit.

Guaranteed

Provided when a limit is specified that is equal to an optionally specified request.

If a container has requests and limits set that would result in a different quality of service for each compute resource, it will be classified as Burstable.

The quality of service has different impacts on different resources, depending on whether the resource is compressible or not. CPU is a compressible resource, whereas memory is an incompressible resource.

With CPU Resources:
  • A BestEffort CPU container is able to consume as much CPU as is available on a node but runs with the lowest priority.

  • A Burstable CPU container is guaranteed to get the minimum amount of CPU requested, but it may or may not get additional CPU time. Excess CPU resources are distributed based on the amount requested across all containers on the node.

  • A Guaranteed CPU container is guaranteed to get the amount requested and no more, even if there are additional CPU cycles available. This provides a consistent level of performance independent of other activity on the node.

With Memory Resources:
  • A BestEffort memory container is able to consume as much memory as is available on the node, but there are no guarantees that the scheduler will place that container on a node with enough memory to meet its needs. In addition, a BestEffort container has the greatest chance of being killed if there is an out of memory event on the node.

  • A Burstable memory container is scheduled on the node to get the amount of memory requested, but it may consume more. If there is an out of memory event on the node, Burstable containers are killed after BestEffort containers when attempting to recover memory.

  • A Guaranteed memory container gets the amount of memory requested, but no more. In the event of an out of memory event, it will only be killed if there are no more BestEffort or Burstable containers on the system.

Specifying Compute Resources via CLI

To specify compute resources via the CLI:

Project Resource Limits

Resource limits can be set per-project by cluster administrators. Developers do not have the ability to create, edit, or delete these limits, but can view them for projects they have access to.