Troubleshoot image builds that fail with an RLIMIT_NOFILE error

Recognize and resolve an image build failure caused by the host’s open-file-descriptor limit.

Symptom

An image build fails with an error similar to the following:

Error: building at STEP "RUN ... dnf install -y ... flightctl-agent ...":
setting "RLIMIT_NOFILE" limit to soft=1048576,hard=1048576
(was soft=524288,hard=524288): operation not permitted

Cause

During a build, buildah tries to raise the open-file-descriptor limit to 1048576. This fails when the host’s hard limit is lower, for example 524288 (systemd’s own default hard limit), because the build container does not have the `CAP_SYS_RESOURCE capability required to raise it further.

The image builder worker pod runs with the SYS_ADMIN, MKNOD, SYS_CHROOT, and SETFCAP capabilities, but not SYS_RESOURCE, and nothing raises the container’s file-descriptor limit automatically. As a result, this error can occur on any host whose default hard limit is lower than 1048576. Configure the host as described below before you build images.

Resolution

Choose the procedure for your deployment platform.

Red Hat OpenShift Container Platform

Apply the following MachineConfig to the worker nodes:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-raise-nofile
spec:
  config:
    ignition:
      version: 3.5.0
    storage:
      files:
        - contents:
            source: data:,%5BManager%5D%0ADefaultLimitNOFILE%3D1048576%3A1048576%0A
          mode: 420
          overwrite: true
          path: /etc/systemd/system.conf.d/60-nofile.conf

This sets DefaultLimitNOFILE=1048576:1048576 in systemd, which raises the limit for every process on the node, including container runtimes.

Important

Applying this MachineConfig triggers the Machine Config Operator to roll the change out to every node in the worker pool: each node is cordoned, its workloads are drained and rescheduled, the node reboots to apply the change, and then rejoins the cluster before the operator moves to the next node. Plan this change during a maintenance window, since it disrupts workloads on each node while its turn in the rollout is in progress.

 Note

DefaultLimitNOFILE is a system-wide default that applies to every service started by systemd that does not already set its own LimitNOFILE= directive.

Red Hat Enterprise Linux with Podman Quadlets

Create a systemd drop-in that raises the limit only for flightctl-imagebuilder-worker.service, without affecting any other service on the host:

$ sudo mkdir -p /etc/systemd/system/flightctl-imagebuilder-worker.service.d
$ sudo tee /etc/systemd/system/flightctl-imagebuilder-worker.service.d/limits.conf <<'EOF'
[Service]
LimitNOFILE=1048576:1048576
EOF

Reload systemd and restart the service:

$ sudo systemctl daemon-reload
$ sudo systemctl restart flightctl-imagebuilder-worker.service

Verification

Confirm that the running service uses the new limit:

$ systemctl show flightctl-imagebuilder-worker.service | grep LimitNOFILE

The output shows both LimitNOFILE and LimitNOFILESoft set to 1048576.