OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history bug_report picture_as_pdf
×

Configuring a cluster-wide proxy

edit

If you are using an existing Virtual Private Cloud (VPC), you can configure a cluster-wide proxy during a Red Hat OpenShift Service on AWS (ROSA) cluster installation or after the cluster is installed. When you enable a proxy, the core cluster components are denied direct access to the internet, but the proxy does not affect user workloads.

Only cluster system egress traffic is proxied, including calls to the cloud provider API.

If you use a cluster-wide proxy, you are responsible for maintaining the availability of the proxy to the cluster. If the proxy becomes unavailable, then it might impact the health and supportability of the cluster.

edit

Prerequisites for configuring a cluster-wide proxy

To configure a cluster-wide proxy, you must meet the following requirements. These requirements are valid when you configure a proxy during installation or postinstallation.

General requirements

  • You are the cluster owner.

  • Your account has sufficient privileges.

  • You have an existing Virtual Private Cloud (VPC) for your cluster.

  • The proxy can access the VPC for the cluster and the private subnets of the VPC. The proxy is also accessible from the VPC for the cluster and from the private subnets of the VPC.

  • You have added the following endpoints to your VPC endpoint:

    • ec2.<aws_region>.amazonaws.com

    • elasticloadbalancing.<aws_region>.amazonaws.com

    • s3.<aws_region>.amazonaws.com

      These endpoints are required to complete requests from the nodes to the AWS EC2 API. Because the proxy works at the container level and not at the node level, you must route these requests to the AWS EC2 API through the AWS private network. Adding the public IP address of the EC2 API to your allowlist in your proxy server is not enough.

      When using a cluster-wide proxy, you must configure the s3.<aws_region>.amazonaws.com endpoint as type Gateway.

Network requirements

  • If your proxy re-encrypts egress traffic, you must create exclusions to the domain and port combinations. The following table offers guidance into these exceptions.

    • Your proxy must exclude re-encrypting the following OpenShift URLs:

      Address Protocol/Port Function

      observatorium-mst.api.openshift.com

      https/443

      Required. Used for Managed OpenShift-specific telemetry.

      sso.redhat.com

      https/443

      The https://cloud.redhat.com/openshift site uses authentication from sso.redhat.com to download the cluster pull secret and use Red Hat SaaS solutions to facilitate monitoring of your subscriptions, cluster inventory, and chargeback reporting.

    • Your proxy must exclude re-encrypting the following site reliability engineering (SRE) and management URLs:

      Address Protocol/Port Function

      *.osdsecuritylogs.splunkcloud.com

      OR

      inputs1.osdsecuritylogs.splunkcloud.com inputs2.osdsecuritylogs.splunkcloud.com inputs4.osdsecuritylogs.splunkcloud.com inputs5.osdsecuritylogs.splunkcloud.com inputs6.osdsecuritylogs.splunkcloud.com inputs7.osdsecuritylogs.splunkcloud.com inputs8.osdsecuritylogs.splunkcloud.com inputs9.osdsecuritylogs.splunkcloud.com inputs10.osdsecuritylogs.splunkcloud.com inputs11.osdsecuritylogs.splunkcloud.com inputs12.osdsecuritylogs.splunkcloud.com inputs13.osdsecuritylogs.splunkcloud.com inputs14.osdsecuritylogs.splunkcloud.com inputs15.osdsecuritylogs.splunkcloud.com

      tcp/9997

      Used by the splunk-forwarder-operator as a log forwarding endpoint to be used by Red Hat SRE for log-based alerting.

      http-inputs-osdsecuritylogs.splunkcloud.com

      https/443

      Used by the splunk-forwarder-operator as a log forwarding endpoint to be used by Red Hat SRE for log-based alerting.
Additional resources
edit

Responsibilities for additional trust bundles

If you supply an additional trust bundle, you are responsible for the following requirements:

  • Ensuring that the contents of the additional trust bundle are valid

  • Ensuring that the certificates, including intermediary certificates, contained in the additional trust bundle have not expired

  • Tracking the expiry and performing any necessary renewals for certificates contained in the additional trust bundle

  • Updating the cluster configuration with the updated additional trust bundle

edit

Configuring a proxy during installation

You can configure an HTTP or HTTPS proxy when you install a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC). You can configure the proxy during installation by using Red Hat OpenShift Cluster Manager or the ROSA CLI (rosa).

edit

Configuring a proxy during installation using OpenShift Cluster Manager

If you are installing a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC), you can use Red Hat OpenShift Cluster Manager to enable a cluster-wide HTTP or HTTPS proxy during installation.

Prior to the installation, you must verify that the proxy is accessible from the VPC that the cluster is being installed into. The proxy must also be accessible from the private subnets of the VPC.

For detailed steps to configure a cluster-wide proxy during installation by using OpenShift Cluster Manager, see Creating a cluster with customizations by using OpenShift Cluster Manager.

edit

Configuring a proxy during installation using the CLI

If you are installing a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC), you can use the ROSA CLI (rosa) to enable a cluster-wide HTTP or HTTPS proxy during installation.

The following procedure provides details about the ROSA CLI (rosa) arguments that are used to configure a cluster-wide proxy during installation. For general installation steps using the ROSA CLI, see Creating a cluster with customizations using the CLI.

Prerequisites

  • You have verified that the proxy is accessible from the VPC that the cluster is being installed into. The proxy must also be accessible from the private subnets of the VPC.

Procedure

  • Specify a proxy configuration when you create your cluster:

    $ rosa create cluster \
 <other_arguments_here> \
 --additional-trust-bundle-file <path_to_ca_bundle_file> \   (1) (2) (3)
 --http-proxy http://<username>:<password>@<ip>:<port> \  (1) (3)
 --https-proxy https://<username>:<password>@<ip>:<port> \  (1) (3)
 --no-proxy example.com (4)
    1 The additional-trust-bundle-file, http-proxy, and https-proxy arguments are all optional.
    2 The additional-trust-bundle-file argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additional-trust-bundle-file argument is required for users who use a TLS-inspecting proxy unless the identity certificate for the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle. This applies regardless of whether the proxy is transparent or requires explicit configuration using the http-proxy and https-proxy arguments.
    3 The http-proxy and https-proxy arguments must point to a valid URL.
    4 A comma-separated list of destination domain names, IP addresses, or network CIDRs to exclude proxying.

    Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

    This field is ignored if neither the httpProxy or httpsProxy fields are set.
Additional resources

Configuring a proxy after installation

You can configure an HTTP or HTTPS proxy after you install a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC). You can configure the proxy after installation by using Red Hat OpenShift Cluster Manager or the ROSA CLI (rosa).

edit

Configuring a proxy after installation using OpenShift Cluster Manager

You can use Red Hat OpenShift Cluster Manager to add a cluster-wide proxy configuration to an existing Red Hat OpenShift Service on AWS cluster in a Virtual Private Cloud (VPC).

You can also use OpenShift Cluster Manager to update an existing cluster-wide proxy configuration. For example, you might need to update the network address for the proxy or replace the additional trust bundle if any of the certificate authorities for the proxy expire.

The cluster applies the proxy configuration to the control plane and compute nodes. While applying the configuration, each cluster node is temporarily placed in an unschedulable state and drained of its workloads. Each node is restarted as part of the process.
Prerequisites

  • You have an Red Hat OpenShift Service on AWS cluster .

  • Your cluster is deployed in a VPC.

Procedure

  1. Navigate to OpenShift Cluster Manager and select your cluster.

  2. Under the Virtual Private Cloud (VPC) section on the Networking page, click Edit cluster-wide proxy.

  3. On the Edit cluster-wide proxy page, provide your proxy configuration details:

    1. Enter a value in at least one of the following fields:

      • Specify a valid HTTP proxy URL.

      • Specify a valid HTTPS proxy URL.

      • In the Additional trust bundle field, provide a PEM encoded X.509 certificate bundle. If you are replacing an existing trust bundle file, select Replace file to view the field. The bundle is added to the trusted certificate store for the cluster nodes. An additional trust bundle file is required if you use a TLS-inspecting proxy unless the identity certificate for the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle. This requirement applies regardless of whether the proxy is transparent or requires explicit configuration using the http-proxy and https-proxy arguments.

    2. Click Confirm.

Verification

  • Under the Virtual Private Cloud (VPC) section on the Networking page, verify that the proxy configuration for your cluster is as expected.

edit

Configuring a proxy after installation using the CLI

You can use the Red Hat OpenShift Service on AWS (ROSA) CLI (rosa) to add a cluster-wide proxy configuration to an existing ROSA cluster in a Virtual Private Cloud (VPC).

You can also use rosa to update an existing cluster-wide proxy configuration. For example, you might need to update the network address for the proxy or replace the additional trust bundle if any of the certificate authorities for the proxy expire.

The cluster applies the proxy configuration to the control plane and compute nodes. While applying the configuration, each cluster node is temporarily placed in an unschedulable state and drained of its workloads. Each node is restarted as part of the process.
Prerequisites

  • You have installed and configured the latest ROSA (rosa) and OpenShift (oc) CLIs on your installation host.

  • You have a ROSA cluster that is deployed in a VPC.

Procedure

  • Edit the cluster configuration to add or update the cluster-wide proxy details:

    $ rosa edit cluster \
 --cluster $CLUSTER_NAME \
 --additional-trust-bundle-file <path_to_ca_bundle_file> \   (1) (2) (3)
 --http-proxy http://<username>:<password>@<ip>:<port> \  (1) (3)
 --https-proxy https://<username>:<password>@<ip>:<port> \  (1) (3)
  --no-proxy example.com (4)
    1 The additional-trust-bundle-file, http-proxy, and https-proxy arguments are all optional.
    2 The additional-trust-bundle-file argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additional-trust-bundle-file argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additional-trust-bundle-file argument is required for users who use a TLS-inspecting proxy unless the identity certificate for the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle. This applies regardless of whether the proxy is transparent or requires explicit configuration using the http-proxy and https-proxy arguments.

    You should not attempt to change the proxy or additional trust bundle configuration on the cluster directly. These changes must be applied by using the ROSA CLI (rosa) or Red Hat OpenShift Cluster Manager. Any changes that are made directly to the cluster will be reverted automatically.
    3 The http-proxy and https-proxy arguments must point to a valid URL.
    4 A comma-separated list of destination domain names, IP addresses, or network CIDRs to exclude proxying.

    Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

    This field is ignored if neither the httpProxy or httpsProxy fields are set.
Verification

  1. List the status of the machine config pools and verify that they are updated:

    $ oc get machineconfigpools
    Example output
    NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-d9a03f612a432095dcde6dcf44597d90   True      False      False      3              3                   3                     0                      31h
worker   rendered-worker-f6827a4efe21e155c25c21b43c46f65e   True      False      False      6              6                   6                     0                      31h

  2. Display the proxy configuration for your cluster and verify that the details are as expected:

    $ oc get proxy cluster -o yaml
    Example output
    apiVersion: config.openshift.io/v1
kind: Proxy
spec:
  httpProxy: http://proxy.host.domain:<port>
  httpsProxy: https://proxy.host.domain:<port>
  <...more...>
status:
  httpProxy: http://proxy.host.domain:<port>
  httpsProxy: https://proxy.host.domain:<port>
  <...more...>

Removing a cluster-wide proxy

You can remove your cluster-wide proxy by using the ROSA CLI. After removing the cluster, you should also remove any trust bundles that are added to the cluster.

edit

Removing the cluster-wide proxy using CLI

You must use the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa, to remove the proxy’s address from your cluster.

Prerequisites

  • You must have cluster administrator privileges.

  • You have installed the ROSA CLI (rosa).

Procedure

  • Use the rosa edit command to modify the proxy. You must pass empty strings to the --http-proxy and --https-proxy arguments to clear the proxy from the cluster:

    $ rosa edit cluster -c <cluster_name> --http-proxy "" --https-proxy ""

    While your proxy might only use one of the proxy arguments, the empty fields are ignored, so passing empty strings to both the --http-proxy and --https-proxy arguments do not cause any issues.
    Example Output
    I: Updated cluster <cluster_name>
Verification

  • You can verify that the proxy has been removed from the cluster by using the rosa describe command:

    $ rosa describe cluster -c <cluster_name>

    Before removal, the proxy IP displays in a proxy section:

    Name:                       <cluster_name>
ID:                         <cluster_internal_id>
External ID:                <cluster_external_id>
OpenShift Version:          .0
Channel Group:              stable
DNS:                        <dns>
AWS Account:                <aws_account_id>
API URL:                    <api_url>
Console URL:                <console_url>
Region:                     us-east-1
Multi-AZ:                   false
Nodes:
 - Control plane:           3
 - Infra:                   2
 - Compute:                 2
Network:
 - Type:                    OVNKubernetes
 - Service CIDR:            <service_cidr>
 - Machine CIDR:            <machine_cidr>
 - Pod CIDR:                <pod_cidr>
 - Host Prefix:             <host_prefix>
Proxy:
 - HTTPProxy:               <proxy_url>
Additional trust bundle:    REDACTED

    After removing the proxy, the proxy section is removed:

    Name:                       <cluster_name>
ID:                         <cluster_internal_id>
External ID:                <cluster_external_id>
OpenShift Version:          .0
Channel Group:              stable
DNS:                        <dns>
AWS Account:                <aws_account_id>
API URL:                    <api_url>
Console URL:                <console_url>
Region:                     us-east-1
Multi-AZ:                   false
Nodes:
 - Control plane:           3
 - Infra:                   2
 - Compute:                 2
Network:
 - Type:                    OVNKubernetes
 - Service CIDR:            <service_cidr>
 - Machine CIDR:            <machine_cidr>
 - Pod CIDR:                <pod_cidr>
 - Host Prefix:             <host_prefix>
Additional trust bundle:    REDACTED
edit

Removing certificate authorities on a Red Hat OpenShift Service on AWS cluster

You can remove certificate authorities (CA) from your cluster with the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa.

Prerequisites

  • You must have cluster administrator privileges.

  • You have installed the ROSA CLI (rosa).

  • Your cluster has certificate authorities added.

Procedure

  • Use the rosa edit command to modify the CA trust bundle. You must pass empty strings to the --additional-trust-bundle-file argument to clear the trust bundle from the cluster:

    $ rosa edit cluster -c <cluster_name> --additional-trust-bundle-file ""
    Example Output
    I: Updated cluster <cluster_name>
Verification

  • You can verify that the trust bundle has been removed from the cluster by using the rosa describe command:

    $ rosa describe cluster -c <cluster_name>

    Before removal, the Additional trust bundle section appears, redacting its value for security purposes:

    Name:                       <cluster_name>
ID:                         <cluster_internal_id>
External ID:                <cluster_external_id>
OpenShift Version:          .0
Channel Group:              stable
DNS:                        <dns>
AWS Account:                <aws_account_id>
API URL:                    <api_url>
Console URL:                <console_url>
Region:                     us-east-1
Multi-AZ:                   false
Nodes:
 - Control plane:           3
 - Infra:                   2
 - Compute:                 2
Network:
 - Type:                    OVNKubernetes
 - Service CIDR:            <service_cidr>
 - Machine CIDR:            <machine_cidr>
 - Pod CIDR:                <pod_cidr>
 - Host Prefix:             <host_prefix>
Proxy:
 - HTTPProxy:               <proxy_url>
Additional trust bundle:    REDACTED

    After removing the proxy, the Additional trust bundle section is removed:

    Name:                       <cluster_name>
ID:                         <cluster_internal_id>
External ID:                <cluster_external_id>
OpenShift Version:          .0
Channel Group:              stable
DNS:                        <dns>
AWS Account:                <aws_account_id>
API URL:                    <api_url>
Console URL:                <console_url>
Region:                     us-east-1
Multi-AZ:                   false
Nodes:
 - Control plane:           3
 - Infra:                   2
 - Compute:                 2
Network:
 - Type:                    OVNKubernetes
 - Service CIDR:            <service_cidr>
 - Machine CIDR:            <machine_cidr>
 - Pod CIDR:                <pod_cidr>
 - Host Prefix:             <host_prefix>
Proxy:
 - HTTPProxy:               <proxy_url>