Before you begin

Configuring authorization for Google Cloud Platform

Roles

Configuring GCP for OpenShift Container Platform requires the following GCP role:

roles/owner

Needed for creating service accounts, cloud storage, instances, images, templates, Cloud DNS entries, and to deploy load balancers and health checks.

delete permissions might also be required if the user is expected to redeploy the environment during testing phases.

You can also create a service account to avoid using personal users when deploying GCP objects.

See the Understanding roles section of the GCP documentation for more information, including steps for how to configure roles.

Scopes and service accounts

GCP uses scopes to determine if an authenticated identity is authorized to perform operations within a resource. For example, if application A with a read-only scope access token can only read, while application B with a read-write scope access token can read and modify data.

The scopes are defined at the GCP API level as https://www.googleapis.com/auth/compute.readonly.

You can specify scopes using the --scopes=[SCOPE,…​] option when creating instances, or you can use the --no-scopes option to create the instance without scopes if you don’t want the instance accessing the GCP API.

All GCP projects include a default [PROJECT_NUMBER]-compute@developer.gserviceaccount.com service account with project editor permissions.

By default, a newly created instance is automatically enabled to run as the default service account with the following access scopes:

You can specify another service account with the --service-account=SERVICE_ACCOUNT option when creating the instance, or explicitly disabling service accounts for the instance using the --no-service-account option using the gcloud CLI.

Google Compute Engine objects

Integrating OpenShift Container Platform with Google Compute Engine (GCE) requires the following components or services.

A GCP project

A GCP project is the base level organizing entity that forms the basis for creating, enabling, and using all GCP services. This includes managing APIs, enabling billing, adding and removing collaborators, and managing permissions.

Project IDs are unique identifiers, and project IDs must be unique across all of Google Cloud Engine. This means you cannot use myproject as a project ID if someone else has created a project with that ID before.

Billing

You cannot create new resources unless billing is attached to an account. The new project can be linked to an existing project or new information can be entered.

Cloud identity and access management

Deploying OpenShift Container Platform requires the proper permissions. A user must be able to create service accounts, cloud storage, instances, images, templates, Cloud DNS entries, and deploy load balancers and health checks. Delete permissions are also helpful in order to be able to redeploy the environment while testing.

You can create service accounts with specific permissions, then use them to deploy infrastructure components instead of regular users. You can also create roles to limit access to different users or service accounts.

GCP instances use service accounts to allow applications to call GCP APIs. For example, OpenShift Container Platform node hosts can call the GCP disk API to provide a persistent volume to an application.

Access control to the various infrastructure, service resources, and fine-grained roles are available using the IAM service. For more information, see the Access cloud overview section of the GCP documentation.

SSH keys

GCP injects SSH public keys as authorized keys so you can log in using SSH in the created instances. You can configure the SSH keys per instance or per project.

You can use existing SSH keys. GCP metadata can help with storing the SSH keys that are injected at boot time in the instances to allow SSH access.

GCP regions and zones

GCP has a global infrastructure that covers regions and availability zones. While deploying OpenShift Container Platform in GCP on different zones can help avoid single-point-of-failures, there are some caveats regarding storage.

GCP disks are created within a zone. Therefore, if a OpenShift Container Platform node host goes down in zone "A" and the pods move to zone "B", the persistent storage cannot be attached to those pods because the disks are in a different zone.

Deploying a single zone of multizone OpenShift Container Platform environment is an important decision to make before installing OpenShift Container Platform. If deploying a multizone environment, the recommended setup is to use three different zones in a single region.

External IP address

So that GCP instances can communicate with the Internet, you must attach an external IP address to the instance. Also, an external IP address is required to communicate with instances deployed in GCP from outside the Virtual Private Cloud (VPC) Network.

Requiring an External IP address for internet access is a limitation of the provider. You can configure firewall rules to block incoming external traffic in instances if not needed.

Cloud DNS

GCP cloud DNS is a DNS service used to publish domain names to the global DNS using GCP DNS servers.

The public cloud DNS zone requires a domain name that you purchased either through Google’s "Domains" service or through a third-party provider. When you create the zone, you must add the name servers provided by Google to the registrar.

See the GCP documentation on Cloud DNS for more information.

GCP VPC networks have an internal DNS service that automatically resolves internal host names.

The internal fully qualified domain name (FQDN) for an instance follows the [HOST_NAME].c.[PROJECT_ID].internal format.

See the GCP documentation on Internal DNS for more information.

Load balancing

The GCP load balancing service enables the distribution of traffic across multiple instances in the GCP cloud.

There are five types of Load Balancing:

HTTPS and TCP proxy load balancing are the only options for using HTTPS health checks for master nodes, which checks the status of /healthz.

Because HTTPS load balancing requires a custom certificate, this implementation uses TCP Proxy load balancing to simplify the process.

See the GCP documentation on Load balancing for more information.

Instances sizes

A successful OpenShift Container Platform environment requires some minimum hardware requirements:

Table 1. Instances sizes
Role Size

Master

n1-standard-8

Node

n1-standard-4

GCP allows you to create custom instance sizes to fit different requirements. See Creating an Instance with a Custom Machine Type for more information, or see Machine types and OpenShift Container Platform Minimum Hardware Requirements for more information about instance sizes.

Storage Options

By default, each GCP instance has a small root persistent disk that contains the operating system. When applications running on the instance require more storage space, you can add additional storage options to the instance:

  • Standard persistent disks

  • SSD persistent disks

  • Local SSDs

  • Cloud storage buckets

For more information, see the GCP documentation on Storage options.

Configuring OpenShift Container Platform for GCE

You can configure OpenShift Container Platform for GCE in two ways:

Option 1: Configuring OpenShift Container Platform for GCP using Ansible

You can configure OpenShift Container Platform for Google Compute Platform (GCP) by modifying the Ansible inventory file at installation time or after installation.

Procedure
  1. At minimum, you must define the openshift_cloudprovider_kind, openshift_gcp_project and openshift_gcp_prefix parameters, as well as the optional openshift_gcp_multizone for multizone deployments and openshift_gcp_network_name if you are not using the default network name.

    Add the following section to the Ansible inventory file at installation to configure your OpenShift Container Platform environment for GCP:

    [OSEv3:vars]
    openshift_cloudprovider_kind=gce
    openshift_gcp_project=<projectid> (1)
    openshift_gcp_prefix=<uid> (2)
    openshift_gcp_multizone=False (3)
    openshift_gcp_network_name=<network name> (4)
    1 Provide the GCP project ID where the existing instances are running. This ID is generated when you create the project in the Google Cloud Platform Console.
    2 Provide a unique string to identify each OpenShift Container Platform cluster. This must be unique across GCP.
    3 Optionally, set to True to trigger multizone deployments on GCP. Set to False by default.
    4 Optionally, provide the network name if not using default network.

    Installing with Ansible also creates and configures the following files to fit your GCP environment:

    • /etc/origin/cloudprovider/gce.conf

    • /etc/origin/master/master-config.yaml

    • /etc/origin/node/node-config.yaml

  2. If you are running load balancer services using GCP, the Compute Engine VM node instances require the ocp suffix. For example, if the value of the openshift_gcp_prefix parameter is set to mycluster, you must tag the nodes with myclusterocp. See Adding and Removing Network Tags for more information on how to add network tags to Compute Engine VM instances.

  3. Optionally, you can configure multizone support.

    The cluster installation process configures single-zone support by default, but you can configure for multiple zones to avoid single-point-of-failures.

    Because GCP disks are created within a zone, deploying OpenShift Container Platform in GCP on different zones can cause problems with storage. If an OpenShift Container Platform node host goes down in zone "A" and the pods move to zone "B", the persistent storage cannot be attached to those pods because the disks are now in a different zone. See Multiple zone limitations in the Kubernetes documentation for more information.

    To enable multizone support using the Ansible inventory file, add the following parameter:

    [OSEv3:vars]
    openshift_gcp_multizone=true

    To return to single-zone support, set the openshift_gcp_multizone value to false and rerun the Ansible inventory file.

Option 2: Manually configuring OpenShift Container Platform for GCE

Manually configuring master hosts for GCE

Perform the following procedure on all master hosts.

Procedure
  1. Add the GCE parameters to the apiServerArguments and controllerArguments sections of the master configuration file at /etc/origin/master/master-config.yaml by default:

    apiServerArguments:
      cloud-provider:
        - "gce"
      cloud-config:
        - "/etc/origin/cloudprovider/gce.conf"*
    controllerArguments:
      cloud-provider:
        - "gce"
      cloud-config:
        - "/etc/origin/cloudprovider/gce.conf"*
  2. When you configure OpenShift Container Platform for GCP using Ansible, the /etc/origin/cloudprovider/gce.conf file is created automatically. Because you are manually configuring OpenShift Container Platform for GCP, you must create the file and enter the following:

    [Global]
    project-id = <project-id> (1)
    network-name = <network-name> (2)
    node-tags = <node-tags> (3)
    node-instance-prefix = <instance-prefix> (4)
    multizone = true (5)
    1 Provide the GCP project ID where the existing instances are running.
    2 Provide the network name if not using the default.
    3 Provide the tag for the GCP nodes. Must contain ocp as a suffix. For example, if the value of the node-instance-prefix parameter is set to mycluster, the nodes must be tagged with myclusterocp.
    4 Provide a unique string to identify your OpenShift Container Platform cluster.
    5 Set to true to trigger multizone deployments on GCP. Set to False by default.

    The cluster installation process configures single-zone support by default.

    Deploying OpenShift Container Platform in GCP on different zones can be helpful to avoid single-point-of-failures, but can cause problems with storage. This is because GCP disks are created within a zone. If an OpenShift Container Platform node host goes down in zone "A" and the pods should be moved to zone "B", the persistent storage cannot be attached to those pods, because the disks are now in a different zone. See Multiple zone limitations in the Kubernetes documentation for more information.

    For running load balancer services using GCP, the Compute Engine VM node instances require the ocp suffix: <openshift_gcp_prefix>ocp. For example, if the value of the openshift_gcp_prefix parameter is set to mycluster, you must tag the nodes with myclusterocp. See Adding and Removing Network Tags for more information on how to add network tags to Compute Engine VM instances.

  3. Restart the OpenShift Container Platform host services:

    # master-restart api
    # master-restart controllers
    # systemctl restart atomic-openshift-node

To return to single-zone support, set the multizone value to false and restart the master and node host services.

Manually configuring node hosts for GCE

Perform the following on all node hosts.

Procedure
  1. Edit the appropriate node configuration map and update the contents of the kubeletArguments section:

    kubeletArguments:
      *cloud-provider:
        - "gce"
      cloud-config:
        - "/etc/origin/cloudprovider/gce.conf"*

    The nodeName must match the instance name in GCP in order for the cloud provider integration to work properly. The name must also be RFC1123 compliant.

  2. Restart the OpenShift Container Platform services on all nodes.

    # systemctl restart atomic-openshift-node

Configuring the OpenShift Container Platform registry for GCP

Google Cloud Platform (GCP) provides object cloud storage that OpenShift Container Platform can use to store container images using the OpenShift Container Platform container registry.

For more information, see Cloud Storage in the GCP documentation.

Prerequisites

You must create the bucket to host the registry images before the installation. The following commands create a regional bucket using the configured service account:

gsutil mb -c regional -l <region> gs://ocp-registry-bucket
cat <<EOF > labels.json
{
  "ocp-cluster": "mycluster"
}
EOF
gsutil label set labels.json gs://ocp-registry-bucket
rm -f labels.json

A bucket’s data is automatically encrypted using a Google-managed key by default. To specify a different key to encrypt the data, see the Data Encryption Options available in GCP.

See the Creating storage buckets documentation for more information.

Procedure

To configure the Ansible inventory file for the registry to use a Google Cloud Storage (GCS) bucket:

[OSEv3:vars]
# GCP Provider Configuration
openshift_hosted_registry_storage_provider=gcs
openshift_hosted_registry_storage_kind=object
openshift_hosted_registry_replicas=1 (1)
openshift_hosted_registry_storage_gcs_bucket=<bucket_name> (2)
openshift_hosted_registry_storage_gcs_keyfile=<bucket_keyfile> (3)
openshift_hosted_registry_storage_gcs_rootdirectory=<registry_directory> (4)
1 The number of replicas to configure.
2 The bucket name to for registry storage.
3 The path on the installer host where the bucket’s keyfile is located if you use a custom key file to encrypt the data.
4 Directory used to store the data. /registry by default

For more information, see Cloud Storage in the GCP documentation.

Manually configuring OpenShift Container Platform registry for GCP

To use GCP object storage, edit the registry’s configuration file and mount to the registry pod.

See the Google Cloud Storage Driver documentation for more information about storage driver configuration files.

Procedure
  1. Export the current /etc/registry/config.yml file:

    $ oc get secret registry-config \
        -o jsonpath='{.data.config\.yml}' -n default | base64 -d \
      >> config.yml.old
  2. Create a new configuration file from the old /etc/registry/config.yml file:

    $ cp config.yml.old config.yml
  3. Edit the file to include the GCP parameters. Specify the bucket and keyfile in the storage section of a registry’s configuration file:

    storage:
      delete:
        enabled: true
      cache:
        blobdescriptor: inmemory
      gcs:
        bucket: ocp-registry (1)
        keyfile: mykeyfile (2)
    1 Replace with the GCP bucket name.
    2 A private service account key file in JSON format. If using the Google Application Default Credentials, do not specify the keyfile parameter.
  4. Delete the registry-config secret:

    $ oc delete secret registry-config -n default
  5. Recreate the secret to reference the updated configuration file:

    $ oc create secret generic registry-config \
        --from-file=config.yml -n default
  6. Redeploy the registry to read the updated configuration:

    $ oc rollout latest docker-registry -n default
Verify the registry is using GCP object storage

To verify if the registry is using GCP bucket storage:

Procedure
  1. After a successful registry deployment using GCP storage, the registry deploymentconfig does not show any information if the registry is using an emptydir instead of GCP bucket storage:

    $ oc describe dc docker-registry -n default
    ...
    Mounts:
      ...
      /registry from registry-storage (rw)
    Volumes:
    registry-storage:
    Type:       EmptyDir (1)
    ...
    1 The temporary directory that shares a pod’s lifetime.
  2. Check if the /registry mountpoint is empty. This is the volume GCP storage will use:

    $ oc exec \
        $(oc get pod -l deploymentconfig=docker-registry \
        -o=jsonpath='{.items[0].metadata.name}')  -i -t -- ls -l /registry
    total 0
  3. If it is empty, it is because the GCP bucket configuration is performed in the registry-config secret:

    $ oc describe secret registry-config
    Name:         registry-config
    Namespace:    default
    Labels:       <none>
    Annotations:  <none>
    
    Type:  Opaque
    
    Data
    ====
    config.yml:  398 bytes
  4. The installer creates a config.yml file with the desired configuration using the extended registry capabilities as seen in Storage in the installation documentation. To view the configuration file, including the storage section where the storage bucket configuration is stored:

    $ oc exec \
        $(oc get pod -l deploymentconfig=docker-registry \
          -o=jsonpath='{.items[0].metadata.name}') \
      cat /etc/registry/config.yml
    
    version: 0.1
    log:
      level: debug
    http:
      addr: :5000
    storage:
      delete:
        enabled: true
      cache:
        blobdescriptor: inmemory
      gcs:
        bucket: ocp-registry
    auth:
      openshift:
        realm: openshift
    middleware:
      registry:
      - name: openshift
      repository:
      - name: openshift
        options:
          pullthrough: True
          acceptschema2: True
          enforcequota: False
      storage:
      - name: openshift

    Or you can view the secret:

    $ oc get secret registry-config -o jsonpath='{.data.config\.yml}' | base64 -d
    version: 0.1
    log:
      level: debug
    http:
      addr: :5000
    storage:
      delete:
        enabled: true
      cache:
        blobdescriptor: inmemory
      gcs:
        bucket: ocp-registry
    auth:
      openshift:
        realm: openshift
    middleware:
      registry:
      - name: openshift
      repository:
      - name: openshift
        options:
          pullthrough: True
          acceptschema2: True
          enforcequota: False
      storage:
      - name: openshift

    You can verify that any image push was successful by viewing Storage in the GCP console, then clicking Browser and selecting the bucket, or by running the gsutil command:

    $ gsutil ls gs://ocp-registry/
    gs://ocp-registry/docker/
    
    $ gsutil du gs://ocp-registry/
    7660385     gs://ocp-registry/docker/registry/v2/blobs/sha256/03/033565e6892e5cc6dd03187d00a4575720a928db111274e0fbf31b410a093c10/data
    7660385     gs://ocp-registry/docker/registry/v2/blobs/sha256/03/033565e6892e5cc6dd03187d00a4575720a928db111274e0fbf31b410a093c10/
    7660385     gs://ocp-registry/docker/registry/v2/blobs/sha256/03/
    ...

If using an emptyDir volume, the /registry mountpoint looks similar to the following:

$ oc exec \
    $(oc get pod -l deploymentconfig=docker-registry \
    -o=jsonpath='{.items[0].metadata.name}')  -i -t -- df -h /registry
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc         30G  226M   30G   1% /registry


$ oc exec \
    $(oc get pod -l deploymentconfig=docker-registry \
    -o=jsonpath='{.items[0].metadata.name}')  -i -t -- ls -l /registry
total 0
drwxr-sr-x. 3 1000000000 1000000000 22 Jun 19 12:24 docker

Configuring OpenShift Container Platform to use GCP storage

OpenShift Container Platform can use GCP storage using persistent volumes mechanisms. OpenShift Container Platform creates the disk in GCP and attaches the disk to the correct instance.

GCP disks are ReadWriteOnce access mode, which means the volume can be mounted as read-write by a single node. See the Access modes section of the Architecture guide for more information.

Procedure
  1. OpenShift Container Platform creates the following storageclass when you use the gce-pd provisioner and if you use the openshift_cloudprovider_kind=gce and openshift_gcp_* variables in the Ansible inventory. Otherwise, if you configured OpenShift Container Platform without using Ansible and the storageclass has not been created at installation time, you can create it manually:

    $ oc get --export storageclass standard -o yaml
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
     annotations:
       storageclass.beta.kubernetes.io/is-default-class: "true"
     creationTimestamp: null
     name: standard
     selfLink: /apis/storage.k8s.io/v1/storageclasses/standard
    parameters:
     type: pd-standard
    provisioner: kubernetes.io/gce-pd
    reclaimPolicy: Delete

    After you request a PV and using the storageclass shown in the previous step, OpenShift Container Platform creates disks in the GCP infrastructure. To verify that the disks were created:

    $ gcloud compute disks list | grep kubernetes
    kubernetes-dynamic-pvc-10ded514-7625-11e8-8c52-42010af00003  us-west1-b  10       pd-standard  READY

Using the GCP external load balancer as a service

You can configure OpenShift Container Platform to use the GCP load balancer by exposing services externally using a LoadBalancer service. OpenShift Container Platform creates the load balancer in GCP and creates the necessary firewall rules.

Procedure
  1. Create a new application:

    $ oc new-app openshift/hello-openshift
  2. Expose the load balancer service:

    $ oc expose dc hello-openshift --name='hello-openshift-external' --type='LoadBalancer'

    This command creates a LoadBalancer service similar to the following example:

    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: hello-openshift
      name: hello-openshift-external
    spec:
      externalTrafficPolicy: Cluster
      ports:
      - name: port-1
        nodePort: 30714
        port: 8080
        protocol: TCP
        targetPort: 8080
      - name: port-2
        nodePort: 30122
        port: 8888
        protocol: TCP
        targetPort: 8888
      selector:
        app: hello-openshift
        deploymentconfig: hello-openshift
      sessionAffinity: None
      type: LoadBalancer
  3. To verify that the service has been created:

    $ oc get svc
    NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                         AGE
    hello-openshift            ClusterIP      172.30.62.10     <none>          8080/TCP,8888/TCP               20m
    hello-openshift-external   LoadBalancer   172.30.147.214   35.230.97.224   8080:31521/TCP,8888:30843/TCP   19m

    The LoadBalancer type and External IP values indicate that the service is using GCP load balancers to expose the application.

OpenShift Container Platform creates the required objects in the GCP infrastructure such as:

  • Firewall rules:

    $ gcloud compute firewall-rules list | grep k8s
    k8s-4612931a3a47c204-node-http-hc        my-net  INGRESS    1000      tcp:10256
    k8s-fw-a1a8afaa7762811e88c5242010af0000  my-net  INGRESS    1000      tcp:8080,tcp:8888

    These firewall rules are applied to instances tagged with <openshift_gcp_prefix>ocp. For example, if the value of the openshift_gcp_prefix parameter is set to mycluster, you must tag the nodes with myclusterocp. See Adding and Removing Network Tags for more information on how to add network tags to Compute Engine VM instances.

  • Health checks:

    $ gcloud compute http-health-checks list | grep k8s
    k8s-4612931a3a47c204-node        10256  /healthz
  • A load balancer:

    $ gcloud compute target-pools list | grep k8s
    a1a8afaa7762811e88c5242010af0000  us-west1  NONE                      k8s-4612931a3a47c204-node
    $ gcloud compute forwarding-rules list | grep a1a8afaa7762811e88c5242010af0000
    a1a8afaa7762811e88c5242010af0000  us-west1  35.230.97.224  TCP          us-west1/targetPools/a1a8afaa7762811e88c5242010af0000

To verify that the load balancer is properly configured, run the following command from an external host:

$ curl 35.230.97.224:8080
Hello OpenShift!