Validating Operators using the scorecard - Developing Operators | Operators

About the scorecard tool
Scorecard configuration
Tests performed
- Basic plug-in
- OLM plug-in
Running the scorecard
Running the scorecard with an OLM-managed Operator

Operator authors should validate that their Operator is packaged correctly and free of syntax errors. As an Operator author, you can use the Operator SDK scorecard tool to validate your Operator packaging and run tests.

OpenShift Container Platform 4.5 supports Operator SDK v0.17.2.

About the scorecard tool

To validate an Operator, the scorecard tool provided by the Operator SDK begins by creating all resources required by any related custom resources (CRs) and the Operator. The scorecard then creates a proxy container in the deployment of the Operator which is used to record calls to the API server and run some of the tests. The tests performed also examine some of the parameters in the CRs.

Scorecard configuration

The scorecard tool uses a configuration file that allows you to configure internal plug-ins, as well as several global configuration options.

Configuration file

The default location for the scorecard tool configuration is the <project_dir>/.osdk-scorecard.*. The following is an example of a YAML-formatted configuration file:

Scorecard configuration file

scorecard:
  output: json
  plugins:
    - basic: (1)
        cr-manifest:
          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
          - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml"
    - olm: (2)
        cr-manifest:
          - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml"
          - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml"
        csv-path: "deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml"

1	`basic` tests configured to test two custom resources (CRs).
2	`olm` tests configured to test two CRs.

Configuration methods for global options take the following priority, highest to lowest:

Command arguments (if available) → configuration file → default

The configuration file must be in YAML format. As the configuration file might be extended to allow configuration of all operator-sdk subcommands in the future, the scorecard configuration must be under a scorecard subsection.

Configuration file support is provided by the viper package. For more info on how viper configuration works, see the README.

Command arguments

While most of the scorecard tool configuration is done using a configuration file, you can also use the following arguments:

Table 1. Scorecard tool arguments
Flag	Type	Description
`--bundle`, `-b`	string	The path to a bundle directory used for the bundle validation test.
`--config`	string	The path to the scorecard configuration file. The default is `<project_dir>/.osdk-scorecard`. The file type and extension must be `.yaml`. If a configuration file is not provided or found at the default location, the scorecard exits with an error.
`--output`, `-o`	string	Output format. Valid options are `text` and `json`. The default format is `text`, which is designed to be a human readable format. The `json` format uses the JSON schema output format used for plug-ins defined later.
`--kubeconfig`, `-o`	string	The path to the `kubeconfig` file. It sets the `kubeconfig` for internal plug-ins.
`--version`	string	The version of scorecard to run. The default and only valid option is `v1alpha2`.
`--selector`, `-l`	string	The label selector to filter tests on.
`--list`, `-L`	bool	If `true`, only print the test names that would be run based on selector filtering.

Configuration file options

The scorecard configuration file provides the following options:

Table 2. Scorecard configuration file options
Option	Type	Description
`bundle`	string	Equivalent of the `--bundle` flag. Operator Lifecycle Manager (OLM) bundle directory path, when specified, runs bundle validation.
`output`	string	Equivalent of the `--output` flag. If this option is defined by both the configuration file and the flag, the flag value takes priority.
`kubeconfig`	string	Equivalent of the `--kubeconfig` flag. If this option is defined by both the configuration file and the flag, the flag value takes priority.
`plugins`	array	An array of plug-in names.

Basic and OLM plug-ins

The scorecard supports the internal basic and olm plug-ins, which are configured by a plugins section in the configuration file.

Table 3. Plug-in options
Option	Type	Description
`cr-manifest`	[]string	The path(s) for CRs being tested. Required if `olm-deployed` is unset or `false`.
`csv-path`	string	The path to the cluster service version (CSV) for the Operator. Required for OLM tests or if `olm-deployed` is set to `true`.
`olm-deployed`	bool	Indicates that the CSV and relevant CRDs have been deployed onto the cluster by OLM.
`kubeconfig`	string	The path to the `kubeconfig` file. If both the global `kubeconfig` and this field are set, this field is used for the plug-in.
`namespace`	string	The namespace to run the plug-ins in. If unset, the default specified by the `kubeconfig` file is used.
`init-timeout`	int	Time in seconds until a timeout during initialization of the Operator.
`crds-dir`	string	The path to the directory containing CRDs that must be deployed to the cluster.
`namespaced-manifest`	string	The manifest file with all resources that run within a namespace. By default, the scorecard combines the `service_account.yaml`, `role.yaml`, `role_binding.yaml`, and `operator.yaml` files from the `deploy` directory into a temporary manifest to use as the namespaced manifest.
`global-manifest`	string	The manifest containing required resources that run globally (not namespaced). By default, the scorecard combines all CRDs in the `crds-dir` directory into a temporary manifest to use as the global manifest.

Currently, using the scorecard with a CSV does not permit multiple CR manifests to be set through the CLI, configuration file, or CSV annotations. You must tear down your Operator in the cluster, re-deploy, and re-run the scorecard for each CR that is tested.

Additional resources

You can either set cr-manifest or your CSV metadata.annotations['alm-examples'] to provide CRs to the scorecard, but not both. See CRD templates for details.

Tests performed

By default, the scorecard tool has a set of internal tests it can run available across two internal plug-ins. If multiple CRs are specified for a plug-in, the test environment is fully cleaned up after each CR so that each CR gets a clean testing environment.

Each test has a short name that uniquely identifies the test. This is useful when selecting a specific test or tests to run. For example:

$ operator-sdk scorecard -o text --selector=test=checkspectest

$ operator-sdk scorecard -o text --selector='test in (checkspectest,checkstatustest)'

Basic plug-in

The following basic Operator tests are available from the basic plug-in:

Table 4. `basic` plug-in tests
Test	Description	Short name
Spec Block Exists	This test checks the custom resources (CRs) created in the cluster to make sure that all CRs have a `spec` block. This test has a maximum score of `1`.	`checkspectest`
Status Block Exists	This test checks the CRs created in the cluster to make sure that all CRs have a `status` block. This test has a maximum score of `1`.	`checkstatustest`
Writing Into CRs Has An Effect	This test reads the scorecard proxy logs to verify that the Operator is making `PUT` or `POST`, or both, requests to the API server, indicating that it is modifying resources. This test has a maximum score of `1`.	`writingintocrshaseffecttest`

OLM plug-in

The following Operator Lifecycle Manager (OLM) integration tests are available from the olm plug-in:

Table 5. `olm` plug-in tests
Test	Description	Short name
OLM Bundle Validation	This test validates the OLM bundle manifests found in the bundle directory as specified by the bundle flag. If the bundle contents contain errors, then the test result output includes the validator log as well as error messages from the validation library.	`bundlevalidationtest`
Provided APIs Have Validation	This test verifies that the CRDs for the provided CRs contain a validation section and that there is validation for each `spec` and `status` field detected in the CR. This test has a maximum score equal to the number of CRs provided by the `cr-manifest` option.	`crdshavevalidationtest`
Owned CRDs Have Resources Listed	This test makes sure that the CRDs for each CR provided by the `cr-manifest` option have a `resources` subsection in the `owned` CRDs section of the CSV. If the test detects used resources that are not listed in the `resources` section, it lists them in the suggestions at the end of the test. This test has a maximum score equal to the number of CRs provided by the `cr-manifest` option.	`crdshaveresourcestest`
Spec Fields With Descriptors	This test verifies that every field in the `spec` sections of custom resources have a corresponding descriptor listed in the CSV. This test has a maximum score equal to the total number of fields in the `spec` sections of each custom resource passed in by the `cr-manifest` option.	`specdescriptorstest`
Status Fields With Descriptors	This test verifies that every field in the `status` sections of custom resources have a corresponding descriptor listed in the CSV. This test has a maximum score equal to the total number of fields in the `status` sections of each custom resource passed in by the `cr-manifest` option.	`statusdescriptorstest`

Additional resources

Owned CRDs

Running the scorecard

Prerequisites

The following prerequisites for the Operator project are checked by the scorecard tool:

Access to a cluster running Kubernetes 1.11.3 or later.
If you want to use the scorecard to check the integration of your Operator project with Operator Lifecycle Manager (OLM), then a cluster service version (CSV) file is also required. This is a requirement when the olm-deployed option is used.
For Operators that were not generated using the Operator SDK (non-SDK Operators):
- Resource manifests for installing and configuring the Operator and custom resources (CRs).
- Configuration getter that supports reading from the KUBECONFIG environment variable, such as the clientcmd or controller-runtime configuration getters. This is required for the scorecard proxy to work correctly.

Procedure

Define a .osdk-scorecard.yaml configuration file in your Operator project.
Create the namespace defined in the RBAC files (role_binding).
Run the scorecard from the root directory of your Operator project:
```
$ operator-sdk scorecard
```
The scorecard return code is 1 if any of the executed texts did not pass and 0 if all selected tests passed.

Running the scorecard with an OLM-managed Operator

The scorecard can be run using a cluster service version (CSV), providing a way to test cluster-ready and non-Operator SDK Operators.

Procedure

The scorecard requires a proxy container in the deployment pod of the Operator to read Operator logs. A few modifications to your CSV and creation of one extra object are required to run the proxy before deploying your Operator with Operator Lifecycle Manager (OLM).

This step can be performed manually or automated using bash functions. Choose one of the following methods.

Manual method:

Create a proxy server secret containing a local kubeconfig file`.

Generate a user name using the namespaced owner reference of the scorecard proxy.
```
$ echo '{"apiVersion":"","kind":"","name":"scorecard","uid":"","Namespace":"'<namespace>'"}' | base64 -w 0 (1)
```
1 Replace <namespace> with the namespace your Operator will deploy in.

Write a Config manifest scorecard-config.yaml using the following template, replacing <username> with the base64 user name generated in the previous step:

apiVersion: v1
kind: Config
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: http://<username>@localhost:8889
  name: proxy-server
contexts:
- context:
    cluster: proxy-server
    user: admin/proxy-server
  name: <namespace>/proxy-server
current-context: <namespace>/proxy-server
preferences: {}
users:
- name: admin/proxy-server
  user:
    username: <username>
    password: unused

Encode the Config as base64:

$ cat scorecard-config.yaml | base64 -w 0

Create a Secret manifest scorecard-secret.yaml:
```
apiVersion: v1
kind: Secret
metadata:
  name: scorecard-kubeconfig
  namespace: <namespace> (1)
data:
  kubeconfig: <kubeconfig_base64> (2)
```
1 Replace <namespace> with the namespace your Operator will deploy in.

2 Replace <kubeconfig_base64> with the Config encoded as base64.
Apply the secret:
```
$ oc apply -f scorecard-secret.yaml
```

Insert a volume referring to the secret into the deployment for the Operator:

spec:
  install:
    spec:
      deployments:
      - name: memcached-operator
        spec:
          ...
          template:
            ...
            spec:
              containers:
              ...
              volumes:
              - name: scorecard-kubeconfig (1)
                secret:
                  secretName: scorecard-kubeconfig
                  items:
                  - key: kubeconfig
                    path: config

1	Scorecard `kubeconfig` volume.

Insert a volume mount and KUBECONFIG environment variable into each container in the deployment of your Operator:

spec:
  install:
    spec:
      deployments:
      - name: memcached-operator
        spec:
          ...
          template:
            ...
            spec:
              containers:
              - name: container1
                ...
                volumeMounts:
                - name: scorecard-kubeconfig (1)
                  mountPath: /scorecard-secret
                env:
                - name: KUBECONFIG (2)
                  value: /scorecard-secret/config
              - name: container2 (3)
                ...

1	Scorecard `kubeconfig` volume mount.
2	Scorecard `kubeconfig` environment variable.
3	Repeat the same for this and all other containers.

Insert the scorecard proxy container into the deployment of your Operator:

spec:
  install:
    spec:
      deployments:
      - name: memcached-operator
        spec:
          ...
          template:
            ...
            spec:
              containers:
              ...
              - name: scorecard-proxy (1)
                command:
                - scorecard-proxy
                env:
                - name: WATCH_NAMESPACE
                  valueFrom:
                    fieldRef:
                      apiVersion: v1
                      fieldPath: metadata.namespace
                image: quay.io/operator-framework/scorecard-proxy:master
                imagePullPolicy: Always
                ports:
                - name: proxy
                  containerPort: 8889

1	Scorecard proxy container.

Automated method:

The community-operators repository has several bash functions that can perform the previous steps in the procedure for you.

Run the following curl command:

$ curl -Lo csv-manifest-modifiers.sh \
    https://raw.githubusercontent.com/operator-framework/community-operators/master/scripts/lib/file

Source the csv-manifest-modifiers.sh file:
```
$ . ./csv-manifest-modifiers.sh
```
Create the kubeconfig secret file:
```
$ create_kubeconfig_secret_file scorecard-secret.yaml "<namespace>" (1)
```
1 Replace <namespace> with the namespace your Operator will deploy in.
Apply the secret:
```
$ oc apply -f scorecard-secret.yaml
```
Insert the kubeconfig volume:
```
$ insert_kubeconfig_volume "<csv_file>" (1)
```
1 Replace <csv_file> with the path to your CSV manifest.

Insert the kubeconfig secret mount:

$ insert_kubeconfig_secret_mount "<csv_file>"

Insert the proxy container:

$ insert_proxy_container "<csv_file>" "quay.io/operator-framework/scorecard-proxy:master"

After inserting the proxy container, follow the steps in the Getting started with the Operator SDK guide to bundle your CSV and custom resource definitions (CRDs) and deploy your Operator on OLM.
After your Operator has been deployed on OLM, define a .osdk-scorecard.yaml configuration file in your Operator project and ensure both the csv-path: <csv_manifest_path> and olm-deployed options are set.
Run the scorecard with both the csv-path: <csv_manifest_path> and olm-deployed options set in your scorecard configuration file:
```
$ operator-sdk scorecard
```

Additional resources

Managing a Go-based Operator using Operator Lifecycle Manager

1	Replace `<namespace>` with the namespace your Operator will deploy in.
2	Replace `<kubeconfig_base64>` with the `Config` encoded as base64.