Audit provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators, or other components of the system.

About the API Audit Log

Audit works at the API server level, logging all requests coming to the server. Each audit log contains two entries:

  1. The request line containing:

    1. A Unique ID allowing to match the response line (see #2)

    2. The source IP of the request

    3. The HTTP method being invoked

    4. The original user invoking the operation

    5. The impersonated user for the operation (self meaning himself)

    6. The impersonated group for the operation (lookup meaning user’s group)

    7. The namespace of the request or <none>

    8. The URI as requested

  2. The response line containing:

    1. The unique ID from #1

    2. The response code

You can view logs for the master nodes for the OpenShift Container Platform API server or the Kubernetes API server.

Example output for the Kubelet API server:

ip-10-0-140-97.ec2.internal {"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2019-04-09T19:56:58Z"},"level":"Metadata","timestamp":"2019-04-09T19:56:58Z","auditID":"6e96c88b-ab6f-44d2-b62e-d1413efd676b","stage":"ResponseComplete","requestURI":"/api/v1/nodes/audit-2019-04-09T14-07-27.129.log","verb":"get","user":{"username":"kube:admin","groups":["system:cluster-admins","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["10.0.57.93"],"userAgent":"oc/v1.14.6+c4799753c (linux/amd64) kubernetes/ba88cb2","objectRef":{"resource":"nodes","name":"audit-2019-04-09T14-07-27.129.log","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2019-04-09T19:56:58.982157Z","stageTimestamp":"2019-04-09T19:56:58.985300Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"system:cluster-admins\""}}

Configuring the API Audit Log level

You can configure the audit feature to set log level, retention policy, and the type of events to log.

Procedure
  1. Set the audit log level:

    1. Get the name of the API server Custom Resource (CR):

      $ oc get APIServer
      
      NAME      AGE
      cluster   18h
    2. Edit the API server CR:

      $ oc edit APIServer cluster
      apiVersion: config.openshift.io/v1
      kind: APIServer
      metadata:
        annotations:
          release.openshift.io/create-only: "true"
        creationTimestamp: 2019-05-09T18:56:37Z
        generation: 2
        name: cluster
        resourceVersion: "562502"
        selfLink: /apis/config.openshift.io/v1/apiservers/cluster
        uid: 2c4e980b-728c-11e9-967d-0a973adad40e
      spec:
        logLevel: "Normal" (1)
        operatorloglevel: "TraceAll" (2)
      1 Specify the log level for an overall component.
      2 Specify the log level for the API server Operator.

      You can set the log level for both settings to one of the following. The setting can be different for each setting:

      • Normal. Normal is the default. Normal working log information, including helpful notices for auditing or common operations. Similar to glog=2.

      • Debug. Debug is for troubleshooting problems. A greater quanitity of notices than Normal, but contain less information than Trace. Common operations might be logged. Similar to glog=4.

      • Trace. Trace is for troubleshooting problems when Debug is not verbose enough. Logs every function call as part of a common operation, including tracing execution of a query. Similar to glog=6.

      • TraceAll. TraceAll is troubleshoting at the level of API content/decoding. Contains complete body content. In production clusters, this setting causes performance degradation and results in a significant number of logs. Similar to ` glog=8`.

Viewing the API Audit Log

You can view the basic audit log.

Procedure

To view the basic audit log:

  1. View the OpenShift Container Platform API server logs

    1. If necessary, get the node IP and name of the log you want to view:

      $ oc --insecure-skip-tls-verify adm node-logs --role=master --path=openshift-apiserver/
      
      ip-10-0-140-97.ec2.internal audit-2019-04-09T00-12-19.834.log
      ip-10-0-140-97.ec2.internal audit-2019-04-09T11-13-00.469.log
      ip-10-0-140-97.ec2.internal audit.log
      ip-10-0-153-35.ec2.internal audit-2019-04-09T00-11-49.835.log
      ip-10-0-153-35.ec2.internal audit-2019-04-09T11-08-30.469.log
      ip-10-0-153-35.ec2.internal audit.log
      ip-10-0-170-165.ec2.internal audit-2019-04-09T00-13-00.128.log
      ip-10-0-170-165.ec2.internal audit-2019-04-09T11-10-04.082.log
      ip-10-0-170-165.ec2.internal audit.log
    2. View the OpenShift Container Platform API server log for a specific master node and timestamp or view all the logs for that master:

      $ oc adm node-logs <node-ip> <log-name> --path=openshift-apiserver/<log-name>

      For example:

      $ oc adm node-logs ip-10-0-140-97.ec2.internal audit-2019-04-08T13-09-01.227.log --path=openshift-apiserver/audit-2019-04-08T13-09-01.227.log
      $ oc adm node-logs ip-10-0-140-97.ec2.internal audit.log --path=openshift-apiserver/audit.log

      The output appears similar to the following:

      ip-10-0-140-97.ec2.internal {"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2019-04-09T18:52:03Z"},"level":"Metadata","timestamp":"2019-04-09T18:52:03Z","auditID":"9708b50d-8956-4c87-b9eb-a53ba054c13d","stage":"ResponseComplete","requestURI":"/","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["10.128.0.1"],"userAgent":"Go-http-client/2.0","responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2019-04-09T18:52:03.914638Z","stageTimestamp":"2019-04-09T18:52:03.915080Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-status-binding\" of ClusterRole \"cluster-status\" to Group \"system:unauthenticated\""}}
  2. View the Kubernetes API server logs:

    1. If necessary, get the node IP and name of the log you want to view:

      $ oc --insecure-skip-tls-verify adm node-logs --role=master --path=kube-apiserver/
      
      ip-10-0-140-97.ec2.internal audit-2019-04-09T14-07-27.129.log
      ip-10-0-140-97.ec2.internal audit-2019-04-09T19-18-32.542.log
      ip-10-0-140-97.ec2.internal audit.log
      ip-10-0-153-35.ec2.internal audit-2019-04-09T19-24-22.620.log
      ip-10-0-153-35.ec2.internal audit-2019-04-09T19-51-30.905.log
      ip-10-0-153-35.ec2.internal audit.log
      ip-10-0-170-165.ec2.internal audit-2019-04-09T18-37-07.511.log
      ip-10-0-170-165.ec2.internal audit-2019-04-09T19-21-14.371.log
      ip-10-0-170-165.ec2.internal audit.log
    2. View the Kubernetes API server log for a specific master node and timestamp or view all the logs for that master:

      $ oc adm node-logs <node-ip> <log-name> --path=kube-apiserver/<log-name>

      For example:

      $ oc adm node-logs ip-10-0-140-97.ec2.internal audit-2019-04-09T14-07-27.129.log --path=kube-apiserver/audit-2019-04-09T14-07-27.129.log
      $ oc adm node-logs ip-10-0-170-165.ec2.internal audit.log --path=kube-apiserver/audit.log

      The output appears similar to the following:

      ip-10-0-140-97.ec2.internal {"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2019-04-09T19:56:58Z"},"level":"Metadata","timestamp":"2019-04-09T19:56:58Z","auditID":"6e96c88b-ab6f-44d2-b62e-d1413efd676b","stage":"ResponseComplete","requestURI":"/api/v1/nodes/audit-2019-04-09T14-07-27.129.log","verb":"get","user":{"username":"kube:admin","groups":["system:cluster-admins","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["10.0.57.93"],"userAgent":"oc/v1.14.6+c4799753c (linux/amd64) kubernetes/ba88cb2","objectRef":{"resource":"nodes","name":"audit-2019-04-09T14-07-27.129.log","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2019-04-09T19:56:58.982157Z","stageTimestamp":"2019-04-09T19:56:58.985300Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"system:cluster-admins\""}}