Enabling JSON logging | Logging | OpenShift Container Platform 4.9

Parsing JSON logs
Configuring JSON log data for Elasticsearch
Forwarding JSON logs to the Elasticsearch log store

You can configure the Log Forwarding API to parse JSON strings into a structured object.

Parsing JSON logs

Logs including JSON logs are usually represented as a string inside the message field. That makes it hard for users to query specific fields inside a JSON document. OpenShift Logging’s Log Forwarding API enables you to parse JSON logs into a structured object and forward them to either OpenShift Logging-managed Elasticsearch or any other third-party system supported by the Log Forwarding API.

To illustrate how this works, suppose that you have the following structured JSON log entry.

Example structured JSON log entry

{"level":"info","name":"fred","home":"bedrock"}

Normally, the ClusterLogForwarder custom resource (CR) forwards that log entry in the message field. The message field contains the JSON-quoted string equivalent of the JSON log entry, as shown in the following example.

Example message field

{"message":"{\"level\":\"info\",\"name\":\"fred\",\"home\":\"bedrock\"",
 "more fields..."}

To enable parsing JSON log, you add parse: json to a pipeline in the ClusterLogForwarder CR, as shown in the following example.

Example snippet showing parse: json

pipelines:
- inputRefs: [ application ]
  outputRefs: myFluentd
  parse: json

When you enable parsing JSON logs by using parse: json, the CR copies the JSON-structured log entry in a structured field, as shown in the following example. This does not modify the original message field.

Example structured output containing the structured JSON log entry

{"structured": { "level": "info", "name": "fred", "home": "bedrock" },
 "more fields..."}

If the log entry does not contain valid structured JSON, the structured field will be absent.

To enable parsing JSON logs for specific logging platforms, see Forwarding logs to third-party systems.

Configuring JSON log data for Elasticsearch

If your JSON logs follow more than one schema, storing them in a single index might cause type conflicts and cardinality problems. To avoid that, you must configure the ClusterLogForwarder custom resource (CR) to group each schema into a single output definition. This way, each schema is forwarded to a separate index.

If you forward JSON logs to the default Elasticsearch instance managed by OpenShift Logging, it generates new indices based on your configuration. To avoid performance issues associated with having too many indices, consider keeping the number of possible schemas low by standardizing to common schemas.

Structure types

You can use the following structure types in the ClusterLogForwarder CR to construct index names for the Elasticsearch log store:

structuredTypeKey (string, optional) is the name of a message field. The value of that field, if present, is used to construct the index name.
- kubernetes.labels.<key> is the Kubernetes pod label whose value is used to construct the index name.
- openshift.labels.<key> is the pipeline.label.<key> element in the ClusterLogForwarder CR whose value is used to construct the index name.
- kubernetes.container_name uses the container name to construct the index name.
structuredTypeName: (string, optional) If structuredTypeKey is not set or its key is not present, OpenShift Logging uses the value of structuredTypeName as the structured type. When you use both structuredTypeKey and structuredTypeName together, structuredTypeName provides a fallback index name if the key in structuredTypeKey is missing from the JSON log data.

Although you can set the value of structuredTypeKey to any field shown in the "Log Record Fields" topic, the most useful fields are shown in the preceding list of structure types.

A structuredTypeKey: kubernetes.labels.<key> example

Suppose the following:

Your cluster is running application pods that produce JSON logs in two different formats, "apache" and "google".
The user labels these application pods with logFormat=apache and logFormat=google.
You use the following snippet in your ClusterLogForwarder CR YAML file.

outputDefaults:
 elasticsearch:
    structuredTypeKey: kubernetes.labels.logFormat (1)
    structuredTypeName: nologformat
pipelines:
- inputRefs: <application>
  outputRefs: default
  parse: json (2)

1	Uses the value of the key-value pair that is formed by the Kubernetes `logFormat` label.
2	Enables parsing JSON logs.

In that case, the following structured log record goes to the app-apache-write index:

{
  "structured":{"name":"fred","home":"bedrock"},
  "kubernetes":{"labels":{"logFormat": "apache", ...}}
}

And the following structured log record goes to the app-google-write index:

{
  "structured":{"name":"wilma","home":"bedrock"},
  "kubernetes":{"labels":{"logFormat": "google", ...}}
}

A structuredTypeKey: openshift.labels.<key> example

Suppose that you use the following snippet in your ClusterLogForwarder CR YAML file.

outputDefaults:
 elasticsearch:
    structuredTypeKey: openshift.labels.myLabel (1)
    structuredTypeName: nologformat
pipelines:
 - name: application-logs
   inputRefs:
   - application
   - audit
   outputRefs:
   - elasticsearch-secure
   - default
   parse: json
   labels:
     myLabel: myValue (2)

1	Uses the value of the key-value pair that is formed by the OpenShift `myLabel` label.
2	The `myLabel` element gives its string value, `myValue`, to the structured log record.

In that case, the following structured log record goes to the app-myValue-write index:

{
  "structured":{"name":"fred","home":"bedrock"},
  "openshift":{"labels":{"myLabel": "myValue", ...}}
}

Additional considerations

The Elasticsearch index for structured records is formed by prepending "app-" to the structured type and appending "-write".
Unstructured records are not sent to the structured index. They are indexed as usual in the application, infrastructure, or audit indices.
If there is no non-empty structured type, forward an unstructured record with no structured field.

It is important not to overload Elasticsearch with too many indices. Only use distinct structured types for distinct log formats, not for each application or namespace. For example, most Apache applications use the same JSON log format and structured type, such as LogApache.

Forwarding JSON logs to the Elasticsearch log store

For an Elasticsearch log store, if your JSON log entries follow different schemas, configure the ClusterLogForwarder custom resource (CR) to group each JSON schema into a single output definition. This way, Elasticsearch uses a separate index for each schema.

Because forwarding different schemas to the same index can cause type conflicts and cardinality problems, you must perform this configuration before you forward data to the Elasticsearch store.

To avoid performance issues associated with having too many indices, consider keeping the number of possible schemas low by standardizing to common schemas.

Procedure

Add the following snippet to your ClusterLogForwarder CR YAML file.

outputDefaults:
 elasticsearch:
    structuredTypeKey: <log record field>
    structuredTypeName: <name>
pipelines:
- inputRefs:
  - application
  outputRefs: default
  parse: json

Optional: Use structuredTypeKey to specify one of the log record fields, as described in the preceding topic, Configuring JSON log data for Elasticsearch. Otherwise, remove this line.
Optional: Use structuredTypeName to specify a <name>, as described in the preceding topic, Configuring JSON log data for Elasticsearch. Otherwise, remove this line.

To parse JSON logs, you must set either structuredTypeKey or structuredTypeName, or both structuredTypeKey and structuredTypeName.
For inputRefs, specify which log types to forward by using that pipeline, such as application, infrastructure, or audit.
Add the parse: json element to pipelines.
Create the CR object:
```
$ oc create -f <file-name>.yaml
```
The Red Hat OpenShift Logging Operator redeploys the Fluentd pods. However, if they do not redeploy, delete the Fluentd pods to force them to redeploy.
```
$ oc delete pod --selector logging-infra=collector
```

Additional resources

Forwarding logs to third-party systems