SupportConsoleDevelopersStart a trialContact

Chapter 2. Forward logs over HTTP

You can forward logs to generic HTTP endpoints by using the HTTP output type, which supports custom authentication, headers, and batch configuration.

2.1. Forward logs over HTTP
Copy link

To enable forwarding logs over HTTP, specify http as the output type in the ClusterLogForwarder custom resource (CR).

Procedure

  • Create or edit the ClusterLogForwarder CR using the template below:

    Example ClusterLogForwarder CR

    apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name>
  namespace: <log_forwarder_namespace>
spec:
  managementState: Managed
  outputs:
  - name: <output_name>
    type: http
    http:
      headers:
          h1: v1
          h2: v2
      authentication:
        username:
          key: username
          secretName: <http_auth_secret>
        password:
          key: password
          secretName: <http_auth_secret>
      timeout: 300
      proxyURL: <proxy_url>
      url: <url>
    tls:
      insecureSkipVerify:
      ca:
        key: <ca_certificate>
        secretName: <secret_name>
  pipelines:
    - inputRefs:
        - application
      name: pipe1
      outputRefs:
        - <output_name>
  serviceAccount:
    name: <service_account_name>

    headers
    Optional: Additional headers to send with each log record.
    authentication.username
    Optional: Username for HTTP Basic authentication. Specify the secret name and key containing the username.
    authentication.password
    Optional: Password for HTTP Basic authentication. Specify the secret name and key containing the password.
    timeout
    Optional: Timeout in seconds for HTTP requests. Default is 300 seconds (5 minutes).
    proxyURL
    Optional: URL of the HTTP/HTTPS proxy to forward logs over http or https from this output. This setting overrides any default proxy settings for the cluster or the node.
    url
    Destination address for logs.
    tls.insecureSkipVerify

    Optional: When set to true, the collector skips verification of the server’s certificate. When set to false (the default), the collector verifies the server’s certificate against the CA specified in tls.ca. If insecureSkipVerify is set to true, the tls.ca field is not required.

    Warning

    Setting insecureSkipVerify to true is not recommended for production environments because it disables certificate validation and makes the connection vulnerable to man-in-the-middle attacks. Use this setting only for testing purposes or when connecting to trusted internal endpoints.

    tls.ca.secretName
    Secret name containing the certificate authority (CA) certificate for validating the server’s TLS certificate. Required when insecureSkipVerify is false or not set.
    tls.ca.key
    The key name in the secret that contains the CA certificate.
    <output_name>
    This value should be the same as the output name.
    serviceAccount.name

    The name of your service account.

    Note

    The tls configuration shown in this example applies to HTTP outputs, but similar TLS settings are available for other output types that support TLS encryption, including Kafka, Syslog, Elasticsearch, and Splunk. For information about configuring TLS for other output types, run the command oc explain clusterlogforwarders.spec.outputs.<output_type>.tls.

2.2. HTTP output format and payload structure
Copy link

When you forward logs to an HTTP endpoint, the log collector sends OpenShift log records as JSON payloads in POST requests. Understanding this format helps you configure receiving HTTP services and implement custom log ingestion endpoints.

The log collector sends logs to HTTP endpoints by using JSON-formatted POST requests. Each request can contain one or more log records, depending on the batching configuration.

2.2.1. Request structure
Copy link

HTTP requests from the log collector have the following structure:

Table 2.1. HTTP headers
HeaderValue

Content-Type

application/json

User-Agent

Vector/version (the collector implementation)

Custom headers

Optional - configurable in the ClusterLogForwarder output specification

The log collector supports two JSON payload formats:

Single record per request
Each HTTP POST request contains one log record as a JSON object.
Batched records per request
Each HTTP POST request contains an array of log records as a JSON array.

You configure batching behavior in the ClusterLogForwarder output specification.

2.2.2. Single record format
Copy link

When configured for single-record requests, the log collector sends each log record as a complete JSON object: 

{
  "kubernetes": {
    "namespace_name": "my-app",
    "pod_name": "web-1",
    "container_name": "nginx",
    "host": "ip-10-0-1-42.ec2.internal",
    "labels": {
      "app": "web",
      "version": "1.0"
    }
  },
  "message": "Error connecting to database",
  "level": "error",
  "@timestamp": "2024-01-15T10:30:00.000Z",
  "hostname": "ip-10-0-1-42.ec2.internal"
}

The HTTP endpoint receives this exact JSON object as the POST request body.

2.2.3. Batched records format
Copy link

When configured for batching, the log collector sends multiple log records as a JSON array: 

[
  {
    "kubernetes": {
      "namespace_name": "my-app",
      "pod_name": "web-1",
      "container_name": "nginx"
    },
    "message": "Request processed",
    "level": "info",
    "@timestamp": "2024-01-15T10:30:00.000Z",
    "hostname": "ip-10-0-1-42.ec2.internal"
  },
  {
    "kubernetes": {
      "namespace_name": "my-app",
      "pod_name": "web-2",
      "container_name": "nginx"
    },
    "message": "Error connecting to database",
    "level": "error",
    "@timestamp": "2024-01-15T10:30:01.000Z",
    "hostname": "ip-10-0-1-42.ec2.internal"
  }
]

Batching reduces network traffic for high-volume log forwarding. Configure batch size and timeout limits to balance throughput and latency.

2.2.4. Field structure
Copy link

The log collector preserves the complete OpenShift log record structure without transformation. All fields from the original log record appear in the JSON payload:

Common fields for all log types:

@timestamp
ISO 8601 timestamp when the log was created
hostname
Kubernetes node name where the log originated
message
Log message content (plain text or nested JSON)
level
Log severity level (for example, info, error, debug)

Fields for application and infrastructure container logs:

kubernetes

Object containing Kubernetes metadata:

  • namespace_name: Kubernetes namespace
  • pod_name: Pod name
  • container_name: Container name
  • host: Node hostname
  • labels: Pod labels as key-value pairs
  • annotations: Pod annotations as key-value pairs

Fields for infrastructure journal logs:

systemd

Object containing systemd journal metadata:

  • t.SYSTEMD_UNIT: Systemd unit name (for example, kubelet.service)
  • t.SYSLOG_IDENTIFIER: Process identifier
SYSLOG_IDENTIFIER
Process name (for example, kubelet, crio)

Fields for audit logs:

k8s_audit_level
Kubernetes API audit level
openshift_audit_level
OpenShift API audit level
auditID
Unique audit event identifier

2.2.5. Authentication and authorization
Copy link

The HTTP output supports the following authentication methods:

Bearer token
Include an Authorization: Bearer token header in all requests.
Basic authentication
Include an Authorization: Basic base64-credentials header in all requests.
TLS client certificates
Mutual TLS authentication by using client certificates.
Custom headers
Static or template-based custom headers for API key authentication.

Configure authentication credentials by using Secrets referenced in the ClusterLogForwarder output specification.

2.2.6. Error handling
Copy link

The HTTP endpoint must return specific status codes for the log collector to handle delivery correctly:

Status codeMeaningCollector behavior

200-299

Success

Log records marked as delivered, removed from retry queue

400

Bad Request

Log records discarded (not retried) - indicates malformed payload

401, 403

Unauthorized/Forbidden

Collector retries with exponential backoff - check credentials

429

Too Many Requests

Collector backs off and retries - rate limiting active

500-599

Server Error

Collector retries with exponential backoff - temporary failure

Connection refused

Network/endpoint unavailable

Collector retries with exponential backoff

Configure retry limits and timeout values in the ClusterLogForwarder output specification to control delivery behavior.

The following infrastructure log from systemd journal: 

{
  "message": "Started container",
  "systemd": {
    "t": {
      "SYSTEMD_UNIT": "kubelet.service"
    }
  },
  "SYSLOG_IDENTIFIER": "kubelet",
  "hostname": "ip-10-0-1-42.ec2.internal",
  "@timestamp": "2024-01-15T10:30:00.000Z"
}

The log collector sends this as an HTTP POST request: 

POST /logs HTTP/1.1
Host: logs.example.com
Content-Type: application/json
Authorization: Bearer <secret>
User-Agent: Vector/0.28.0

{
  "message": "Started container",
  "systemd": {
    "t": {
      "SYSTEMD_UNIT": "kubelet.service"
    }
  },
  "SYSLOG_IDENTIFIER": "kubelet",
  "hostname": "ip-10-0-1-42.ec2.internal",
  "@timestamp": "2024-01-15T10:30:00.000Z"
}
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top