Using a Kafka source - Event sources | Serverless | OpenShift Container Platform 4.7

Prerequisites
Creating a Kafka event source by using the web console
Creating a Kafka event source by using the Knative CLI
- Knative CLI --sink flag
Creating a Kafka event source by using YAML
Additional resources

You can create a Knative Kafka event source that reads events from an Apache Kafka cluster and passes these events to a sink.

Prerequisites

You can use the KafkaSource event source with OpenShift Serverless after you have Knative Eventing and Knative Kafka installed on your cluster.

Creating a Kafka event source by using the web console

You can create and verify a Kafka event source from the OpenShift Container Platform web console.

Prerequisites

The OpenShift Serverless Operator, Knative Eventing, and the KnativeKafka custom resource are installed on your cluster.
You have logged in to the web console.
You are in the Developer perspective.
You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.

Procedure

Navigate to the Add page and select Event Source.
In the Event Sources page, select Kafka Source in the Type section.
Configure the Kafka Source settings:
1. Add a comma-separated list of Bootstrap Servers.
2. Add a comma-separated list of Topics.
3. Add a Consumer Group.
4. Select the Service Account Name for the service account that you created.
5. Select the Sink for the event source. A Sink can be either a Resource, such as a channel, broker, or service, or a URI.
6. Enter a Name for the Kafka event source.
Click Create.

Verification

You can verify that the Kafka event source was created and is connected to the sink by viewing the Topology page.

In the Developer perspective, navigate to Topology.
View the Kafka event source and sink.

Creating a Kafka event source by using the Knative CLI

This section describes how to create a Kafka event source by using the kn command.

Prerequisites

The OpenShift Serverless Operator, Knative Eventing, Knative Serving, and the KnativeKafka custom resource (CR) are installed on your cluster.
You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.
You have access to a Red Hat AMQ Streams (Kafka) cluster that produces the Kafka messages you want to import.

Procedure

To verify that the Kafka event source is working, create a Knative service that dumps incoming events into the service logs:
```
$ kn service create event-display \
    --image quay.io/openshift-knative/knative-eventing-sources-event-display
```
Create a KafkaSource CR:
```
$ kn source kafka create <kafka_source_name> \
    --servers <cluster_kafka_bootstrap>.kafka.svc:9092 \
    --topics <topic_name> --consumergroup my-consumer-group \
    --sink event-display
```
Replace the placeholder values in this command with values for your source name, bootstrap servers, and topics.

The --servers, --topics, and --consumergroup options specify the connection parameters to the Kafka cluster. The --consumergroup option is optional.

Optional: View details about the KafkaSource CR you created:

$ kn source kafka describe <kafka_source_name>

Example output

Name:              example-kafka-source
Namespace:         kafka
Age:               1h
BootstrapServers:  example-cluster-kafka-bootstrap.kafka.svc:9092
Topics:            example-topic
ConsumerGroup:     example-consumer-group

Sink:
  Name:       event-display
  Namespace:  default
  Resource:   Service (serving.knative.dev/v1)

Conditions:
  OK TYPE            AGE REASON
  ++ Ready            1h
  ++ Deployed         1h
  ++ SinkProvided     1h

Verification steps

Trigger the Kafka instance to send a message to the topic:

$ oc -n kafka run kafka-producer \
    -ti --image=quay.io/strimzi/kafka:latest-kafka-2.7.0 --rm=true \
    --restart=Never -- bin/kafka-console-producer.sh \
    --broker-list <cluster_kafka_bootstrap>:9092 --topic my-topic

Enter the message in the prompt. This command assumes that:

The Kafka cluster is installed in the kafka namespace.
The KafkaSource object has been configured to use the my-topic topic.

Verify that the message arrived by viewing the logs:

$ oc logs $(oc get pod -o name | grep event-display) -c user-container

Example output

☁️  cloudevents.Event
Validation: valid
Context Attributes,
  specversion: 1.0
  type: dev.knative.kafka.event
  source: /apis/v1/namespaces/default/kafkasources/example-kafka-source#example-topic
  subject: partition:46#0
  id: partition:46/offset:0
  time: 2021-03-10T11:21:49.4Z
Extensions,
  traceparent: 00-161ff3815727d8755848ec01c866d1cd-7ff3916c44334678-00
Data,
  Hello!

Knative CLI --sink flag

When you create an event-producing custom resource by using the Knative (kn) CLI, you can specify a sink where events are sent to from that resource, by using the --sink flag.

The following example creates a sink binding that uses a service, http://event-display.svc.cluster.local, as the sink:

Example command using the --sink flag

$ kn source binding create bind-heartbeat \
  --namespace sinkbinding-example \
  --subject "Job:batch/v1:app=heartbeat-cron" \
  --sink http://event-display.svc.cluster.local \ (1)
  --ce-override "sink=bound"

1	`svc` in `http://event-display.svc.cluster.local` determines that the sink is a Knative service. Other default sink prefixes include `channel`, and `broker`.

Creating a Kafka event source by using YAML

You can create a Kafka event source by using YAML.

Prerequisites

The OpenShift Serverless Operator, Knative Eventing, and the KnativeKafka custom resource are installed on your cluster.
You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.

Procedure

Create a KafkaSource object as a YAML file:

apiVersion: sources.knative.dev/v1beta1
kind: KafkaSource
metadata:
  name: <source_name>
spec:
  consumerGroup: <group_name> (1)
  bootstrapServers:
  - <list_of_bootstrap_servers>
  topics:
  - <list_of_topics> (2)
  sink:
  - <list_of_sinks> (3)

1	A consumer group is a group of consumers that use the same group ID, and consume data from a topic.
2	A topic provides a destination for the storage of data. Each topic is split into one or more partitions.
3	A sink specifies where events are sent to from a source.

Only the v1beta1 version of the API for KafkaSource objects on OpenShift Serverless is supported. Do not use the v1alpha1 version of this API, as this version is now deprecated.

Example KafkaSource object

apiVersion: sources.knative.dev/v1beta1
kind: KafkaSource
metadata:
  name: kafka-source
spec:
  consumerGroup: knative-group
  bootstrapServers:
  - my-cluster-kafka-bootstrap.kafka:9092
  topics:
  - knative-demo-topic
  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: event-display

Apply the KafkaSource YAML file:
```
$ oc apply -f <filename>
```

Verification

Verify that the Kafka event source was created by entering the following command:

$ oc get pods

Example output

NAME                                    READY     STATUS    RESTARTS   AGE
kafkasource-kafka-source-5ca0248f-...   1/1       Running   0          13m

Additional resources

See Understanding event sources.
See Knative Kafka.
See the Red Hat AMQ Streams documentation for more information about Kafka concepts.