Knative Kafka provides integration options for you to use supported versions of the Apache Kafka message streaming platform with OpenShift Serverless. Kafka provides options for event source, channel, broker, and event sink capabilities.
Knative Kafka functionality is available in an OpenShift Serverless installation if a cluster administrator has installed the KnativeKafka
custom resource.
Knative Kafka is not currently supported for IBM Z and IBM Power Systems. |
Knative Kafka provides additional options, such as:
Kafka source
Kafka channel
Kafka broker
Kafka sink
Using Kafka components in an event-driven architecture provides "at least once" event delivery. This means that operations are retried until a return code value is received. This makes applications more resilient to lost events; however, it might result in duplicate events being sent.
For the Kafka event source, there is a fixed number of retries for event delivery by default. For Kafka channels, retries are only performed if they are configured in the Kafka channel Delivery
spec.
See the Event delivery documentation for more information about delivery guarantees.
You can create a Kafka source that reads events from an Apache Kafka cluster and passes these events to a sink. You can create a Kafka source by using the OpenShift Container Platform web console, the Knative (kn
) CLI, or by creating a KafkaSource
object directly as a YAML file and using the OpenShift CLI (oc
) to apply it.
After Knative Kafka is installed on your cluster, you can create a Kafka source by using the web console. Using the OpenShift Container Platform web console provides a streamlined and intuitive user interface to create a Kafka source.
The OpenShift Serverless Operator, Knative Eventing, and the KnativeKafka
custom resource are installed on your cluster.
You have logged in to the web console.
You have access to a Red Hat AMQ Streams (Kafka) cluster that produces the Kafka messages you want to import.
You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.
In the Developer perspective, navigate to the +Add page and select Event Source.
In the Event Sources page, select Kafka Source in the Type section.
Configure the Kafka Source settings:
Add a comma-separated list of Bootstrap Servers.
Add a comma-separated list of Topics.
Add a Consumer Group.
Select the Service Account Name for the service account that you created.
Select the Sink for the event source. A Sink can be either a Resource, such as a channel, broker, or service, or a URI.
Enter a Name for the Kafka event source.
Click Create.
You can verify that the Kafka event source was created and is connected to the sink by viewing the Topology page.
In the Developer perspective, navigate to Topology.
View the Kafka event source and sink.
You can use the kn source kafka create
command to create a Kafka source by using the Knative (kn
) CLI. Using the Knative CLI to create event sources provides a more streamlined and intuitive user interface than modifying YAML files directly.
The OpenShift Serverless Operator, Knative Eventing, Knative Serving, and the KnativeKafka
custom resource (CR) are installed on your cluster.
You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in OpenShift Container Platform.
You have access to a Red Hat AMQ Streams (Kafka) cluster that produces the Kafka messages you want to import.
You have installed the Knative (kn
) CLI.
Optional: You have installed the OpenShift CLI (oc
) if you want to use the verification steps in this procedure.
To verify that the Kafka event source is working, create a Knative service that dumps incoming events into the service logs:
$ kn service create event-display \
--image quay.io/openshift-knative/knative-eventing-sources-event-display
Create a KafkaSource
CR:
$ kn source kafka create <kafka_source_name> \
--servers <cluster_kafka_bootstrap>.kafka.svc:9092 \
--topics <topic_name> --consumergroup my-consumer-group \
--sink event-display
Replace the placeholder values in this command with values for your source name, bootstrap servers, and topics. |
The --servers
, --topics
, and --consumergroup
options specify the connection parameters to the Kafka cluster. The --consumergroup
option is optional.
Optional: View details about the KafkaSource
CR you created:
$ kn source kafka describe <kafka_source_name>
Name: example-kafka-source
Namespace: kafka
Age: 1h
BootstrapServers: example-cluster-kafka-bootstrap.kafka.svc:9092
Topics: example-topic
ConsumerGroup: example-consumer-group
Sink:
Name: event-display
Namespace: default
Resource: Service (serving.knative.dev/v1)
Conditions:
OK TYPE AGE REASON
++ Ready 1h
++ Deployed 1h
++ SinkProvided 1h
Trigger the Kafka instance to send a message to the topic:
$ oc -n kafka run kafka-producer \
-ti --image=quay.io/strimzi/kafka:latest-kafka-2.7.0 --rm=true \
--restart=Never -- bin/kafka-console-producer.sh \
--broker-list <cluster_kafka_bootstrap>:9092 --topic my-topic
Enter the message in the prompt. This command assumes that:
The Kafka cluster is installed in the kafka
namespace.
The KafkaSource
object has been configured to use the my-topic
topic.
Verify that the message arrived by viewing the logs:
$ oc logs $(oc get pod -o name | grep event-display) -c user-container
☁️ cloudevents.Event
Validation: valid
Context Attributes,
specversion: 1.0
type: dev.knative.kafka.event
source: /apis/v1/namespaces/default/kafkasources/example-kafka-source#example-topic
subject: partition:46#0
id: partition:46/offset:0
time: 2021-03-10T11:21:49.4Z
Extensions,
traceparent: 00-161ff3815727d8755848ec01c866d1cd-7ff3916c44334678-00
Data,
Hello!