What is Apache Kafka?
Apache Kafka is a distributed streaming platform designed to build real-time pipelines and can be used as a message broker or as a replacement for a log aggregation solution for big data applications.
TL;DR
Run the application using Docker Compose
The main folder of this repository contains a functional
curl -sSL https://raw.githubusercontent.com/bitnami/containers/main/bitnami/kafka/docker-compose.yml > docker-compose.yml
docker-compose up -d
Why use Bitnami Images?
-
Bitnami closely tracks upstream source changes and promptly publishes new versions of this image using our automated systems.
-
With Bitnami images the latest bug fixes and features are available as soon as possible.
-
Bitnami containers, virtual machines and cloud images use the same components and configuration approach - making it easy to switch between formats based on your project needs.
-
All our images are based on
-
All Bitnami images available in Docker Hub are signed with
-
Bitnami container images are released on a regular basis with the latest distribution packages available.
How to deploy Apache Kafka in Kubernetes?
Deploying Bitnami applications as Helm Charts is the easiest way to get started with our applications on Kubernetes. Read more about the installation in the
Bitnami containers can be used with
Why use a non-root container?
Non-root container images add an extra layer of security and are generally recommended for production environments. However, because they run as a non-root user, privileged tasks are typically off-limits. Learn more about non-root containers
Supported tags and respective Dockerfile
links
Learn more about the Bitnami tagging policy and the difference between rolling tags and immutable tags
You can see the equivalence between the different tags by taking a look at the tags-info.yaml
file present in the branch folder, i.e bitnami/ASSET/BRANCH/DISTRO/tags-info.yaml
.
Subscribe to project updates by watching the
Get this image
The recommended way to get the Bitnami Apache Kafka Docker Image is to pull the prebuilt image from the
docker pull bitnami/kafka:latest
To use a specific version, you can pull a versioned tag. You can view the
docker pull bitnami/kafka:[TAG]
If you wish, you can also build the image yourself by cloning the repository, changing to the directory containing the Dockerfile and executing the docker build
command. Remember to replace the APP
, VERSION
and OPERATING-SYSTEM
path placeholders in the example command below with the correct values.
git clone https://github.com/bitnami/containers.git
cd bitnami/APP/VERSION/OPERATING-SYSTEM
docker build -t bitnami/APP:latest .
Persisting your data
If you remove the container all your data and configurations will be lost, and the next time you run the image the database will be reinitialized. To avoid this loss of data, you should mount a volume that will persist even after the container is removed.
Note: If you have already started using your database, follow the steps on
The image exposes a volume at /bitnami/kafka
for the Apache Kafka data. For persistence you can mount a directory at this location from your host. If the mounted directory is empty, it will be initialized on the first run.
Using Docker Compose:
This requires a minor change to the
kafka
NOTE: As this is a non-root container, the mounted files and directories must have the proper permissions for the UID
1001
.
Connecting to other containers
Using
Containers attached to the same network can communicate with each other using the container name as the hostname.
Using the Command Line
In this example, we will create an Apache Kafka client instance that will connect to the server instance that is running on the same docker network as the client.
Step 1: Create a network
docker network create app-tier --driver bridge
Step 2: Launch the Zookeeper server instance
Use the --network app-tier
argument to the docker run
command to attach the Zookeeper container to the app-tier
network.
docker run -d --name zookeeper-server \
--network app-tier \
-e ALLOW_ANONYMOUS_LOGIN=yes \
bitnami/zookeeper:latest
Step 2: Launch the Apache Kafka server instance
Use the --network app-tier
argument to the docker run
command to attach the Apache Kafka container to the app-tier
network.
docker run -d --name kafka-server \
--network app-tier \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 \
bitnami/kafka:latest
Step 3: Launch your Apache Kafka client instance
Finally we create a new container instance to launch the Apache Kafka client and connect to the server created in the previous step:
docker run -it --rm \
--network app-tier \
-e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 \
bitnami/kafka:latest kafka-topics.sh --list --bootstrap-server kafka-server:9092
Using a Docker Compose file
When not specified, Docker Compose automatically sets up a new network and attaches all deployed services to that network. However, we will explicitly define a new bridge
network named app-tier
. In this example we assume that you want to connect to the Apache Kafka server from your own custom application image which is identified in the following snippet by the service name myapp
.
version
IMPORTANT:
Please update the
YOUR_APPLICATION_IMAGE
placeholder in the above snippet with your application imageConfigure Apache Kafka and ZooKeeper persistence, and configure them either via environment variables or by
In your application container, use the hostname
kafka
to connect to the Apache Kafka server
Launch the containers using:
docker-compose up -d
Configuration
The configuration can easily be setup with the Bitnami Apache Kafka Docker image using the following environment variables:
-
ALLOW_PLAINTEXT_LISTENER
: Allow to use the PLAINTEXT listener. Default: no. -
KAFKA_INTER_BROKER_USER
: Apache Kafka inter broker communication user. Default: user. -
KAFKA_INTER_BROKER_PASSWORD
: Apache Kafka inter broker communication password. Default: bitnami. -
KAFKA_CERTIFICATE_PASSWORD
: Password for certificates. No defaults. -
KAFKA_HEAP_OPTS
: Apache Kafka's Java Heap size. Default: -Xmx1024m -Xms1024m. -
KAFKA_ZOOKEEPER_PROTOCOL
: Authentication protocol for Zookeeper connections. Allowed protocols: PLAINTEXT, SASL, SSL, and SASL_SSL. Defaults: PLAINTEXT. -
KAFKA_ZOOKEEPER_USER
: Apache Kafka Zookeeper user for SASL authentication. No defaults. -
KAFKA_ZOOKEEPER_PASSWORD
: Apache Kafka Zookeeper user password for SASL authentication. No defaults. -
KAFKA_ZOOKEEPER_TLS_KEYSTORE_PASSWORD
: Apache Kafka Zookeeper keystore file password and key password. No defaults. -
KAFKA_ZOOKEEPER_TLS_TRUSTSTORE_PASSWORD
: Apache Kafka Zookeeper truststore file password. No defaults. -
KAFKA_ZOOKEEPER_TLS_VERIFY_HOSTNAME
: Verify Zookeeper hostname on TLS certificates. Defaults: true. -
KAFKA_ZOOKEEPER_TLS_TYPE
: Choose the TLS certificate format to use. Allowed values:JKS
,PEM
. Defaults: JKS. -
KAFKA_CFG_SASL_ENABLED_MECHANISMS
: Allowed mechanism when using SASL either for clients, inter broker, or zookeeper comunications. Allowed values:PLAIN
,SCRAM-SHA-256
,SCRAM-SHA-512
or a comma separated combination of those values. Default: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512 -
KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL
: SASL mechanism to use for inter broker communications. No defaults. -
KAFKA_TLS_CLIENT_AUTH
: Configures kafka brokers to request client authentication. Allowed values:required
,requested
,none
. Defaults: required. -
KAFKA_TLS_TYPE
: Choose the TLS certificate format to use. Allowed values:JKS
,PEM
. Defaults: JKS. -
KAFKA_CLIENT_USERS
: Users that will be created into Zookeeper when using SASL for client communications. Separated by commas. Default: user -
KAFKA_CLIENT_PASSWORDS
: Passwords for the users specified atKAFKA_CLIENT_USERS
. Separated by commas. Default: bitnami -
KAFKA_CFG_MAX_PARTITION_FETCH_BYTES
: The maximum amount of data per-partition the server will return. Default: 1048576 -
KAFKA_CFG_MAX_REQUEST_SIZE
: The maximum size of a request in bytes. Default: 1048576 -
KAFKA_ENABLE_KRAFT
: Whether to enable Kafka Raft (KRaft) mode. Default: no -
KAFKA_KRAFT_CLUSTER_ID
: Kafka cluster ID when using Kafka Raft (KRaft). No defaults.
Additionally, any environment variable beginning with KAFKA_CFG_
will be mapped to its corresponding Apache Kafka key. For example, use KAFKA_CFG_BACKGROUND_THREADS
in order to set background.threads
or KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE
in order to configure auto.create.topics.enable
.
docker run --name kafka -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 -e ALLOW_PLAINTEXT_LISTENER=yes -e KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true bitnami/kafka:latest
or by modifying the
kafka
Apache Kafka development setup example
To use Apache Kafka in a development setup, create the following docker-compose.yml
file:
version
To deploy it, run the following command in the directory where the docker-compose.yml
file is located:
docker-compose up -d
Kafka without Zookeeper (KRaft)
Apache Kafka Raft (KRaft) makes use of a new quorum controller service in Kafka which replaces the previous controller and makes use of an event-based variant of the Raft consensus protocol. This greatly simplifies Kafka's architecture by consolidating responsibility for metadata into Kafka itself, rather than splitting it between two different systems: ZooKeeper and Kafka.
More Info can be found here: https://developer.confluent.io/learn/kraft/
NOTE: According to
Configuration here has been crafted from the
version: "3"
services:
- zookeeper:
- image: 'bitnami/zookeeper:latest'
- ports:
- - '2181:2181'
- environment:
- - ALLOW_ANONYMOUS_LOGIN=yes
kafka:
image: 'bitnami/kafka:latest'
ports:
- '9092:9092'
environment:
+ - KAFKA_ENABLE_KRAFT=yes
+ - KAFKA_CFG_PROCESS_ROLES=broker,controller
+ - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
- - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
+ - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
+ - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
- KAFKA_BROKER_ID=1
+ - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@127.0.0.1:9093
- - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- depends_on:
- - zookeeper
Accessing Apache Kafka with internal and external clients
In order to use internal and external clients to access Apache Kafka brokers you need to configure one listener for each kind of client.
To do so, add the following environment variables to your docker-compose:
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
+ - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
+ - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9093
+ - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093
+ - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT
And expose the external port:
(the internal, client one can still be used within the docker network)
ports:
- - '9092:9092'
+ - '9093:9093'
Note: To connect from an external machine, change localhost
above to your host's external IP/hostname and include EXTERNAL://0.0.0.0:9093
in KAFKA_CFG_LISTENERS
to allow for remote connections.
Producer and consumer using external client
These clients, from the same host, will use localhost
to connect to Apache Kafka.
kafka-console-producer.sh --bootstrap-server 127.0.0.1:9093 --topic test
kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9093 --topic test --from-beginning
If running these commands from another machine, change the address accordingly.
Producer and consumer using internal client
These clients, from other containers on the same Docker network, will use the kafka container service hostname to connect to Apache Kafka.
kafka-console-producer.sh --bootstrap-server kafka:9092 --topic test
kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning
Similarly, application code will need to use bootstrap.servers=kafka:9092
More info about Apache Kafka listeners can be found in
Security
The Bitnami Apache Kafka docker image disables the PLAINTEXT listener for security reasons. You can enable the PLAINTEXT listener by adding the next environment variable, but remember that this configuration is not recommended for production.
ALLOW_PLAINTEXT_LISTENER=yes
In order to configure authentication, you must configure the Apache Kafka listeners properly. This container assumes the names below will be used for the listeners:
-
INTERNAL: used for inter-broker communications.
-
CLIENT: used for communications with clients that are within the same network as Apache Kafka brokers.
Let's see an example to configure Apache Kafka with SASL_SSL
authentication for communications with clients, and SSL
authentication for inter-broker communication.
The environment variables below should be defined to configure the listeners, and the SASL credentials for client communications:
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:SSL,CLIENT:SASL_SSL
KAFKA_CFG_LISTENERS=INTERNAL://:9093,CLIENT://:9092
KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka:9093,CLIENT://kafka:9092
KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
KAFKA_CLIENT_USERS=user
KAFKA_CLIENT_PASSWORDS=password
You must also use your own certificates for SSL. You can drop your Java Key Stores or PEM files into /opt/bitnami/kafka/config/certs
. If the JKS or PEM certs are password protected (recommended), you will need to provide it to get access to the keystores:
KAFKA_CERTIFICATE_PASSWORD=myCertificatePassword
If the truststore is mounted in a different location than /opt/bitnami/kafka/config/certs/kafka.truststore.jks
, /opt/bitnami/kafka/conf/certs/kafka.truststore.pem
, /bitnami/kafka/conf/certs/kafka.truststore.jks
or /bitnami/kafka/conf/certs/kafka.truststore.pem
, set the KAFKA_TLS_TRUSTSTORE_FILE
variable.
The following script can help you with the creation of the JKS and certificates:
Keep in mind the following notes:
-
When prompted to enter a password, use the same one for all.
-
Set the Common Name or FQDN values to your Apache Kafka container hostname, e.g.
kafka.example.com
. After entering this value, when prompted "What is your first and last name?", enter this value as well.
-
As an alternative, you can disable host name verification setting the environment variable
KAFKA_CFG_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM
to an empty string.
-
-
When setting up a Apache Kafka Cluster (check the "Setting up an Apache Kafka Cluster") for more information), each Apache Kafka broker and logical client needs its own keystore. You will have to repeat the process for each of the brokers in the cluster.
The following docker-compose file is an example showing how to mount your JKS certificates protected by the password certificatePassword123
. Additionally it is specifying the Apache Kafka container hostname and the credentials for the client and zookeeper users.
version
In order to get the required credentials to consume and produce messages you need to provide the credentials in the client. If your Apache Kafka client allows it, use the credentials you've provided.
While producing and consuming messages using the bitnami/kafka
image, you'll need to point to the consumer.properties
and/or producer.properties
file, which contains the needed configuration to work. You can find this files in the /opt/bitnami/kafka/conf
directory.
Use this to generate messages using a secure setup:
export KAFKA_OPTS="-Djava.security.auth.login.config=/opt/bitnami/kafka/conf/kafka_jaas.conf"
kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --topic test --producer.config /opt/bitnami/kafka/conf/producer.properties
Use this to consume messages using a secure setup
export KAFKA_OPTS="-Djava.security.auth.login.config=/opt/bitnami/kafka/conf/kafka_jaas.conf"
kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test --consumer.config /opt/bitnami/kafka/conf/consumer.properties
If you use other tools to use your Apache Kafka cluster, you'll need to provide the required information. You can find the required information in the files located at /opt/bitnami/kafka/conf
directory.
InterBroker communications
When configuring your broker to use SASL
or SASL_SSL
for inter-broker communications, you can provide the SASL credentials using these environment variables:
-
KAFKA_INTER_BROKER_USER
: Apache Kafka inter broker communication user. Deprecated in favor ofKAFKA_CLIENT_USERS
. -
KAFKA_INTER_BROKER_PASSWORD
: Apache Kafka inter broker communication password. Deprecated in favor ofKAFKA_CLIENT_PASSWORDS
.
Apache Kafka client configuration
When configuring Apache Kafka with SASL
or SASL_SSL
for communications with clients, you can provide your SASL credentials using this environment variables:
-
KAFKA_CLIENT_USERS
: Apache Kafka client user. Default: user -
KAFKA_CLIENT_PASSWORDS
: Apache Kafka client user password. Default: bitnami
Apache Kafka ZooKeeper client configuration
There are different options of configuration to connect a Zookeeper server.
In order to connect a Zookeeper server without authentication, you should provide the environment variables below:
-
KAFKA_ZOOKEEPER_PROTOCOL
: PLAINTEXT.
In order to authenticate Apache Kafka against a Zookeeper server with SASL
, you should provide the environment variables below:
-
KAFKA_ZOOKEEPER_PROTOCOL
: SASL. -
KAFKA_ZOOKEEPER_USER
: Apache Kafka Zookeeper user for SASL authentication. No defaults. -
KAFKA_ZOOKEEPER_PASSWORD
: Apache Kafka Zookeeper user password for SASL authentication. No defaults.
In order to authenticate Apache Kafka against a Zookeeper server with SSL
, you should provide the environment variables below:
-
KAFKA_ZOOKEEPER_PROTOCOL
: SSL. -
KAFKA_ZOOKEEPER_TLS_KEYSTORE_PASSWORD
: Apache Kafka Zookeeper keystore file password and key password. No defaults. -
KAFKA_ZOOKEEPER_TLS_TRUSTSTORE_PASSWORD
: Apache Kafka Zookeeper truststore file password. No defaults. -
KAFKA_ZOOKEEPER_TLS_VERIFY_HOSTNAME
: Verify Zookeep