Install and setup Kafka on Windows

Kafka has become a very popular distributed message delivery service which decouples different services while making message delivery between service easy, fast and reliable. In this post, we will walk through how to install and set up Kafka on Windows.

Pre-requisite

• Java
• Apache ZooKeeper

Installation

Go to Kafka's download link and download the latest stable release, we downloaded 2.2.0 as of this writing. After downloading, copy the installation file to some folder and unzip it. 

Since Kafka uses ZooKeeper, so you also need to install ZooKeeper, The download link is at http://zookeeper.apache.org/releases.html#download. After it's installed, its folder structure looks like:

Setup

First let's set up ZooKeeper.

1. In conf folder, copy and paste zoo_sample.cfg to zoo.cfg
2. Open zoo.cfg, update dataDir to some real folder like dataDir=D:\Software\apache-zookeeper-3.5.5-bin.tar\apache-zookeeper-3.5.5-bin\apache-zookeeper-3.5.5-bin\data
3. Please check whether JAVA_HOME environment variable is set correctly to the JRE folder
4. Add the bin folder path to the Path environment variable

Now come back to Kafka setup. If you open the bin/windows directory, you can find the kafka-server-start.bat batch file which is the script to start the Kafka server. It needs a server.properties file which is sitting in the config folder. Now if you try to run below command, you may see below error.

kafka-server-start ../../config/server.properties

The output may look like.

[2019-06-01 13:41:50,073] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
        at kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:96)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
        at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:361)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:205)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
        at kafka.Kafka$.main(Kafka.scala:75)
        at kafka.Kafka.main(Kafka.scala)
[2019-06-01 13:41:50,087] INFO shutting down (kafka.server.KafkaServer)
[2019-06-01 13:41:50,104] INFO shut down completed (kafka.server.KafkaServer)
[2019-06-01 13:41:50,106] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2019-06-01 13:41:50,121] INFO shutting down (kafka.server.KafkaServer)

This is because Kafka uses Zookeeper, so a ZooKeeper server needs to be started first. Please open a command console, and type zkserver. You should see the zookeeper server starts in a few seconds.

Now if you run the kafka-server-start command again, you should see the Kafka server starts.

Testing

Now the server is up. We can create a topic which then producers and consumers can produce and consume data to and from.

Run below command to create a topic named test and it has only one partition and one replication instance.

kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

Once the topic is created, you can verify it by run below command to list the topic.

kafka-topics --list --bootstrap-server localhost:9092

It will print the name of the topic as test which means the topic test has been created. Congrats.

Once the topic is created successfully, messages can be published to this topic. Let's give a try. 

kafka-console-producer --broker-list localhost:9092 --topic test

Above command will start a producer which subscribes to the broker we created earlier. Type some messages and press enter. The messages will be sent to topic test.

Now let's start a consumer to see whether we can consume the messages published. 

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

Above command starts a new consumer and will consume message from beginning.

So we can confirm that the messages have been successfully published and consumed. 

There are lots of other set up can be done for Kafka to support differnt purposes. For example, we can set up a cluster of more than one node, can create multiple partitions. You can refer to Kafka's QuickStart guide for more information or come back later where we may provide you some guides on how those can be done.