Introduction

In this post we will focus on setting up and running Apache KAFKA on Windows OS. We will also have a quick look at creating Topics, sending and consuming the messages from Topic.

Install Directory Setup

JRE Server, Zookeeper and KAFKA can be installed in any folder on Windows machine. However, for this post I will install all the components in folders as shown in below image:

image

  • Base Folder  - C:\KAFKA_Setup
  • JREServer      - C:\KAFKA_Setup\JREServer
  • Zookeeper    - C:\KAFKA_Setup\Zookeeper
  • KAFKA           - C:\KAFKA_Setup\Kafka
  • Temp             - As name implies, this is a temporary folder that we will use to extract the JREServer, Zookeeper and KAFKA components before moving to their respective folders (As listed above).

JRE Installation

Download JRE Server for Windows from here. After opening the link “Accept the license” and click on Download for Windows version as shown in below image:

image

As listed above, I will use the temp folder to extract the JRE Server files. Once extracted copy the jdk (jdk1.8.0_144) folder to JREServer folder as shown in image below.

image

Now, we will create an environment variable and setup path variable for the above folder. Navigate to the Environment Variables as shown in image below:

image

image

Click New as shown in above image following which add the details of the System Variable as shown in image below:

image

Once the above variable is created, edit the Path System Variable and add a new entry for Java application path as shown in image below:

image

To check whether Java has been successfully installed, open a command prompt and run following command: java –version

The output should show the version that we downloaded from the oracle website above. The output should be as shown in below image.

image

Zookeeper Installation

There are 2 options to setup and run Zookeeper.

  1. Host Zookeeper independently
  2. Host Zookeeper that comes packaged with KAFKA.

In this post we will install Zookeeper separately and use it with KAFKA.

Download Zookeeper from here. Click on Download as shown in below image.image 

You will be taken to Apache mirror site and click the suggested mirror site as shown in below image.

image

Click on the above highlighted mirror site and you will be taken to Zookeeper releases as shown in below image.

image

Click on the stable release as highlighted in above image and that will take you to the stable version download page as shown in below image.

image

Click on the above Zookeeper version download link and then extract the downloaded copy in the Temp folder. Afterwards move the extracted copy to the Zookeeper folder and it should eventually be as shown in below image.

image

Now we need to update environment variables for Zookeeper following the same steps that we followed to configure JRE server. Add the following system variable:

Variable name: ZOOKEEPER_HOME, Variable value: C:\KAFKA_Setup\Zookeeper\zookeeper-3.4.10

Once the above variable is created, add a new entry in Path System Variable with value - %ZOOKEEPER_HOME%\bin.

Validate the path variable is updated to correct value by running path command in command prompt as shown in image below.

image

Now navigate to the “C:\KAFKA_Setup\Zookeeper\zookeeper-3.4.10\conf” folder and rename zoo_sample.cfg file to zoo.cfg file. After renaming the file open the file with any editor (notepad). Once opened search for dataDir configuration setting and update the value as shown in image below

image 

Start Zookeeper

Open Command prompt and run the following zkserver command to start ZooKeeper on default port (2181). If everything is configured correctly Zookeeper should run as shown in image below.

image

 

KAFKA Installation

Download KAFKA from here. Click on Download as shown in below image.

image

Clicking on above link will take you to Apache download page as shown in below image.

image

Click on the link as shown in above image to start the download. Once downloaded extract the KAFKA components in temp folder followed by moving to KAFKA folder as shown in image below.

image

Navigate to the Kafka folder at following path: C:\KAFKA_Setup\Kafka\kafka_2.11-0.11.0.0\config. Locate and open for editing "server.properties" file. Search in file log.dirs and update it as following - log.dirs=kafka-logs (as shown in image below)

image

Kafka by default will run on default port 9092 & connect to zookeeper on default port which is 2181 (Zookeeper connection settings can be updated in server.properties file itself).

Start KAFKA

  • Make sure that Zookeeper is started as shown in Start Zookeeper section earlier.
  • Open another command prompt and navigate to KAFKA root folder (C:\KAFKA_Setup\Kafka\kafka_2.11-0.11.0.0).
  • Run the following command to start KAFKA - bin\windows\kafka-server-start.bat config\server.properties

KAFKA server should start as shown in below image (Left command prompt window is running Zookeeper & right one is running KAFKA server).

image

In this section we ran the Zookeeper and KAFKA server. In the next section we will verify the overall setup of KAFKA server by creating a topic, sending message to the Topic and consuming the message from the topic.

Verify Zookeeper and KAFKA setup

In this section we will use the KAFKA server to send and consume messages from the topics. Make sure that Zookeeper and KAFKA are running.

Create Topic

  1. KAFKA comes up with many command line utilities. We will use kafka-topics.bat to create the topics.
  2. Open a new command prompt and navigate to KAFKA root folder (C:\KAFKA_Setup\Kafka\kafka_2.11-0.11.0.0)
  3. Execute the following command to create a topic named messagetopic - bin\windows\kafka-topics.bat –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic messagetopic
  4. Now check the list of topics again by executing following command - bin\windows\kafka-topics.bat –list –zookeeper localhost:2181

Send Message To Topic

  1. We will use Kafka-console-producer.bat to send the message to topic created in previous step.
  2. Open a new command prompt and navigate to KAFKA root folder (C:\KAFKA_Setup\Kafka\kafka_2.11-0.11.0.0)
  3. To send a message to Topic (messagetopic) created in previous step run the following command - bin\windows\kafka-console-producer.bat –broker-list localhost:9092 –topic messagetopic
  4. The above command will open a channel to Kafka topic and you can send many messages. Type message and press enter to send the message to KAFKA topic.

Consume Messages from Topic

  1. We will use Kafka-console-consumer.bat to send the message to topic created in previous step.
  2. Open a new command prompt and navigate to KAFKA root folder (C:\KAFKA_Setup\Kafka\kafka_2.11-0.11.0.0).
  3. To receive a message from topic (messagetopic) created in Create Topic step run the following command - bin\windows\kafka-console-consumer.bat -bootstrap-server localhost:9092 -topic messagetopic -from-beginning
  4. You should see the message that you have sent in the previous step.
  5. Try sending messages from the producer console window and you should immediately see the message in the consumer window.

The above steps to create topics, send and consume messages from topics is shown in image below.

image

Conclusion

In this post we covered how to setup KAFKA on a Windows machine and verified communication from producer to consumer via Kafka topics. In the next post we will go through details of KAFKA operations like List Topics, Modify Topics, Delete Topics, Topic Partitions, Multiple producers, Consumer Groups, Offsets etc.