Install Clustered (4 nodes) Apache Kafka and Zookeeper on Ubuntu 10.04

http://www.jonzobrist.com/2012/04/17/install-apache-kafka-and-zookeeper-on-ubuntu-10-04/

Roughly followed quickstart guide
http://incubator.apache.org/kafka/quickstart.html

Need Java? Pissed that Ubuntu dropped official Sun JDK from their Partner Repository?
Use
https://launchpad.net/~ferramroberto/+archive/java
Need this to add-apt-repository
apt-get install -y python-software-properties
sudo add-apt-repository ppa:ferramroberto/java
apt-get update
apt-get -y install sun-java6-jdk
java -version
#Should return something like
java version “1.6.0_26″
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Note that repository is yet to have a JDK for Ubuntu 12.04

I worked around this with a binary install of the latest 1.6 JDK to /usr/java

#Setup the Kafka user, install Kafka

I’m using 4 nodes, their IP’s are 192.168.1.50,51,52,53, all run Kafka and Zookeeper. The Zookeeper Admin guide recommends an odd number for the ensemble so majority rule is less likely to be split evenly/split brain style. although I did get an error from one of them urging me to run either Kafka or Zookeeper in an odd number of nodes, which I’m not sure the reasoning behind.

groupadd -g 3320 kafka
useradd -m -d /usr/local/kafka -s /bin/bash -u 3320 -g 3320 kafka
wget http://people.apache.org/~nehanarkhede/kafka-0.7.0-incubating/kafka-0.7.0-incubating-src.tar.gz
tar -zxvf kafka-0.7.0-incubating-src.tar.gz
cd kafka-0.7.0-incubating-src
mv * ~kafka/
chown -R kafka:kafka ~kafka
chown -R kafka:kafka ~kafka/*

su kafka
cd
./sbt update
./sbt package
#Now edit ~kafka/.bashrc and add these lines, you may have to change the quotes, since web clipboards will likely paste in non working versions of fancy quotes
declare -x JAVA_OPTS=”-Xmx3600M -Xms256M”
declare -x PATH=”${HOME}/bin:${PATH}”
mkdir -p ~/zookeeper ~/logs ~/run/logs
#Configure Kafka
#setup myid file, myid has to match the server’s setting in ~kafka/config/zookeeper.properties
#This will be different on each server (1,2,3,4)
echo “2” > /usr/local/kafka/zookeeper/myid
#Setup Kafka’s config file, server.properties, it needs direct pointers to all servers in the Kafka cluster on the zk.connect setting

 brokerid=2 port=9092 num.threads=8 socket.send.buffer=1048576 socket.receive.buffer=1048576 max.socket.request.bytes=104857600 log.dir=/usr/local/kafka/logs num.partitions=1 log.flush.interval=10000 log.default.flush.interval.ms=1000 log.default.flush.scheduler.interval.ms=1000 log.retention.hours=168 log.file.size=536870912 log.cleanup.interval.mins=1 enable.zookeeper=true zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181 zk.connectiontimeout.ms=1000000

#Setup Zookeeper’s config file, needs server.X for each node along with IP:port:port
vi config/zookeeper.properties
I ran this bash one liner on each node to get the server.X lines

echo "server.$(grep brokerid server.properties  | sed 's/=/ /' | awk '{ print $2 }')=$(hostname -I):2888:3888"

Then removed the spaces and put them into zookeeper.properties

dataDir=/tmp/zookeeper clientPort=2181 maxClientCnxns=0 tickTime=2000 dataDir=/usr/local/kafka/zookeeper/ initLimit=5 syncLimit=2 server.1=192.168.1.50:2888:3888 server.2=192.168.1.51:2888:3888 server.3=192.168.1.52:2888:3888 server.4=192.168.1.53:2888:3888

#Setup Kafka’s producer.properties

 broker.list=1:192.168.1.50:9092,2:192.168.1.51:9092,3:192.168.1.52:9092,4:192.168.1.53:9092 zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181 producer.type=sync compression.codec=0 serializer.class=kafka.serializer.StringEncoder

#Setup Kafka’s consumer.properties

 zk.connect=192.168.1.50:2181,192.168.1.51:2181,192.168.1.52:2181,192.168.1.53:2181 zk.connectiontimeout.ms=1000000 groupid=test-consumer-group

#Start it all, the docs say to use something like djb’s daemon tools, but for now I just run these in a screen, as user kafka

screen -R zookeepercd~/bin/zookeeper-server-start.sh ~/config/zookeeper.propertiesCTRL+A Dscreen -R kafkacd~/bin/kafka-server-start.sh ~/config/server.propertiesCTRL+A D

Check zookeeper

telnet localhost 2181ruok

#expect back imok

#Send some Kafka messages
#Start Producer

bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic testThis is a testThis is more testing

#Shutdown producer
CTRL+D
#Start consumer
#Receive the messages

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

#Expect to see
This is a test
This is more testing
References
http://incubator.apache.org/kafka/quickstart.html
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html
http://zookeeper.apache.org/doc/r3.4.3/recipes.html
http://zookeeper.apache.org/doc/trunk/zookeeperJMX.html

Download Kafka from
http://incubator.apache.org/kafka/downloads.html