使用Java写kafka生产者消费者

快速启动

步骤1：下载代码

下载最近的一个稳定版本。

> tar xzf kafka-<VERSION>.tgz> cd kafka-<VERSION>> ./sbt update> ./sbt package

步骤2：启动服务器

Kafka brokers and consumers use this for co-ordination.

 bin/zookeeper-server-start.sh config/zookeeper.properties  [2010-11-21 23:45:02,335] INFO Reading configuration from: config/zookeeper.properties "><b>>斌/ zookeeper-server-start.sh配置/ zookeeper.properties</b> [ 2010-11-21 23:45:02335 ]信息读取配置：配置/ zookeeper.properties …

现在开始卡夫卡服务器：

> bin/kafka-server-start.sh config/server.propertiesjkreps-mn-2:kafka-trunk jkreps$ bin/kafka-server-start.sh config/server.properties [2010-11-21 23:51:39,608] INFO starting log cleaner every 60000 ms (kafka.log.LogManager)[2010-11-21 23:51:39,628] INFO connecting to ZK: localhost:2181 (kafka.server.KafkaZooKeeper)...

步骤3：发送一些信息

卡夫卡是一个命令行客户端将从标准输入并把它作为消息的卡夫卡群。默认情况下，每个线将作为一个单独的消息发送。这个话题测试时自动创建的消息发送给它的。省略记录你应该看到这样的事情：

 
> bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test This is a messageThis is another message

步骤4：启动消费

kafka也有一个命令行的消费者，将把信息到标准输出。

> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginningThis is a messageThis is another message

下面是一些很简单的用kafka发送消息的例子，更完整的例子可以在实例目录中找到kafka的源代码。

生产者代码

生产API

Here are examples of using the producer API - kafka.producer.Producer<T> -

1. 首先，开始的ZooKeeper服务器本地实例

   ./bin/zookeeper-server-start.sh config/zookeeper.properties

2. 接下来，开始kafka配置

   ./bin/kafka-server-start.sh config/server.properties

3. 现在，创建所有配置默认发现使用ZooKeeper生产者。

4. 使用异步生产者以及gzip压缩。这个缓冲区写入内存直到batch.size或queue.time达到。然后，数据被发送到kafka

   Hadoop的消费者

   聚合和数据加载到Hadoop提供横向扩展的解决方案是我们的一个基本用例。为了支持这个用例，我们提供了一个基于Hadoop的消费滋生了许多地图任务并行从卡夫卡群拉数据。这提供了非常快速的基于拉的Hadoop数据的负载能力（我们能够完全饱和，只有少数卡夫卡服务器网络）。

   在Hadoop的用户使用信息可以发现在这里。

   简单的消费

   import kafka.api.FetchRequest;import kafka.javaapi.consumer.SimpleConsumer;import kafka.javaapi.message.ByteBufferMessageSet;import kafka.message.Message;import kafka.message.MessageSet;import kafka.utils.Utils;...

   // create a consumer to connect to the kafka server running on localhost, port 9092, socket timeout of 10 secs, socket receive buffer of ~1MBSimpleConsumer consumer = new SimpleConsumer("127.0.0.1", 9092, 10000, 1024000);long offset = 0;while (true) {  // create a fetch request for topic “test”, partition 0, current offset, and fetch size of 1MB  FetchRequest fetchRequest = new FetchRequest("test", 0, offset, 1000000);  // get the message set from the consumer and print them out  ByteBufferMessageSet messages = consumer.fetch(fetchRequest);  for(MessageAndOffset msg : messages) {    System.out.println("consumed: " + Utils.toString(msg.message.payload(), "UTF-8"));    // advance the offset after consuming each message    offset = msg.offset;  }}

   官方给的示例并不是很完整，以下代码是经过我补充的并且编译后能运行的。

   Producer Code
   import java.util.*;import kafka.message.Message;import kafka.producer.ProducerConfig;import kafka.javaapi.producer.Producer;import kafka.javaapi.producer.ProducerData;
   public class ProducerSample {
    public static void main(String[] args) {  ProducerSample ps = new ProducerSample();
     Properties props = new Properties();  props.put("zk.connect", "127.0.0.1:2181");  props.put("serializer.class", "kafka.serializer.StringEncoder");
     ProducerConfig config = new ProducerConfig(props);  Producer<String, String> producer = new Producer<String, String>(config);  ProducerData<String, String> data = new ProducerData<String, String>("test-topic", "test-message2");  producer.send(data);  producer.close(); }}
   Consumer Code
   import java.nio.ByteBuffer;import java.util.HashMap;import java.util.List;import java.util.Map;import java.util.Properties;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import kafka.consumer.Consumer;import kafka.consumer.ConsumerConfig;import kafka.consumer.KafkaStream;import kafka.javaapi.consumer.ConsumerConnector;import kafka.message.Message;import kafka.message.MessageAndMetadata;
   public class ConsumerSample {
    public static void main(String[] args) {                // specify some consumer properties  Properties props = new Properties();  props.put("zk.connect", "localhost:2181");  props.put("zk.connectiontimeout.ms", "1000000");  props.put("groupid", "test_group");
                   // Create the connection to the cluster  ConsumerConfig consumerConfig = new ConsumerConfig(props);  ConsumerConnector consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);
                   // create 4 partitions of the stream for topic “test-topic”, to allow 4 threads to consume  HashMap<String, Integer> map = new HashMap<String, Integer>();  map.put("test-topic", 4);  Map<String, List<KafkaStream<Message>>> topicMessageStreams =    consumerConnector.createMessageStreams(map);  List<KafkaStream<Message>> streams = topicMessageStreams.get("test-topic");
                   // create list of 4 threads to consume from each of the partitions   ExecutorService executor = Executors.newFixedThreadPool(4);
                   // consume the messages in the threads  for (final KafkaStream<Message> stream : streams) {   executor.submit(new Runnable() {    public void run() {     for (MessageAndMetadata msgAndMetadata : stream) {      // process message (msgAndMetadata.message())      System.out.println("topic: " + msgAndMetadata.topic());      Message message = (Message) msgAndMetadata.message();      ByteBuffer buffer = message.payload();     <SPAN style="WHITE-SPACE: pre"> </SPAN>byte[] bytes = new byte[message.payloadSize()];      buffer.get(bytes);      String tmp = new String(bytes);      System.out.println("message content: " + tmp);     }    }   });  }
    }}
   分别启动zookeeper,kafka server之后，依次运行Producer,Consumer的代码
   运行ProducerSample：
   运行ConsumerSample:
   

   <trans data-src="" create="" a="" consumer="" to="" connect="" the="" kafka="" server="" running="" on="" localhost,="" port="" 9092,="" socket="" timeout="" of="" 10="" secs,="" receive="" buffer="" ~1mb="" simpleconsumer="" simpleconsumer("127.0.0.1",="" 10000,="" 1024000);"="" style="background: transparent;">