kafka实践

安全终端模拟软件使用：xshell+sftp

集群部署：1 部署zookeeper集群

                     2 配置kafka/conf/server.properties 

                       配置 broker的id. 每个broker的id必须是唯一的

                       配置log.dirs存放log的目录

                       配置zookeeper.connect的连接,zookeeper集群,按逗号分开即可

                       配置host.name,为IP即可

kafka启动了，关闭运行窗口，kafka立刻停止运行，启动命令如下：

 bin/kafka-server-start.sh config/server.properties > stdout.log 2>&1 &

同事用这个命令启动，关闭运行窗口kafka后台会继续运行，百思不得其解

最后使用另外一种启动方式

nohup bin/kafka-server-start.sh config/server.properties > stdout.log 2>&1 &

就可以解决此问题

2016-08-26 14:27:50,240 [ERROR][main][com.ucloudlink.ass.container.Main.main(Main.java:86)] - Error creating bean with name 'balanceReminderConsumer' defined in URL [jar:file:/opt/crm/123-456-interface/lib/123-456-interface-0.0.1.jar!/com/u/ass/reminder/BalanceReminderConsumer.class]: Invocation of init method failed; nested exception is kafka.common.ConsumerRebalanceFailedException: BALANCEREMINDERGROP_sdf1472192861490-64d4c7e8 can't rebalance after 4 retries 
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'balanceReminderConsumer' defined in URL [jar:file:/opt/crm/ucl-crm-interface/lib/ucl-crm-interface-0.0.1.jar!/com/u/ass/reminder/BalanceReminderConsumer.class]: Invocation of init method failed; nested exception is kafka.common.ConsumerRebalanceFailedException: BALANCEREMINDERGROP_sdf1472192861490-64d4c7e8 can't rebalance after 4 retries 
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1553) ~[spring-beans-4.0.1.RELEASE.jar:4.0.1.RELEASE] 

解决方案：

zkclient-0.1.jar

zkclient-0.7.jar

冲突，删除zkclient-0.1.jar即可

生产者、消费者、重写分区接口

public class KafkaStatleConsumer

{
private final ConsumerConnector consumer;

private final String topic = "topictest";

private final String zk = "xxx:2181";

private final int numThreads = 3;

public static void main(String[] args)
{
KafkaStatleConsumer consumer = new KafkaStatleConsumer();
consumer.run();
}

public Properties getConsumerProps()
{
Properties consumerProps = new Properties();
consumerProps.put("zookeeper.connect", zk);
return consumerProps;
}

public KafkaStatleConsumer()
{
consumer = kafka.consumer.Consumer.createJavaConsumerConnector(createConsumerConfig());
}

private ConsumerConfig createConsumerConfig()
{
// 不同主题的不同groupID的consumerProps，需要不同设置，不能用同一个单例consumerProps
Properties props = getConsumerProps();
props.put("group.id", "testConsumer");
props.put("zookeeper.session.timeout.ms", "10000");
return new ConsumerConfig(props);
}

public void run()
{
Map<String, Integer> topickMap = new HashMap<String, Integer>();
topickMap.put(topic, numThreads);
Map<String, List<KafkaStream<byte[], byte[]>>> streamMap = consumer.createMessageStreams(topickMap);

List<KafkaStream<byte[], byte[]>> streams = streamMap.get(topic);
ExecutorService executor = Executors.newFixedThreadPool(numThreads);

/**
* 多线程消费分区
*/
for (final KafkaStream<byte[], byte[]> stream : streams)
{
Runnable runnable = new Runnable()
{
public void run()
{
ConsumerIterator<byte[], byte[]> it = stream.iterator();
while (it.hasNext())
{
MessageAndMetadata<byte[], byte[]> mam = it.next();
System.out.println(Thread.currentThread().getName()
+ ": partition[" + mam.partition() + "],"
+ "offset[" + mam.offset() + "], "
+ new String(mam.message()));
consumer.commitOffsets();
}
}
};
executor.submit(runnable);
}
}

}

public class KafkaStatleProducer
{
private final String server = "xxxx:9092";
private final static String topicName = "topictest";
private Properties props = new Properties();
private Producer<String, String> producer;

public static Properties getProductProps()
{
Properties producterProps;
producterProps = new Properties();
producterProps.setProperty("metadata.broker.list", "xxx:9092");
return producterProps;
}

public static void main(String[] args)
{
Properties props = new Properties();
props.put("metadata.broker.list", brokerList);
// 是否等待消息被提交的通知
props.put("request.required.acks", "1");

                //异步发送
props.put("producer.type", "async");
props.put("key.serializer.class", "kafka.serializer.StringEncoder");
//props.put("partitioner.class", BalanceReminderConstants.PATITIONER);
props.put("zk.connectiontimeout.ms", CONNECTION_TIMEOUT);
props.put("group.id", BalanceReminderConstants.BALANCEREMINDER_GROP);
props.put("queue.buffer.max.ms", "1000");
props.put("queue.buffering.max.ms", "1000");
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
try
{
int i = 0;
while (i < 100)
{
String msg = "sdfsdfsdf: " + i;
KeyedMessage<String, String> data = new KeyedMessage<String, String>(
topicName, i + "", msg);
producer.send(data);


System.out.println(data);
data = null;
i++;
}
} catch (Exception e)
{
e.printStackTrace();
}
producer.close();
}
}

public class PartitionerExt extends DefaultPartitioner
{

public PartitionerExt(VerifiableProperties props)
{
super(props);
}
public int partition(Object arg0, int arg1)
{
int result = Math.abs(arg0.hashCode()) % arg1;
System.out.println("current DefaultPartitioner :" + result);
return result;
}
}

如果你的分区数是N，那么最好线程数也保持为N，这样通常能够达到最大的吞吐量。超过N的配置只是浪费系统资源，因为多出的线程不会被分配到任何分区.

topic下的一个分区只能被同一个consumer group下的一个consumer线程来消费，但反之并不成立，即一个consumer线程可以消费多个分区的数据.即一个消费线程可以对应若干个分区，但一个分区只能被具体某一个消费线程消费

Kafka 0.8.2之后推出了Java版的全新的producer，这个producer有个参数batch.size，默认是16KB。它会为每个分区缓存消息，一旦满了就打包将消息批量发出

生产消息时不用重写DefaultPartitioner，指定发送时的Key就可以保证发送到特定分区，kafka内部保证

可通过 get /consumers/BALANCEREMINDERGROP/offsets/BALANCEREMINDERTOPICNAME/1 在zookeeper客户端获取kafka特定分区是否消费和offsets的值

一般使用单拉取线程，多线程消费，如果要使用多线程还要按用户保证消费顺序的话，可使用队列，将特定用户放入对列（每个线程配置一个队列）

在生产消息时可指定key,不用重写partition,即可实现用一个用户发送到特定分区

在server.properties中可配置分区数，在生产者生产消息时自动自动按配置分区数根据TOPIC创建分区

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic BALANCEREMINDERTOPICNAME

kafka-topics.bat --describe --zookeeper localhost:2181 --topic BALANCEREMINDERTOPICNAME

kafka-console-producer.bat --broker-list localhost:9092 --topic BALANCEREMINDERTOPICNAME

kafka-topics.bat zookeeper   localhost:2181 -alter --partitions 3 --topic BALANCEREMINDERTOPICNAME

kafka-console-consumer.bat --zookeeper localhost:2181 --from-beginning --topic BALANCEREMINDERTOPICNAME

kafka-topics.bat --delete --zookeeper localhost:2181 --topic BALANCEREMINDERTOPICNAME

zookeeper查看节点是否消费，get /consumers/grop/offsets/toppicName

kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group REMINDERGROP  --topic REMINDERTOPICNAME  --zookeeper 10.1.114.174:2181

GROUP     TOPIC     PID       OFFSET             LOGSIZE   LAG
消费者组  话题id    分区id    当前已消费的条数   总条数    未消费的条数