kafka_2.9.2-0.8.1.1安装、测试、集群
来源:互联网 发布:淘宝客收入 编辑:程序博客网 时间:2024/04/29 20:11
Kafka版本:kafka_2.9.2-0.8.1.1
官网:http://kafka.apache.org/
官方文档:http://kafka.apache.org/documentation.html#quickstart
一、安装
下载解压
[root@rs229 ~]# wget -c -P /root http://mirrors.cnnic.cn/apache/kafka/0.8.1.1/kafka_2.9.2-0.8.1.1.tgz
# tar xzf kafka_2.9.2-0.8.1.1.tgz
# cd kafka_2.9.2-0.8.1.1
二、配置
# cd kafka_2.9.2-0.8.1.1
# cd config
主要是配置 server.properties 和 zookeeper.properties
[ 配置文件简单,大家根据文件里的注释配一下就好 ]
三、启动
启动自带的zookeeper,也可以不用
[root@rs229 kafka_2.9.2-0.8.1.1]#bin/zookeeper-server-start.sh config/zookeeper.properties &
启动kafka server,不使用自带的要注意修改zookeeper地址
[root@rs229 kafka_2.9.2-0.8.1.1]#bin/kafka-server-start.sh config/server.properties &
[ 意后台启动服务 ]
使用介绍:
创建topic
[root@rs229 kafka_2.9.2-0.8.1.1]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test &
列出topic
[root@rs229 kafka_2.9.2-0.8.1.1]# bin/kafka-topics.sh --list --zookeeper localhost:2181
test
producer
# Send some messages (发送一些消息)
输入一条信息(Thisis a message: The you smile until forever),并且Ctrl+z退出shell
[root@rs229 kafka_2.9.2-0.8.1.1]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message: The you smile until forever
comsumer
# Start a consumer(开启一个消费者)
输入命令之后打印出一些信息,最后面显示了刚刚输入的信息:Thisis a message: The you smile until forever
[root@rs229 kafka_2.9.2-0.8.1.1]# bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
This is a message: The you smile until forever
四、集群
多个brocker 整目录拷贝多份就可以了
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-1.properties
新的配置
config/server-1.properties:
broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1
config/server-2.properties:
broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2
[ 注意:真正集群要设置host.name和advertised.host.name这两个属性 ]
启动
JMX_PORT=9997 bin/kafka-server-start.sh config/server-1.properties &
JMX_PORT=9998 bin/kafka-server-start.sh config/server-2.properties &
伪分布式集群启动要加:JMX_PORT=
五、java 客户端连接
消息生产者代码示例
import java.util.Collections;import java.util.Date;import java.util.Properties;import java.util.Random;import kafka.javaapi.producer.Producer;import kafka.producer.KeyedMessage;import kafka.producer.ProducerConfig;/** * 详细可以参考:https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example * @author Fung * */public class ProducerDemo { public static void main(String[] args) { Random rnd = new Random(); int events=100; // 设置配置属性 Properties props = new Properties(); props.put("metadata.broker.list","ip1:9092,ip2:9092,ip3:9092"); props.put("serializer.class", "kafka.serializer.StringEncoder"); // key.serializer.class默认为serializer.class props.put("key.serializer.class", "kafka.serializer.StringEncoder"); // 可选配置,如果不配置,则使用默认的partitioner props.put("partitioner.class", "com.catt.kafka.demo.PartitionerDemo"); // 触发acknowledgement机制,否则是fire and forget,可能会引起数据丢失 // 值为0,1,-1,可以参考 // http://kafka.apache.org/08/configuration.html props.put("request.required.acks", "1"); ProducerConfig config = new ProducerConfig(props); // 创建producer Producer<String, String> producer = new Producer<String, String>(config); // 产生并发送消息 long start=System.currentTimeMillis(); for (long i = 0; i < events; i++) { long runtime = new Date().getTime(); String ip = "192.168.2." + i;//rnd.nextInt(255); String msg = runtime + ",www.example.com," + ip; //如果topic不存在,则会自动创建,默认replication-factor为1,partitions为0 KeyedMessage<String, String> data = new KeyedMessage<String, String>( "mytopic", "hello kafka"); producer.send(data); } System.out.println("耗时:" + (System.currentTimeMillis() - start)); // 关闭producer producer.close(); }}
消息消费者代码示例
import java.util.HashMap;import java.util.List;import java.util.Map;import java.util.Properties; import kafka.consumer.ConsumerConfig;import kafka.consumer.ConsumerIterator;import kafka.consumer.KafkaStream;import kafka.javaapi.consumer.ConsumerConnector;import kafka.serializer.StringDecoder;import kafka.utils.VerifiableProperties;public class ConsumerDemo { private final ConsumerConnector consumer; private static ConsumerDemo ConsumerDemo; private ConsumerDemo() { Properties props = new Properties(); //zookeeper 配置 props.put("zookeeper.connect", KafkaProperties.zkConnect); //group props.put("group.id", "jd-group"); //zk连接超时 props.put("zookeeper.session.timeout.ms", KafkaProperties.zkSessionTimeOut); props.put("zookeeper.sync.time.ms", KafkaProperties.zkSyncTime); props.put("auto.commit.interval.ms", KafkaProperties.autoCommitInterval); props.put("auto.offset.reset", "smallest"); //序列化类 props.put("serializer.class", "kafka.serializer.StringEncoder"); ConsumerConfig config = new ConsumerConfig(props); consumer = kafka.consumer.Consumer.createJavaConsumerConnector(config); } public static ConsumerDemo getInstance(){ if(ConsumerDemo == null){ ConsumerDemo = new ConsumerDemo(); } return ConsumerDemo; } public void consume(String topic) { Map<String, Integer> topicCountMap = new HashMap<String, Integer>(); topicCountMap.put(topic, new Integer(1)); StringDecoder keyDecoder = new StringDecoder(new VerifiableProperties()); StringDecoder valueDecoder = new StringDecoder(new VerifiableProperties()); Map<String, List<KafkaStream<String, String>>> consumerMap = consumer.createMessageStreams(topicCountMap,keyDecoder,valueDecoder); KafkaStream<String, String> stream = consumerMap.get(topic).get(0); ConsumerIterator<String, String> it = stream.iterator(); while (it.hasNext()) System.out.println(it.next().message()); } public static void main(String[] args) { ConsumerDemo.getInstance().consume("mytopic"); }}
详细java api使用见:
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
[ 转载请写明本文地址 ]
- kafka_2.9.2-0.8.1.1安装、测试、集群
- kafka_2.9.2-0.8.1.1伪分布式集群安装
- kafka_2.9.1-0.8.1.1 集群的安装
- kafka_2.9.2-0.8.1.1安装
- Kafka安装配置(kafka_2.9.2-0.8.1.1)
- kafka_2.11-0.10.1.1集群搭建安装配置
- kafka_2.9.2-0.8.1.1 deploy部署
- Kafka_2.10-0.10.0.0安装与测试
- 基于zookeeper-3.4.5-cdh5.7.1的kafka_2.9.2-0.8.1.1的安装
- Kafka_2.10-0.10.0.0集群安装与配置
- kafka_2.12-0.10.2.1集群搭建安装配置
- 8--安装kafka_2.10-0.10.2.0集群(三个节点)
- kafka_2.11-1.0.0单机和集群的安装和配置
- _00022 Flume-1.5.0+Kafka_2.9.2-0.8.1.1+Storm-0.9.2 分布式环境整合
- Flume-1.5.0+Kafka_2.9.2-0.8.1.1+Storm-0.9.2 分布式环境整合
- Flume-1.5.0+Kafka_2.9.2-0.8.1.1+Storm-0.9.2 分布式环境
- MetaQ集群安装测试
- Codis 集群安装测试
- 一个简单的智能指针的实现
- 字符串函数
- 将url或者字符串转换成MD5格式(md5加密算法)的写法
- Swift 2.2 基础语法
- Problem C
- kafka_2.9.2-0.8.1.1安装、测试、集群
- MONGODB范围查询
- 【Spring】【笔记】《Spring In Action》第一章 读书笔记
- Android声明和使用权限
- 面向对象(二)
- 练习011-012
- Poj 2342 Anniversary party【入门树型dp】
- OSX PHP添加gettext支持
- day5.19总结_音乐播放器_页面更新和播放