利用kafka内建的特殊topic 做offsets 提交和检索
来源:互联网 发布:电脑扫描仪软件 编辑:程序博客网 时间:2024/06/05 09:43
In Kafka releases through 0.8.1.1, consumers commit their offsets to ZooKeeper. ZooKeeper does not scale extremely well (especially for writes) when there are a large number of offsets (i.e., consumer-count * partition-count
). Fortunately, Kafka now provides an ideal mechanism for storing consumer offsets. Consumers can commit their offsets in Kafka by writing them to a durable (replicated) and highly available topic. Consumers can fetch offsets by reading from this topic (although we provide an in-memory offsets cache for faster access). i.e., offset commits are regular producer requests (which are inexpensive) and offset fetches are fast memory look ups.
The official Kafka documentation describes how the feature works and how to migrate offsets from ZooKeeper to Kafka. This wiki provides sample code that shows how to use the new Kafka-based offset storage mechanism.
Step 1: Discover and connect to the offset manager for a consumer group by issuing a consumer metadata request to any broker
Step 2: Issue the OffsetCommitRequest or OffsetFetchRequest to the offset manager
- 利用kafka内建的特殊topic 做offsets 提交和检索
- 整合sparkstreaming和kafka,手动管理kafka的offsets(重点)
- 删除kafka 的topic
- Kafka的Topic操作
- 删除kafka的topic
- kafka删除和创建topic
- Kafka 学习 -- Topic的创建
- Kafka的topic限制是多少
- kafka topic的基本操作
- 如何删除Kafka的Topic
- kafka topic
- Kafka 学习笔记(2) - Topic 和 Log
- kafka修改topic分区的位置
- Kafka动态增加Topic的副本(Replication)
- 彻底删除kafka的topic方法
- kafka增加topic 的partitions数量
- 彻底删除kafka的topic方法
- HadoopConsumer——消费kafka中若干topic的消息,追加存储至hdfs的不同文件内
- Hbase安装
- 20160419 线性渐变
- android之baseAdapter实现listview里控件点击和左滑收藏(个人感觉已完美解决)
- SVM推导帖子收藏
- 2014
- 利用kafka内建的特殊topic 做offsets 提交和检索
- 【HDU】 1035 Robot Motion
- WCF服务怎么获得宿主的ip地址
- myeclipse10 下ssh框架搭建检测以及常见问题
- java 笔记
- 次方求模
- hdu——1556Color the ball(树状数组)
- 山东省第五届ACM大学生程序设计竞赛-angry_birds_again_and_again(积分)
- 面试题89:字符串数组首尾字符相同