How-to: write own Kafka Partitioner based on requirement
来源:互联网 发布:网络推销员好做吗 编辑:程序博客网 时间:2024/05/18 01:28
Kafka's default partitioner is based on hash first element which is generated by spliting log via tab. In our usage, this is not a normal case. Our logs are normal log which the first element should be split by blank, and we hope the partitioner is based on the first word.
Then we could generate a new console producer class, and in the main class set the properties like following:
object OurConsoleProducer {
...
def main(args: Array[String]) {
val config = new ProducerConfig(args)
val reader = Class.forName(config.readerClass).newInstance().asInstanceOf[MessageReader]
val props = new Properties
...
props.put("key.separator", " ")
...
val config = new ProducerConfig(args)
val reader = Class.forName(config.readerClass).newInstance().asInstanceOf[MessageReader]
val props = new Properties
...
props.put("key.separator", " ")
...
props.put("partitioner.class", "OurPartitioner")
...
In the partitioner class, we are going to put log started with some word to some spacified partitioner.
/*
* p-01:1(log started with p-01(splited by blank) will be stored in partitioner 1)
* p-01:1(log started with p-01(splited by blank) will be stored in partitioner 1)
* p-02:2
* p-03:3
* p-04:4
* p-05:5
* p-06:6
* p-07:7
* Otheres:0
*/
package kafka.producer
import kafka.utils._
class OurPartitioner(props: VerifiableProperties = null) extends Partitioner {
private val random = new java.util.Random
def partition(key: Any, numPartitions: Int): Int = {
// Utils.abs(key.hashCode) % numPartitions
if(numPartitions != 8){
System.err.print("Warning: the partition number should be 7 while is "+Int)
}
val a = key.asInstanceOf[Array[Byte]].mkString(" ")
if(a.equals("102 105 114 115 116 112 50 112 45 48 49")){//p-01 ascii code
System.out.println("1")
1
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 50")){//p-02 ascii code
2
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 51")){//p-03 ascii code
3
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 52")){//p-04 ascii code
4
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 53")){//p-05 ascii code
5
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 54")){//p-06 ascii code
6
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 55")){//p-07 ascii code
7
}else{
0
}
}
}
*/
package kafka.producer
import kafka.utils._
class OurPartitioner(props: VerifiableProperties = null) extends Partitioner {
private val random = new java.util.Random
def partition(key: Any, numPartitions: Int): Int = {
// Utils.abs(key.hashCode) % numPartitions
if(numPartitions != 8){
System.err.print("Warning: the partition number should be 7 while is "+Int)
}
val a = key.asInstanceOf[Array[Byte]].mkString(" ")
if(a.equals("102 105 114 115 116 112 50 112 45 48 49")){//p-01 ascii code
System.out.println("1")
1
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 50")){//p-02 ascii code
2
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 51")){//p-03 ascii code
3
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 52")){//p-04 ascii code
4
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 53")){//p-05 ascii code
5
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 54")){//p-06 ascii code
6
}else if(a.equals("102 105 114 115 116 112 50 112 45 48 55")){//p-07 ascii code
7
}else{
0
}
}
}
The console shell should be like:
>> cat our-console-producer.sh
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
export KAFKA_HEAP_OPTS="-Xmx512M"
fi
exec $(dirname $0)/kafka-run-class.sh kafka.tools.OurConsoleProducer $@
Then create our-topic with 8 partitioner, and transfer our logs to our-console-producer.sh via pipleline.
0 0
- How-to: write own Kafka Partitioner based on requirement
- How to write your own custom Form
- How to learn on your own
- how to write one native program based on android source build system
- How To Write Your Own Ebook(r) In 7 Days!
- How to Write Your Own Linux Kernel Module with a Simple Example
- How to do reasoning/inference based on Jena
- Generic Netlink HOW-TO based on Jamal's original doc
- how to list tabes based on column name in sybase?
- QT Creator based on VS2010 OpenGL-----How to config compiler
- Building your own kernel based on CentOS
- Write my own UserManager on Android
- how to write and run jpa idempotent on mysql
- How To Install Apache Kafka on Ubuntu 14.04
- How To Install Apache Kafka on Ubuntu 14.04
- How to write Bibliography
- how to write makefile
- How to write makefile
- Universal Serial Bus 软件结构
- Android Studio系列教程六--Gradle多渠道打包
- android spinner(下拉选项卡)的使用
- 协议(delegate)和类别(category)
- FPGA 练习
- How-to: write own Kafka Partitioner based on requirement
- 小知识点
- 获取手机信息
- 产生随机的数组
- POJ 1018(二分)
- Android 操作系统的内存回收机制
- C#委托、事件学习之(三)——热水器烧水案例
- Apache2 虚拟主机配置
- Android Activity 销毁后线程会不会被GC回收?