Kafka and Samza: Real-time stream processing
来源:互联网 发布:长春淘宝客服工资多少 编辑:程序博客网 时间:2024/05/21 09:42
As we known, for big data analysis, we have those two already learned[1]:
Batch Processing is map-reduce. And Iterative Processing is Spark. These two have one thing in common which is what they are processing is a fixed data. Once the processing job starts, you cannot change the input data at all. This gives some disadvantage for real time data analysis.
Now, for real time analysis, we introduce stream processing. Here is a concept of stream processing[1]:
In our situation of Kafka + Samza, Samza is the processing framework. Kafka only is a source of organising stream as topics and messages. Now, let's take a look of the details.
Here is some concepts in Kafka:
Here are some basic concepts about Samza:
NM = Node Manager; RM = Resource Manager.
Here is a typical job of Samza:
In general, one task in Samza is one consumer in Kafka. One stream in the input streams is one partition of topic in kafka.
Reference:
[1] 15619 Cloud Computing CMU
- Kafka and Samza: Real-time stream processing
- Data-Intensive Systems:Real-time Stream Processing
- Real-time stream processing: The next step for Apache Flink
- Flume or Kafka for Real-Time Event Processing
- Introducing Kafka Streams: Stream Processing Made Simple
- Real-time model scoring for streaming data – a prototype based on Oracle Stream Explorer and Oracle
- A (quasi-) real-time video processing on iOS
- A (quasi-) real-time video processing on iOS
- Real-Time Rendering-第三章 The Graphics Processing Unit
- Samza/Kafka机理解析
- UNIX环境中Real time, User time and Sys time
- UNIX环境中Real time, User time and Sys time
- UNIX环境中Real time, User time and Sys time
- UNIX环境中Real time, User time and Sys time
- RTSP(Real Time Stream Protocol,实时流协议)
- Real Time Messaging Protocol Chunk Stream(RTMP数据块流)
- Real-Time Applications with Java and CORBA
- Linux for Embedded and Real-Time Applications
- 设计模式经典书籍推荐
- linux命令大全
- 笔试题目——编程题
- happens-before规则
- MySQL批量SQL插入各种性能优化
- Kafka and Samza: Real-time stream processing
- 关于爱情
- Process 'command '**/ndk-build.cmd'' finished with non-zero exit value 1
- centos6.5 安装docker
- svn错误:Can't convert string from 'UTF-8' to native encoding
- linux下的文件io编程实现文件的拷贝
- [bzoj2154]crash的数字表格 解题报告
- java实现两个有序数组的合并
- 详解Android线程池ThreadPoolExecutor的教程