Spark学习笔记(23)Transformation、Action等源码图解
来源:互联网 发布:linux编译比特币 编辑:程序博客网 时间:2024/06/05 20:27
本期内容:
1. Spark Streaming的Transformation、Action源码图解
2. Spark Streaming的Input、Output源码图解
StreamingContext成员:socketStream
/**
* Create a input stream from TCP source hostname:port. Data is received using
* a TCP socket and the receive bytes it interepreted as object using the given
* converter.
* @param hostname Hostname to connect to for receiving data
* @param port Port to connect to for receiving data
* @param converter Function to convert the byte stream to objects
* @param storageLevel Storage level to use for storing the received objects
* @tparam T Type of the objects received (after converting bytes to objects)
*/
def socketStream[T: ClassTag](
hostname: String,
port: Int,
converter: (InputStream) => Iterator[T],
storageLevel: StorageLevel
): ReceiverInputDStream[T] = {
new SocketInputDStream[T](this, hostname, port, converter, storageLevel)
}
SocketInputDStream:
class SocketInputDStream[T: ClassTag](
ssc_ : StreamingContext,
host: String,
port: Int,
bytesToObjects: InputStream => Iterator[T],
storageLevel: StorageLevel
) extends ReceiverInputDStream[T](ssc_) {
def getReceiver(): Receiver[T] = {
new SocketReceiver(host, port, bytesToObjects, storageLevel)
}
}
ReceiverInputDStream:
abstract class ReceiverInputDStream[T: ClassTag](ssc_ : StreamingContext)
extends InputDStream[T](ssc_) {
...
InputDStream:
ssc.graph.addInputStream(this)
把ImputDStream放入到了graph中。
DStream.foreachRDD产生ForEachDStream对象,该对象通过register对象也放入了graph中。
阅读全文
0 0
- Spark学习笔记(23)Transformation、Action等源码图解
- Spark学习笔记 --- Transformation and Action
- Spark定制班第24课:Spark Streaming的Transformation、Action、Input和Output源码图解
- spark源码之RDD(2)transformation和action
- 第24课:Spark Streaming的Transformation、Action、Input和Output源码图解
- Spark Streaming的Transformation、Action、Input和Output源码图解(第24课)
- Spark学习笔记(22)Spark Streaming架构源码图解
- Spark学习笔记(24)StreamingContext及JobScheduler源码图解
- 图解Spark Transformation算子
- 图解Spark Transformation算子
- Spark:Transformation和Action
- Spark: Transformation和Action
- Spark: Transformation和Action
- Spark Transformation和Action
- Spark学习笔记(25)Spark Streaming的JobGenerator源码图解
- spark中的action和transformation
- spark中的action和transformation
- spark中的action和transformation
- 搭建自己的Shadowsocks服务器
- java文件读写操作之io流
- Spark学习笔记(22)Spark Streaming架构源码图解
- 优先队列
- 序——为什么读史
- Spark学习笔记(23)Transformation、Action等源码图解
- Tomcat系列—基于端口的虚拟主机配置
- RecyclerView的简单使用
- 【2017年最全】 iOS面试题及答案
- MySQL B+树索引和Hash索引的区别
- Spark学习笔记(24)StreamingContext及JobScheduler源码图解
- ApplicationContext的事件机制
- spring boot入门(三) 热部署
- ubuntu16.04安装jdk以及详细jdk环境配置教程,亲测没毛病。