Flume 读取JMS 消息队列消息,并将消息写入HDFS
来源:互联网 发布:淘宝免费转微信链接 编辑:程序博客网 时间:2024/06/04 19:53
利用Apache Flume 读取JMS 消息队列消息,并将消息写入HDFS,flume agent配置如下:
flume-agent.conf
#name the components on this agent
agentHdfs.sources = jms_source
agentHdfs.sinks = hdfs_sink
agentHdfs.channels = mem_channel
# Describe/configure the source
agentHdfs.sources.jms_source.type = jms
# Bind to all interfaces
agentHdfs.sources.jms_source.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
agentHdfs.sources.jms_source.connectionFactory = ConnectionFactory
agentHdfs.sources.jms_source.destinationName = BUSINESS_DATA #AMQ queue
agentHdfs.sources.jms_source.providerURL = tcp://hadoop-master:61616
agentHdfs.sources.jms_source.destinationType = QUEUE
# Describe the sink
agentHdfs.sinks.hdfs_sink.type = hdfs
agentHdfs.sinks.hdfs_sink.hdfs.path hdfs://hadoop-master/data/flume/%Y-%m-%d/%H
agentHdfs.sinks.hdfs_sink.hdfs.filePrefix = %{hostname}/events-
agentHdfs.sinks.hdfs_sink.hdfs.maxOpenFiles = 5000
agentHdfs.sinks.hdfs_sink.hdfs.batchSize= 500
agentHdfs.sinks.hdfs_sink.hdfs.fileType = DataStream
agentHdfs.sinks.hdfs_sink.hdfs.writeFormat =Text
agentHdfs.sinks.hdfs_sink.hdfs.rollSize = 0
agentHdfs.sinks.hdfs_sink.hdfs.rollCount = 1000000
agentHdfs.sinks.hdfs_sink.hdfs.rollInterval = 600
agentHdfs.sinks.hdfs_sink.hdfs.useLocalTimeStamp = true
# Use a channel which buffers events in memory
agentHdfs.channels.mem_channel.type = memory
agentHdfs.channels.mem_channel.capacity = 1000
agentHdfs.channels.mem_channel.transactionCapacity = 100
# Bind the source and sink to the channel
agentHdfs.sources.jms_source.channels = mem_channel
agentHdfs.sinks.hdfs_sink.channel = mem_channel
- Flume 读取JMS 消息队列消息,并将消息写入HDFS
- JMS消息队列
- 消息队列JMS
- jms-activemq消息队列
- JMS消息队列入门
- 消息队列与JMS
- JMS消息队列
- flume实时接收kafka消息并保存至HDFS
- spark读取 kafka nginx网站日志消息 并写入HDFS中
- spark读取 kafka nginx网站日志消息 并写入HDFS中
- spark读取kafka nginx网站日志消息 并写入HDFS中
- 消息队列之JMS消息服务讲解
- JMS之消息队列 ActiveMQ
- JMS消息队列之ActiveMQ
- Java消息队列--JMS概述
- Java消息队列--JMS概述
- Java JMS消息队列技术
- Java消息队列--JMS概述
- 多任务的同步与互斥
- python对象数据的读写权限
- linux caffe 64bit
- VS2008向工具箱中添加控件解决 未能实例化 设计时授权
- 正则表达式30分钟入门教程
- Flume 读取JMS 消息队列消息,并将消息写入HDFS
- java 数组:java.lang.ClassCastException
- Apache性能监控
- Android多行gridview横向滑动的实现
- myeclipse 6.5 注册机 MyEclipseGen
- objective-c中类的申明、实现、访问
- jQuery学习二
- 文章标题
- 《Java并发编程的艺术》作者方腾飞:感谢Java,带我一路前行!