STORM入门之(TridentAPI,Aggregation)
来源:互联网 发布:cf刷枪软件视频教程 编辑:程序博客网 时间:2024/06/07 06:14
基本介绍
Aggregation是Trident的基本基本api 主要作用是聚合,如下聚合方法作用是记录单词出现的次数
package com.storm.trident;import org.apache.storm.shade.org.apache.commons.exec.util.MapUtils;import org.apache.storm.trident.operation.BaseAggregator;import org.apache.storm.trident.operation.TridentCollector;import org.apache.storm.trident.tuple.TridentTuple;import org.apache.storm.tuple.Values;import java.util.HashMap;import java.util.Map;public class WordAggregat extends BaseAggregator<Map<String, Integer>> { public static Map<String, Integer> map = new HashMap<String, Integer>(); @Override public Map<String, Integer> init(Object batchId, TridentCollector collector) { return new HashMap<String, Integer>(); } @Override public void aggregate(Map<String, Integer> val, TridentTuple tuple,TridentCollector collector) { String location = tuple.getString(0); Integer i = map.get(location); if(null == i){ i = 0; }else{ i = i+1; } map.put(location, i); } @Override public void complete(Map<String, Integer> val, TridentCollector collector) { for (String key : map.keySet()) { System.out.println("key= "+ key + " and value= " + map.get(key)); } collector.emit(new Values(map)); }}
(2)aggregate循环执行
(3)complet结束循环执行
Trident构建
package com.storm.topology;import com.storm.spout.FixedBatchSpout;import com.storm.trident.Split;import com.storm.trident.WordAggregat;import com.storm.trident.WordFilter;import org.apache.storm.Config;import org.apache.storm.LocalCluster;import org.apache.storm.generated.StormTopology;import org.apache.storm.redis.common.config.JedisClusterConfig;import org.apache.storm.redis.trident.state.RedisClusterState;import org.apache.storm.trident.TridentTopology;import org.apache.storm.trident.operation.builtin.Count;import org.apache.storm.tuple.Fields;import org.apache.storm.tuple.Values;import java.net.InetSocketAddress;import java.util.HashSet;import java.util.Set;public class TTopology { public static void main(String[] args){ TridentTopology topology = new TridentTopology(); FixedBatchSpout spout = new FixedBatchSpout(new Fields("sentence"), 1, new Values("the cow jumped over the moon"), new Values("the man went to the store and bought some candy"), new Values("four score and seven years ago"), new Values("how many apples can you eat")); spout.setCycle(true); topology.newStream("batch-spout",spout).parallelismHint(2) .each(new Fields("sentence"), new Split(), new Fields("word")).project(new Fields("word")) //project方法只发送word字段 .aggregate(new Fields("word"),new WordAggregat(),new Fields("agg")); StormTopology stormTopology = topology.build(); LocalCluster cluster = new LocalCluster(); Config conf = new Config(); conf.setDebug(true); cluster.submitTopology("soc", conf,stormTopology); } }
结果
出现单词的次数
阅读全文
0 0
- STORM入门之(TridentAPI,Aggregation)
- STORM入门之(TridentAPI,Each)
- STORM入门之(TridentAPI,partition)
- STORM入门之(基本Shell命令)
- STORM入门之(Topology简易Demo)
- STORM入门之(集成KafkaBolt)
- STORM入门之(集成KafkaSpout)
- STORM入门之(集成ElasticSearch)
- STORM入门之(集成Redis)
- STORM入门之(TridentTopology集成Kafka)
- STORM入门之(yaml配置文件说明)
- STORM入门之(集成HDFS)
- Storm入门之第一章
- Storm入门之第一章
- Storm入门之第一章
- Storm入门之第一章
- Mongo入门:数据去重之MapReduce,Aggregation的简单使用(另附相关网络资源)
- Storm入门之附录A
- K:栈和队列的比较
- Linux下时间不同步解决
- 一、概述
- Windows下Anaconda2(Python2)和Anaconda3(Python3)的共存
- 2017-8-31
- STORM入门之(TridentAPI,Aggregation)
- [笔记分享] [SCons] Qualcomm SCons Overview
- 如何将本地项目上传至GitHub
- html5 视频播放及border
- 什么是僵尸进程?
- java反射
- jdbc存储过程
- Android混淆配置
- scala中xml修改元素详解