Kestrel and Storm
来源:互联网 发布:数据总表按字段拆分 编辑:程序博客网 时间:2024/04/30 20:34
This page explains how to use to Storm to consume items from a Kestrel cluster.
Preliminaries
Storm
This tutorial uses examples from the storm-kestrel project and the storm-starter project. It's recommended that you clone those projects and follow along with the examples. ReadSetting up development environment and Creating a new Storm project to get your machine set up.
Kestrel
It assumes you are able to run locally a Kestrel server as described here.
Kestrel Server and Queue
A single kestrel server has a set of queues. A Kestrel queue is a very simple message queue that runs on the JVM and uses the memcache protocol (with some extensions) to talk to clients. For details, look at the implementation of theKestrelClient class provided in storm-kestrel project.
Each queue is strictly ordered following the FIFO (first in, first out) principle. To keep up with performance items are cached in system memory; though, only the first 128MB is kept in memory. When stopping the server, the queue state is stored in a journal file.
Further, details can be found here.
Kestrel is:
- fast
- small
- durable
- reliable
For instance, Twitter uses Kestrel as the backbone of its messaging infrastructure as describedhere.
Add items to Kestrel
At first, we need to have a program that can add items to a Kestrel queue. The following method takes benefit of the KestrelClient implementation instorm-kestrel. It adds sentences into a Kestrel queue randomly chosen out of an array that holds five possible sentences.
private static void queueSentenceItems(KestrelClient kestrelClient, String queueName)throws ParseError, IOException {String[] sentences = new String[] { "the cow jumped over the moon", "an apple a day keeps the doctor away", "four score and seven years ago", "snow white and the seven dwarfs", "i am at two with nature"};Random _rand = new Random();for(int i=1; i<=10; i++){String sentence = sentences[_rand.nextInt(sentences.length)];String val = "ID " + i + " " + sentence;boolean queueSucess = kestrelClient.queue(queueName, val);System.out.println("queueSucess=" +queueSucess+ " [" + val +"]");}}
Remove items from Kestrel
This method dequeues items from a queue without removing them.
private static void dequeueItems(KestrelClient kestrelClient, String queueName) throws IOException, ParseError {for(int i=1; i<=12; i++){Item item = kestrelClient.dequeue(queueName);if(item==null){System.out.println("The queue (" + queueName + ") contains no items.");}else{byte[] data = item._data;String receivedVal = new String(data);System.out.println("receivedItem=" + receivedVal);}}
This method dequeues items from a queue and then removes them.
private static void dequeueAndRemoveItems(KestrelClient kestrelClient, String queueName) throws IOException, ParseError {for(int i=1; i<=12; i++){Item item = kestrelClient.dequeue(queueName);if(item==null){System.out.println("The queue (" + queueName + ") contains no items.");}else{int itemID = item._id;byte[] data = item._data;String receivedVal = new String(data);kestrelClient.ack(queueName, itemID);System.out.println("receivedItem=" + receivedVal);}}}
Add Items continuously to Kestrel
This is our final program to run in order to add continuously sentence items to a queue calledsentence_queue of a locally running Kestrel server.
In order to stop it type a closing bracket char ']' in console and hit 'Enter'.
import java.io.IOException; import java.io.InputStream; import java.util.Random; import backtype.storm.spout.KestrelClient; import backtype.storm.spout.KestrelClient.Item; import backtype.storm.spout.KestrelClient.ParseError; public class AddSentenceItemsToKestrel { /** * @param args */ public static void main(String[] args) { InputStream is = System.in; char closing_bracket = ']';int val = closing_bracket;boolean aux = true;try {KestrelClient kestrelClient = null;String queueName = "sentence_queue";while(aux){kestrelClient = new KestrelClient("localhost",22133);queueSentenceItems(kestrelClient, queueName);kestrelClient.close();Thread.sleep(1000);if(is.available()>0){ if(val==is.read()) aux=false;}}} catch (IOException e) {// TODO Auto-generated catch blocke.printStackTrace();}catch (ParseError e) {// TODO Auto-generated catch blocke.printStackTrace();} catch (InterruptedException e) {// TODO Auto-generated catch blocke.printStackTrace();}System.out.println("end"); }}
Using KestrelSpout
This topology reads sentences off of a Kestrel queue using KestrelSpout, splits the sentences into its constituent words (Bolt: SplitSentence), and then emits for each word the number of times it has seen that word before (Bolt: WordCount). How data is processed is described in detail in Guaranteeing message processing.
TopologyBuilder builder = new TopologyBuilder(); builder.setSpout("sentences", new KestrelSpout("localhost",22133,"sentence_queue",new StringScheme())); builder.setBolt("split", new SplitSentence(), 10) .shuffleGrouping("sentences"); builder.setBolt("count", new WordCount(), 20) .fieldsGrouping("split", new Fields("word"));
Execution
At first, start your local kestrel server in production or development mode.
Than, wait about 5 seconds in order to avoid a ConnectException.
Now execute the program to add items to the queue and launch the Storm topology. The order in which you launch the programs is of no importance.
If you run the topology with TOPOLOGY_DEBUG you should see tuples being emitted in the topology.
- Kestrel and Storm
- storm+kestrel+zookeeper 部署
- storm+kestrel+zookeeper
- storm and spark
- Storm and Heron
- flume-kestrel
- 征服 Kestrel
- Comparing Hadoop, Spark, and Storm
- kestrel : twitter queue
- HTTPSQS &jafka & kestrel & beanstalkd
- kestrel消息队列介绍
- 征服 Kestrel<转>
- kestrel消息队列介绍
- kestrel源代码阅读
- A working guide to kestrel
- Kestrel持久化队列服务器
- STORM
- storm
- Suse Linux下内存使用率精细解读
- menuconfig---Unable to find the Ncurses libraries问题解决
- oracle数据库恢复方法及ORA-01991错误的解决
- 手机号码信息的获取
- jms activeMQ与spring的集成
- Kestrel and Storm
- 链表(创建,插入,删除和打印输出)
- CIRCOS教程翻译 4.6——Label format
- 常见错误 集
- MFC 灾难性故障
- 【设计模式】bridge模式
- 两分钟学会在GitHub托管代码
- 剑指offer面试题33把数组排成最小的数,就是确定sort规则就可以了
- java 调用webservice的各种方法总结