lmax.disruptor高效内存消息队列spring整合

lmax.disruptor高效内存消息队列spring整合

lmax.distruptor是一种高效的本地内存队列。特性就在此不啰嗦了，需要请自行百度。本文要解决的，是一种以distruptor为基础的快速开发模版。

概念

为了有效的设计开发模板，必须先弄清楚distruptor的几个核心概念，才能有效的下手：
1. RingBuffer
distruptor的核心概念-环形缓冲区，用于存储消息对象。

2.WorkerPool
distruptor的消费者线程池，支持多线程消费消息对象。

3. WorkHandler
事件消费者，在消费者里填写你自己的代码，用于消费

目标

先抛开代码不说，想想我们希望以一个什么模式解决问题：

1.有一些消息，希望分发到我们的队列里
2.我要知道队列容量有多大
3.我要知道有几个线程来消费这些消息，它们是怎样处理的

OK，目标确定了，我们模板的核心代码都是围绕这些来进行动作处理的，与之无关的动作都应该模版化

如何来用

先不看框架代码，根据我们的目标看看最终能够达到效果的步骤，以一个短信分发消息为例：

step 1 定义消息结构

设计我们要分发的消息。消息结构如下

/** * 短信关键数据. * @author JIM * */public class SmsParam {    /**     * 模板参数     */    private Map<String, String> paramMap = new HashMap<String, String>();    /**     * 接收人手机号     */    private String receiverMobile;    /**     * 短信模版     */    private String msgTemplate;    public String getReceiverMobile() {        return receiverMobile;    }    public void setReceiverMobile(String receiverMobile) {        this.receiverMobile = receiverMobile;    }    public String getMsgTemplate() {        return msgTemplate;    }    public void setMsgTemplate(String msgTemplate) {        this.msgTemplate = msgTemplate;    }    public Map<String, String> getParamMap() {        return paramMap;    }    public void setParamMap(Map<String, String> paramMap) {        this.paramMap = paramMap;    }    @Override    public String toString() {        return "SmsParam [paramMap=" + paramMap + ", receiverMobile=" + receiverMobile + ", msgTemplate=" + msgTemplate                + "]";    }}

step 2 定义Rubuffer消息

给消息定义一个包装器，该包装器为RingBuffer容器的队列对象，只需要简单继承即可：

public class SmsParamEvent extends ValueWrapper<SmsParam>{}

step 3 收到消息要做什么

编写我们收到消息后要处理的Handler事件代码：

/** * 发送短信并写数据库工作任务. * @author JIM * */public class SmsParamEventHandler implements WorkHandler<SmsParamEvent>{    @Override    public void onEvent(SmsParamEvent event) throws Exception {        try {            SmsMsgService smsMsgService = SpringContextHelper.getBean(SmsMsgService.class);            ISmsAdapter smsAdapter = smsMsgService.getSmsAdapter();            SmsParam smsParam = event.getValue();            //使用注入的短信处理器发送短信，此处省略若干字        }catch(Throwable tr) {            tr.printStackTrace();        }    }}

step 4 准备好Ringbuffer和工作线程

定义了初始化大小、工作线程数量（默认是1）、以及在bean初始化结束时准备好线程。并关联好SmsParamEvent及线程处理动作SmsParamEventHandler。默认的线程等待策略是BlockingWaitStrategy，如果你想极高的响应把CPU占用率弄到50%以上允许覆盖getStrategy使用YieldingWaitStrategy。

/** * 短信队列. * @author JIM * */@Componentpublic class SmsParamEventQueueHelper extends BaseQueueHelper<SmsParam, SmsParamEvent, SmsParamEventHandler> implements InitializingBean{    private static final int QUEUE_SIZE = 1024;    private int threadNum;    @Value("${app.sms.smsMsgEventQueue.threadNum:1}")    public void setThreadNum(int threadNum) {        this.threadNum = threadNum;    }    @Override    protected int getThreadNum() {        return threadNum;    }    @Override    protected int getQueueSize() {        return QUEUE_SIZE;    }    @Override    protected Class<SmsParamEvent> eventClass() {        return SmsParamEvent.class;    }    @Override    protected Class<SmsParamEventHandler> eventHandlerClass() {        return SmsParamEventHandler.class;    }    @Override    public void afterPropertiesSet() throws Exception {        this.init();    }}

step 5 在需要的地方发送消息

发送消息

SmsParam smsParam = new SmsParam();smsParam.putParam("code", validateCode);smsParam.setReceiverMobile(userMobile);            SpringContextHelper.getBean(SmsParamEventQueueHelper.class).publishEvent(smsParam);

---

不拖泥带水，需要一个队列分发扩展，动作就这么多。

相关的代码片段

全局的spring容器工具，凑字数用：

import org.apache.commons.logging.Log;import org.apache.commons.logging.LogFactory;import org.springframework.context.ApplicationContext;import org.springframework.context.ApplicationContextAware;import org.springframework.stereotype.Component;import java.util.HashMap;import java.util.Map;@Componentpublic class SpringContextHelper implements ApplicationContextAware {    private static Log log = LogFactory.getLog(SpringContextHelper.class);    private static ApplicationContext applicationContext;    @SuppressWarnings("all")    public void setApplicationContext(ApplicationContext context) {        if (this.applicationContext != null) {            log.error("ApplicationContextHolder already holded 'applicationContext'.");        }        this.applicationContext = context;        log.debug("holded applicationContext,displayName:" + applicationContext.getDisplayName());    }    public static ApplicationContext getApplicationContext() {        if (applicationContext == null) {            throw new IllegalStateException("'applicationContext' property is null,ApplicationContextHolder not yet init.");        }        return applicationContext;    }    private static Map<Class<?>,Object> beans = new HashMap<Class<?>, Object>();    public static <T> T getBean(Class<T> requiredType) {        if(beans.get(requiredType)!=null){            return (T)beans.get(requiredType);        } else {            Object instance = getApplicationContext().getBean(requiredType);            beans.put(requiredType,instance);            return (T)instance;        }    }    public static Object getBean(String beanName) {        return getApplicationContext().getBean(beanName);    }    public static boolean containsBean(String beanName) {        return getApplicationContext().containsBean(beanName);    }    public static <T> Map<String, T> getBeansOfType(Class<T> requiredType){        return getApplicationContext().getBeansOfType(requiredType);    }    public static void cleanHolder() {        applicationContext = null;    }}

消息包装抽象类

public abstract class ValueWrapper<T> {    private T value;    public ValueWrapper() {}    public ValueWrapper(T value) {        this.value = value;    }    public T getValue() {        return value;    }    public void setValue(T value) {        this.value = value;    }}

队列处理Helper类

/** * lmax.disruptor 高效队列处理模板. * 支持初始队列，即在init()前进行发布。 *  * 调用init()时才真正启动线程开始处理 * 系统退出自动清理资源. *  * @author JIM * * @param <D> 消息类 * @param <E> 消息包装类 * @param <H> 消息处理类 */public abstract class BaseQueueHelper<D, E extends ValueWrapper<D>, H extends WorkHandler<E>> {    private Disruptor<E> disruptor;    private RingBuffer<E> ringBuffer;    private List<D> initQueue = new ArrayList<D>();    /**     * 需要多少线程来消耗队列.     * @return     */    protected abstract int getThreadNum();    /**     * @return 队列长度，必须是2的幂     */    protected abstract int getQueueSize();    /**     * @return     */    protected abstract Class<E> eventClass();    protected abstract Class<H> eventHandlerClass();    //记录所有的队列，系统退出时统一清理资源    private static List<BaseQueueHelper> queueHelperList = new ArrayList<BaseQueueHelper>();    @SuppressWarnings("unchecked")    public void init() {        disruptor = new Disruptor<E>(new EventFactory<E>(){            @Override            public E newInstance() {                try {                    return (E)eventClass().newInstance();                } catch (Exception e) {                    e.printStackTrace();                    return null;                }            }        }, getQueueSize(), DaemonThreadFactory.INSTANCE, ProducerType.SINGLE, getStrategy());        H[] eventHandlers = (H[])Array.newInstance(eventHandlerClass(),getThreadNum());        for(int i = 0 ;i < getThreadNum(); i ++) {            try {                eventHandlers[i] = (H)eventHandlerClass().newInstance();            } catch (Exception e) {                e.printStackTrace();            }        }        disruptor.handleEventsWithWorkerPool(eventHandlers);        ringBuffer = disruptor.start();        for(D data: initQueue) {            ringBuffer.publishEvent(new EventTranslatorOneArg<E, D>(){                @Override                public void translateTo(E event, long sequence, D data) {                    event.setValue(data);                }            }, data);        }        //加入资源清理钩子        synchronized(queueHelperList) {            if(queueHelperList.isEmpty()) {                Runtime.getRuntime().addShutdownHook(new Thread() {                    @Override                    public void run() {                        for(BaseQueueHelper baseQueueHelper: queueHelperList) {                            baseQueueHelper.shutdown();                        }                    }                });            }            queueHelperList.add(this);        }    }    /**     * 如果要改变线程执行优先级，override此策略.     * YieldingWaitStrategy会提高响应并在闲时占用70%以上CPU，慎用     * SleepingWaitStrategy会降低响应更减少CPU占用，用于日志等场景.     * @return     */    protected WaitStrategy getStrategy() {        return new BlockingWaitStrategy();    }    /**     * 插入队列消息，支持在对象init前插入队列，则在队列建立时立即发布到队列处理.     * @param data     */    public synchronized void publishEvent(D data) {        if(ringBuffer == null) {            initQueue.add(data);            return;        }        ringBuffer.publishEvent(new EventTranslatorOneArg<E, D>(){            @Override            public void translateTo(E event, long sequence, D data) {                event.setValue(data);            }        }, data);    }    public void shutdown() {        disruptor.shutdown();    }}

最后买个关子，eventClass()和eventHandlerClass()还可以进一步简化掉，有兴趣深入的可以想想怎么做。

【完】

---