kafka consumer不能消费消息及其处理办法

我这里的Kafka Consumer代码是拷贝网上的，就是开一个线程监听kafka topic，一有消息就处理。开始的代码是这样的：

public void kafkaStart() {        final String topic = hipchatAction.properties.getProperty("kafka.hipchat.topic");        final int partitionNum = Integer.valueOf(hipchatAction.properties.getProperty("kafka.hipchat.topic.partitionNum"));        log.debug("Comes to kafkaStart() with topic : " + topic + ", partitionNum : " + partitionNum);        Map<String, Integer> topicCountMap = new HashMap<>();        topicCountMap.put(topic, partitionNum);        Map<String, List<KafkaStream<byte[], byte[]>>> streams = connector.createMessageStreams(topicCountMap);        List<KafkaStream<byte[], byte[]>> partitions = streams.get(topic);        ExecutorService executor = Executors.newFixedThreadPool(partitionNum * 2);        for (KafkaStream<byte[], byte[]> partition : partitions) {            /**             * Here also it is an unknown issue, if use anonymous inner class, then the thread seems died, must have             * a named inner class!             */            executor.execute(new MessageRunner(partition));        }    }    class MessageRunner implements Runnable {        private KafkaStream<byte[], byte[]> partition;        MessageRunner(KafkaStream<byte[], byte[]> partition) {            this.partition = partition;        }        public void run() {            ConsumerIterator<byte[], byte[]> it = partition.iterator();            while (it.hasNext()) {                MessageAndMetadata<byte[], byte[]> msgMeta = it.next();                /**                 * In order to consume Chinese message, here should use charset.                 */                String jsonStr = new String(msgMeta.message(), StandardCharsets.UTF_8);                log.debug("********* Message to be consumed in UTF-8 is :: " + jsonStr);                KafkaMsgVO msg = new Gson().fromJson(jsonStr, KafkaMsgVO.class);                hipchatAction.sendMessageToRoom(msg.getRoomName(), msg.getToken(), msg.getMsgText());            }        }    }

方法kafkaStart()会在Spring容器启动的时候执行。

对于 KafkaMsgVO msg = new Gson().fromJson(jsonStr, KafkaMsgVO.class); 这行，因为我的程序对消息要求是一个对象名为 KafkaMsgVO 的JSON格式数据，但是测试的时候测试人员就随便发了一条消息，没有进行对象的属性赋值并组装成 JSON 数据，所以抛出 JSONSyntaxException。问题就出在这，因为一旦抛出异常，这个线程就被破坏了，没办法进行后续消息的消费，尽管 kafka consumer 仍然能感知到 topic 里面有新的消息到来。

解决办法：

为了不让异常破坏 consumer 线程，我进行了出错代码的位置转移，将得到的消息直接通过 Apache AKKA 进行转发，然后由 AKKA 的 onReceive() 对消息进行处理，也就是出错代码移到 onReceive() 里面，这样就保证了 consumer 线程的健壮性。代码如下：

class MessageRunner implements Runnable {        private KafkaStream<byte[], byte[]> partition;        MessageRunner(KafkaStream<byte[], byte[]> partition) {            this.partition = partition;        }        public void run() {            ConsumerIterator<byte[], byte[]> it = partition.iterator();            while (it.hasNext()) {                MessageAndMetadata<byte[], byte[]> msgMeta = it.next();                /**                 * In order to consume Chinese message, here should use charset.                 */                String jsonStr = new String(msgMeta.message(), StandardCharsets.UTF_8);                log.debug("********* Message to be consumed in UTF-8 is :: " + jsonStr);                ActorRef sender = akkaSystem.getMsgConductor();                sender.tell(new AkkaAdaptor(jsonStr, hipchatAction), sender);            }        }    }

一旦消息被 AKKA 转发后，对消息的处理代码就放在了AKKA 的 Actor 里面了：

} else if (message instanceof AkkaAdaptor) {AkkaAdaptor akkaAdaptor = (AkkaAdaptor)message;String textMessage = akkaAdaptor.getTextMessage();KafkaMsgVO msg = null;try {msg = new Gson().fromJson(textMessage, KafkaMsgVO.class);} catch (Exception e) {log.debug(textMessage + " is malformed, it may miss some important property (value).");return;}HipchatAction hipchatAction = akkaAdaptor.getHipchatAction();log.debug("Kafka message sent by AKKA is :: " + msg.getMsgText());hipchatAction.sendMessageToRoom(msg.getRoomName(), msg.getToken(), msg.getMsgText());}