Active MQ KahaDB Invalid location

0. Environment
OS: CentOS 5.4
Active MQ: 5.5.1
Active MQ configuration: 

</pre><p><pre name="code" class="html"><persistenceAdapter>    <kahaDB directory="${activemq.base}/data/kahadb"/></persistenceAdapter>

 <transportConnectors>            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>        </transportConnectors>

1. Problem Description

Pending messages in queue cannot be consumed and we can't delete data in Active MQ administration page.

In our production back end, we encountered this exception:

2015-01-19 12:39:43,813 | ERROR | QueueStorePrefetch383867497 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Schedulerjava.lang.RuntimeException: java.io.IOException: Invalid location: 542164:30466873, : java.lang.NegativeArraySizeExceptionat org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1028)at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:785)at org.apache.activemq.broker.region.Queue.access$100(Queue.java:84)at org.apache.activemq.broker.region.Queue$2.run(Queue.java:124)at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)at java.util.TimerThread.mainLoop(Timer.java:512)at java.util.TimerThread.run(Timer.java:462)Caused by: java.io.IOException: Invalid location: 542164:30466873, : java.lang.NegativeArraySizeExceptionat org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:94)at org.apache.kahadb.journal.Journal.read(Journal.java:601)at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)... 9 more2015-01-19 12:39:43,813 | ERROR | Problem retrieving message for browse | org.apache.activemq.broker.region.Queue | ActiveMQ Broker[localhost] Schedulerjava.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Invalid location: 542164:30466873, : java.lang.NegativeArraySizeExceptionat org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:111)at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1028)at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:785)at org.apache.activemq.broker.region.Queue.access$100(Queue.java:84)at org.apache.activemq.broker.region.Queue$2.run(Queue.java:124)at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)at java.util.TimerThread.mainLoop(Timer.java:512)at java.util.TimerThread.run(Timer.java:462)Caused by: java.lang.RuntimeException: java.io.IOException: Invalid location: 542164:30466873, : java.lang.NegativeArraySizeExceptionat org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:263)at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:108)... 8 moreCaused by: java.io.IOException: Invalid location: 542164:30466873, : java.lang.NegativeArraySizeExceptionat org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:94)at org.apache.kahadb.journal.Journal.read(Journal.java:601)at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:817)at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:260)... 9 more

2. Analysis

3. Workaround

Good suggestion from Reference 3:

ActiveMQ v5.5.1 Corrupt data log found recovery: Symptom: The ActiveMQ slave process died and will not restarted. Root Cause: Corrupt data log found Root Cause verification: Search the affected ActiveMQ log file for the following entries in sequence: Corrupt journal records found Failed to discard data file Failed to start ActiveMQ JMS Message Broker shutting down Recovery: Shutdown the ActiveMQ master instance. Rename the Kaha db storage file Restart the ActiveMQ Master and Slave instances Note: the journal data affected by the corruption will be lost. The affected journal data will need to be identify and resent to the ActiveMQ appropriate queue.

4. Solution

Reference:

[1] http://activemq.apache.org/message-cursors.html

[2] https://issues.apache.org/jira/browse/AMQ-4943

[3] http://activemq.2283324.n4.nabble.com/KahaDB-corruption-td3321382.html