An experience of fixing a memory-corruption bug (1)

During the last 4 months, I was disturbed by a memory-corruption bug, and this bug will cause program crash. Until last Monday, I found the root cause and fixed it. This debug process is a difficult but memorable experience, so I will share it in this article.

My program works as a SMS Hub. When it receives a SMS, it will allocate a structure in heap memory like this:

After processing the SMS, the program will free the memory, and send the SMS to the next Hub or Operator. 

Since last November, the program will crash sometimes, and the cause is the second element in array a (a[1]) will be changed from a valid value to NULL.

(1)  Checking the logs and reproduced the bug 

Firstly, I checked the commercial logs, but there were no clues can be found. And the SMS which killed program also seemed no different than others. I also tried to use this SMS to reproduce the bug in testbed, but also failed.

(2) Using libumem

Because our program runs in Solaris, I linked the libumem and hoped it can help me. After a few days, the program crashed again. But the tags before and after the corrupted memory are all OK, so it is not a memory off-bound bug. I also checked the memory before and after the corrupted memory, but nothing valuable can be found.

(3) Adding more logs

Until then, the only thing I can think is adding more logs. After adding enough logs, I found the variable is modified between functions in the same thread: when leaving the last function, the variable is OK, but entering the next function, the variable is changed. So I can make sure the variable is destroyed by another thread. But how can I find the murderer?