TokuDB Slave Crash

MySQL版本：

Server version:         5.6.24-72.2-log Percona Server (GPL), Release 72.2, Revision 8d0f85b

现象：

zabbix用的库，用了TokuDB引擎的分区表，利用主库的备份拉起一套从库后。发现mysql启动后，是可以正常读写的。单独start slave io_thread 没问题。 

只要 start slave; 或 start slave sql_thread; mysql必定重启。

解决过程：

初看日志中的错误，一头雾水，找不到头绪。

尝试了重装，调整参数配置，重灌数据等方法，均未解决。

看来还是没找到问题的根源。

仔细一想，既然是start slave才能导致重启，必然是跟slave 的运行机制有关。

于是搜索 slave crash，找到了淘宝的一篇分析slave crash的文章。
http://mysql.taobao.org/monthly/2015/10/05/


引用其中的一段话：
start slave thread 在等待 sql thread 的过程中没有释放其占用的锁资源；
sql thread 在初始化 worker 信息时没有处理获取信息失败的情况。


start slave 为什么会访问 slave_worker_info 且一直持有锁资源?
答：start slave 的过程中需要初始化 worker信息，执行 crash recovery 等一系列操作，对于已经执行的不再执行，对于没有执行的继续执行，详情见global_init_info，mts_recovery_groups 等函数，在autocommit=0 的情况下，start slave thread 没有commit 或者执行完之前是不会释放锁资源的。

可惜俺认识源码，源码不认识我，所以就没去看那些源码......

mysql> show variables where variable_name in ('autocommit','master_info_repository','relay_log_info_repository');
+---------------------------+-------+
| Variable_name         | Value |
+---------------------------+-------+
| autocommit           | ON   |
| master_info_repository   | TABLE |
| relay_log_info_repository | TABLE |
+---------------------------+-------+
3 rows in set (0.00 sec)


解决方法：
将master_info_repository，relay_log_info_repository 设置为FILE后，重启slave。问题解决。


错误日志如下：
/mnt/workspace/percona-server-5.6-binaries-release/label_exp/debian-wheezy-x64/percona-server-5.6.24-72.2/storage/tokudb/ft-index/portability/toku_pthread.h:205 toku_mutex_lock: Assertion `r == 0' failed (errno=0) (r=22)
: Success
Backtrace: (Note: toku_do_assert=0x0x7ff77fccfc90)
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7ff77fccec8b]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(+0x8adb3)[0x7ff77fccedb3]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(+0xabf11)[0x7ff77fceff11]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(+0xc5d0d)[0x7ff77fd09d0d]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_Z35toku_logger_save_rollback_cmdinsertP7tokutxn7FILENUMP10BYTESTRING+0x2a)[0x7ff77fd2ab0a]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_Z20toku_ft_maybe_insertP9ft_handleP10__toku_dbtS2_P7tokutxnb10__toku_lsnb11ft_msg_type+0x94)[0x7ff77fd45da4]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(+0x42f59)[0x7ff77fc86f59]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_Z16env_put_multipleP13__toku_db_envP9__toku_dbP13__toku_db_txnPK10__toku_dbtS7_jPS2_P9DBT_ARRAYSA_Pj+0x51f)[0x7ff77fd66fbf]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb32insert_rows_to_dictionaries_multEP10__toku_dbtS1_P13__toku_db_txnP3THD+0x4c8)[0x7ff77fcaee78]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb9write_rowEPh+0x653)[0x7ff77fcaf623]
/Percona-Server-5.6/bin/mysqld(_ZN7handler12ha_write_rowEPh+0xef)[0x5c5c8f]
/Percona-Server-5.6/bin/mysqld(_ZN12ha_partition9write_rowEPh+0x103)[0xb6c523]
/Percona-Server-5.6/bin/mysqld(_ZN7handler12ha_write_rowEPh+0x9d)[0x5c5c3d]
/Percona-Server-5.6/bin/mysqld(_ZN20Write_rows_log_event9write_rowEPK14Relay_log_infob+0x135)[0x88b965]
/Percona-Server-5.6/bin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEPK14Relay_log_info+0x19)[0x88bc89]
/Percona-Server-5.6/bin/mysqld(_ZN14Rows_log_event12do_apply_rowEPK14Relay_log_info+0x26)[0x87b9c6]
/Percona-Server-5.6/bin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x526)[0x889be6]
/Percona-Server-5.6/bin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x6a)[0x8831ea]
/Percona-Server-5.6/bin/mysqld(_Z26apply_event_and_update_posPP9Log_eventP3THDP14Relay_log_info+0x25e)[0x8ba3ee]
/Percona-Server-5.6/bin/mysqld(handle_slave_sql+0x120b)[0x8bddeb]
/Percona-Server-5.6/bin/mysqld(pfs_spawn_thread+0x143)[0xb03a83]
/lib64/libpthread.so.0[0x3a74a07a51]
/lib64/libc.so.6(clone+0x6d)[0x3a746e893d]
Engine status:
BUILD_ID = 0
Env panic code: 18446744073709551615
Env panic string: /mnt/workspace/percona-server-5.6-binaries-release/label_exp/debian-wheezy-x64/percona-server-5.6.24-72.2/storage/tokudb/ft-index/portability/toku_pthread.h:205 toku_mutex_lock: Assertion `r == 0' failed (errno=0) (r=22)


time of environment creation: Wed Jan  4 10:49:11 2017
time of engine startup: Thu Jan  5 10:29:28 2017
time now: Thu Jan  5 10:36:28 2017
db opens: 1194
db closes: 0
num open dbs now: 1194
max open dbs: 1194
period, in ms, that recovery log is automatically fsynced: 0
dictionary inserts: 0
dictionary inserts fail: 0
dictionary deletes: 398
dictionary deletes fail: 0
dictionary updates: 0
dictionary updates fail: 0
dictionary broadcast updates: 0
dictionary broadcast updates fail: 0
dictionary multi inserts: 1836
dictionary multi inserts fail: 0
dictionary multi deletes: 0
dictionary multi deletes fail: 0
dictionary updates multi: 0
dictionary updates multi fail: 0
le: max committed xr: 1
le: max provisional xr: 2
le: expanded: 0
le: max memsize: 56
le: size of leafentries before garbage collection (during message application): 0
le: size of leafentries after garbage collection (during message application): 0
le: size of leafentries before garbage collection (outside message application): 0
le: size of leafentries after garbage collection (outside message application): 0
checkpoint: period: 60
checkpoint: footprint: 0
checkpoint: last checkpoint began : Thu Jan  5 10:35:28 2017
checkpoint: last complete checkpoint began : Thu Jan  5 10:35:28 2017
checkpoint: last complete checkpoint ended: Thu Jan  5 10:35:28 2017
checkpoint: time spent during checkpoint (begin and end phases): 0
checkpoint: time spent during last checkpoint (begin and end phases): 0
checkpoint: last complete checkpoint LSN: 668275881
checkpoint: checkpoints taken : 8
checkpoint: checkpoints failed: 0
checkpoint: waiters now: 0
checkpoint: waiters max: 0
checkpoint: non-checkpoint client wait on mo lock: 0
checkpoint: non-checkpoint client wait on cs lock: 0
checkpoint: checkpoint begin time: 178
checkpoint: long checkpoint begin time: 0
checkpoint: long checkpoint begin count: 0
cachetable: miss: 1043
cachetable: miss time: 163503
cachetable: prefetches: 0
cachetable: size current: 46579253
cachetable: size limit: 17371719680
cachetable: size writing: 0
cachetable: size nonleaf: 33785462
cachetable: size leaf: 12634415
cachetable: size rollback: 159376
cachetable: size cachepressure: 33557198
cachetable: size currently cloned data for checkpoint: 0
cachetable: evictions: 0
cachetable: cleaner executions: 419
cachetable: cleaner period: 1
cachetable: cleaner iterations: 5
cachetable: number of waits on cache pressure: 0
cachetable: time waiting on cache pressure: 0
cachetable: number of long waits on cache pressure: 0
cachetable: long time waiting on cache pressure: 0
locktree: memory size: 82766
locktree: memory size limit: 2104356096
locktree: number of times lock escalation ran: 0
locktree: time spent running escalation (seconds): 0.000000
locktree: latest post-escalation memory size: 0
locktree: number of locktrees open now: 1196
locktree: number of pending lock requests: 0
locktree: number of locktrees eligible for the STO: 4
locktree: number of times a locktree ended the STO early: 0
locktree: time spent ending the STO early (seconds): 0.000000
locktree: number of wait locks: 0
locktree: time waiting for locks: 0
locktree: number of long wait locks: 0
locktree: long time waiting for locks: 0
locktree: number of lock timeouts: 0
locktree: number of waits on lock escalation: 0
locktree: time waiting on lock escalation: 0
locktree: number of long waits on lock escalation: 0
locktree: long time waiting on lock escalation: 0
ft: dictionary updates: 0
ft: dictionary broadcast updates: 0
ft: descriptor set: 0
ft: messages ignored by leaf due to msn: 0
ft: total search retries due to TRY_AGAIN: 237
ft: searches requiring more tries than the height of the tree: 0
ft: searches requiring more tries than the height of the tree plus three: 0
ft: leaf nodes flushed to disk (not for checkpoint): 399
ft: leaf nodes flushed to disk (not for checkpoint) (bytes): 250368
ft: leaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 259717
ft: leaf nodes flushed to disk (not for checkpoint) (seconds): 0.003984
ft: nonleaf nodes flushed to disk (not for checkpoint): 3
ft: nonleaf nodes flushed to disk (not for checkpoint) (bytes): 2145280
ft: nonleaf nodes flushed to disk (not for checkpoint) (uncompressed bytes): 10734381
ft: nonleaf nodes flushed to disk (not for checkpoint) (seconds): 0.010821
ft: leaf nodes flushed to disk (for checkpoint): 0
ft: leaf nodes flushed to disk (for checkpoint) (bytes): 0
ft: leaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 0
ft: leaf nodes flushed to disk (for checkpoint) (seconds): 0.000000
ft: nonleaf nodes flushed to disk (for checkpoint): 0
ft: nonleaf nodes flushed to disk (for checkpoint) (bytes): 0
ft: nonleaf nodes flushed to disk (for checkpoint) (uncompressed bytes): 0
ft: nonleaf nodes flushed to disk (for checkpoint) (seconds): 0.000000
ft: uncompressed / compressed bytes written (leaf): UNKNOWN STATUS TYPE: 6
ft: uncompressed / compressed bytes written (nonleaf): UNKNOWN STATUS TYPE: 6
ft: uncompressed / compressed bytes written (overall): UNKNOWN STATUS TYPE: 6
ft: nonleaf node partial evictions: 0
ft: nonleaf node partial evictions (bytes): 0
ft: leaf node partial evictions: 0
ft: leaf node partial evictions (bytes): 0
ft: leaf node full evictions: 0
ft: leaf node full evictions (bytes): 0
ft: nonleaf node full evictions: 0
ft: nonleaf node full evictions (bytes): 0
ft: leaf nodes created: 0
ft: nonleaf nodes created: 0
ft: leaf nodes destroyed: 0
ft: nonleaf nodes destroyed: 0
ft: bytes of messages injected at root (all trees): 78864
ft: bytes of messages flushed from h1 nodes to leaves: 0
ft: bytes of messages currently in trees (estimate): 78864
ft: messages injected at root: 1446
ft: broadcast messages injected at root: 0
ft: basements decompressed as a target of a query: 1
ft: basements decompressed for prelocked range: 0
ft: basements decompressed for prefetch: 0
ft: basements decompressed for write: 0
ft: buffers decompressed as a target of a query: 0
ft: buffers decompressed for prelocked range: 0
ft: buffers decompressed for prefetch: 0
ft: buffers decompressed for write: 23
ft: pivots fetched for query: 121
ft: pivots fetched for query (bytes): 3928576
ft: pivots fetched for query (seconds): 0.001989
ft: pivots fetched for prefetch: 0
ft: pivots fetched for prefetch (bytes): 0
ft: pivots fetched for prefetch (seconds): 0.000000
ft: pivots fetched for write: 407
ft: pivots fetched for write (bytes): 434176
ft: pivots fetched for write (seconds): 0.000381
ft: basements fetched as a target of a query: 121
ft: basements fetched as a target of a query (bytes): 1105408
ft: basements fetched as a target of a query (seconds): 0.000323
ft: basements fetched for prelocked range: 0
ft: basements fetched for prelocked range (bytes): 0
ft: basements fetched for prelocked range (seconds): 0.000000
ft: basements fetched for prefetch: 0
ft: basements fetched for prefetch (bytes): 0
ft: basements fetched for prefetch (seconds): 0.000000
ft: basements fetched for write: 444
ft: basements fetched for write (bytes): 872448
ft: basements fetched for write (seconds): 0.000285
ft: buffers fetched as a target of a query: 0
ft: buffers fetched as a target of a query (bytes): 0
ft: buffers fetched as a target of a query (seconds): 0.000000
ft: buffers fetched for prelocked range: 0
ft: buffers fetched for prelocked range (bytes): 0
ft: buffers fetched for prelocked range (seconds): 0.000000
ft: buffers fetched for prefetch: 0
ft: buffers fetched for prefetch (bytes): 0
ft: buffers fetched for prefetch (seconds): 0.000000
ft: buffers fetched for write: 80
ft: buffers fetched for write (bytes): 4164608
ft: buffers fetched for write (seconds): 0.001280
ft: leaf compression to memory (seconds): 0.081140
ft: leaf serialization to memory (seconds): 0.000757
ft: leaf decompression to memory (seconds): 0.024833
ft: leaf deserialization to memory (seconds): 0.007450
ft: nonleaf compression to memory (seconds): 0.283292
ft: nonleaf serialization to memory (seconds): 0.029127
ft: nonleaf decompression to memory (seconds): 0.059308
ft: nonleaf deserialization to memory (seconds): 0.071366
ft: promotion: roots split: 0
ft: promotion: leaf roots injected into: 1592
ft: promotion: h1 roots injected into: 0
ft: promotion: injections at depth 0: 1162
ft: promotion: injections at depth 1: 284
ft: promotion: injections at depth 2: 365
ft: promotion: injections at depth 3: 100
ft: promotion: injections lower than depth 3: 0
ft: promotion: stopped because of a nonempty buffer: 1433
ft: promotion: stopped at height 1: 0
ft: promotion: stopped because the child was locked or not at all in memory: 13
ft: promotion: stopped because the child was not fully in memory: 0
ft: promotion: stopped anyway, after locking the child: 0
ft: basement nodes deserialized with fixed-keysize: 443
ft: basement nodes deserialized with variable-keysize: 401
ft: promotion: succeeded in using the rightmost leaf shortcut: 981
ft: promotion: tried the rightmost leaf shorcut but failed (out-of-bounds): 0
ft: promotion: tried the rightmost leaf shorcut but failed (child reactive): 1
ft: cursor skipped deleted leaf entries: 0
ft flusher: total nodes potentially flushed by cleaner thread: 0
ft flusher: height-one nodes flushed by cleaner thread: 0
ft flusher: height-greater-than-one nodes flushed by cleaner thread: 0
ft flusher: nodes cleaned which had empty buffers: 0
ft flusher: nodes dirtied by cleaner thread: 0
ft flusher: max bytes in a buffer flushed by cleaner thread: 0
ft flusher: min bytes in a buffer flushed by cleaner thread: 18446744073709551615
ft flusher: total bytes in buffers flushed by cleaner thread: 0
ft flusher: max workdone in a buffer flushed by cleaner thread: 0
ft flusher: min workdone in a buffer flushed by cleaner thread: 18446744073709551615
ft flusher: total workdone in buffers flushed by cleaner thread: 0
ft flusher: times cleaner thread tries to merge a leaf: 0
ft flusher: cleaner thread leaf merges in progress: 0
ft flusher: cleaner thread leaf merges successful: 0
ft flusher: nodes dirtied by cleaner thread leaf merges: 0
ft flusher: total number of flushes done by flusher threads or cleaner threads: 0
ft flusher: number of in memory flushes: 0
ft flusher: number of flushes that read something off disk: 0
ft flusher: number of flushes that triggered another flush in child: 0
ft flusher: number of flushes that triggered 1 cascading flush: 0
ft flusher: number of flushes that triggered 2 cascading flushes: 0
ft flusher: number of flushes that triggered 3 cascading flushes: 0
ft flusher: number of flushes that triggered 4 cascading flushes: 0
ft flusher: number of flushes that triggered 5 cascading flushes: 0
ft flusher: number of flushes that triggered over 5 cascading flushes: 0
ft flusher: leaf node splits: 0
ft flusher: nonleaf node splits: 0
ft flusher: leaf node merges: 0
ft flusher: nonleaf node merges: 0
ft flusher: leaf node balances: 0
hot: operations ever started: 0
hot: operations successfully completed: 0
hot: operations aborted: 0
hot: max number of flushes from root ever required to optimize a tree: 0
txn: begin: 2394
txn: begin read only: 0
txn: successful commits: 2392
txn: aborts: 1
logger: next LSN: 668279196
logger: writes: 408
logger: writes (bytes): 197974
logger: writes (uncompressed bytes): 197974
logger: writes (seconds): 0.004156
logger: number of long logger write operations: 0
indexer: number of indexers successfully created: 0
indexer: number of calls to toku_indexer_create_indexer() that failed: 0
indexer: number of calls to indexer->build() succeeded: 0
indexer: number of calls to indexer->build() failed: 0
indexer: number of calls to indexer->close() that succeeded: 0
indexer: number of calls to indexer->close() that failed: 0
indexer: number of calls to indexer->abort(): 0
indexer: number of indexers currently in existence: 0
indexer: max number of indexers that ever existed simultaneously: 0
loader: number of loaders successfully created: 0
loader: number of calls to toku_loader_create_loader() that failed: 0
loader: number of calls to loader->put() succeeded: 0
loader: number of calls to loader->put() failed: 0
loader: number of calls to loader->close() that succeeded: 0
loader: number of calls to loader->close() that failed: 0
loader: number of calls to loader->abort(): 0
loader: number of loaders currently in existence: 0
loader: max number of loaders that ever existed simultaneously: 0
memory: number of malloc operations: 0
memory: number of free operations: 0
memory: number of realloc operations: 0
memory: number of malloc operations that failed: 0
memory: number of realloc operations that failed: 0
memory: number of bytes requested: 0
memory: number of bytes used (requested + overhead): 0
memory: number of bytes freed: 0
memory: largest attempted allocation size: 0
memory: size of the last failed allocation attempt: 0
memory: estimated maximum memory footprint: 0
memory: mallocator version: 3.6.0-0-g46c0af68bd248b04df75e4f92d5fb804c3d75340
memory: mmap threshold: 4194304
filesystem: ENOSPC redzone state: 0
filesystem: threads currently blocked by full disk: 0
filesystem: number of operations rejected by enospc prevention (red zone): 0
filesystem: most recent disk full: Thu Jan  1 08:00:00 1970
filesystem: number of write operations that returned ENOSPC: 0
filesystem: fsync time: 466349
filesystem: fsync count: 2407
filesystem: long fsync time: 0
filesystem: long fsync count: 0
context: tree traversals blocked by a full fetch: 0
context: tree traversals blocked by a partial fetch: 0
context: tree traversals blocked by a full eviction: 0
context: tree traversals blocked by a partial eviction: 0
context: tree traversals blocked by a message injection: 0
context: tree traversals blocked by a message application: 0
context: tree traversals blocked by a flush: 0
context: tree traversals blocked by a the cleaner thread: 0
context: tree traversals blocked by something uninstrumented: 0
context: promotion blocked by a full fetch (should never happen): 0
context: promotion blocked by a partial fetch (should never happen): 0
context: promotion blocked by a full eviction (should never happen): 0
context: promotion blocked by a partial eviction (should never happen): 0
context: promotion blocked by a message injection: 0
context: promotion blocked by a message application: 0
context: promotion blocked by a flush: 0
context: promotion blocked by the cleaner thread: 0
context: promotion blocked by something uninstrumented: 0
context: something uninstrumented blocked by something uninstrumented: 0


Memory usage:
Arena 0:
system bytes     =          0
in use bytes     =          0
Total (incl. mmap):
system bytes     =          0
in use bytes     =          0
max mmap regions =          0
max mmap bytes   =          0
02:36:28 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/


key_buffer_size=4194304
read_buffer_size=131072
max_used_connections=2
max_threads=2050
thread_count=2
connection_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 819597 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.


Thread pointer: 0x7ff6dcc13000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff6f00b2820 thread_stack 0x40000
/Percona-Server-5.6/bin/mysqld(my_print_stacktrace+0x2c)[0x8f308c]
/Percona-Server-5.6/bin/mysqld(handle_fatal_signal+0x461)[0x67b2b1]
/lib64/libpthread.so.0[0x3a74a0f790]
/lib64/libc.so.6(gsignal+0x35)[0x3a74632625]
/lib64/libc.so.6(abort+0x175)[0x3a74633e05]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(+0x8adb8)[0x7ff77fccedb8]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(+0xabf11)[0x7ff77fceff11]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(+0xc5d0d)[0x7ff77fd09d0d]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_Z35toku_logger_save_rollback_cmdinsertP7tokutxn7FILENUMP10BYTESTRING+0x2a)[0x7ff77fd2ab0a]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_Z20toku_ft_maybe_insertP9ft_handleP10__toku_dbtS2_P7tokutxnb10__toku_lsnb11ft_msg_type+0x94)[0x7ff77fd45da4]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(+0x42f59)[0x7ff77fc86f59]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_Z16env_put_multipleP13__toku_db_envP9__toku_dbP13__toku_db_txnPK10__toku_dbtS7_jPS2_P9DBT_ARRAYSA_Pj+0x51f)[0x7ff77fd66fbf]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb32insert_rows_to_dictionaries_multEP10__toku_dbtS1_P13__toku_db_txnP3THD+0x4c8)[0x7ff77fcaee78]
/Percona-Server-5.6/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb9write_rowEPh+0x653)[0x7ff77fcaf623]
/Percona-Server-5.6/bin/mysqld(_ZN7handler12ha_write_rowEPh+0xef)[0x5c5c8f]
/Percona-Server-5.6/bin/mysqld(_ZN12ha_partition9write_rowEPh+0x103)[0xb6c523]
/Percona-Server-5.6/bin/mysqld(_ZN7handler12ha_write_rowEPh+0x9d)[0x5c5c3d]
/Percona-Server-5.6/bin/mysqld(_ZN20Write_rows_log_event9write_rowEPK14Relay_log_infob+0x135)[0x88b965]
/Percona-Server-5.6/bin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEPK14Relay_log_info+0x19)[0x88bc89]
/Percona-Server-5.6/bin/mysqld(_ZN14Rows_log_event12do_apply_rowEPK14Relay_log_info+0x26)[0x87b9c6]
/Percona-Server-5.6/bin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x526)[0x889be6]
/Percona-Server-5.6/bin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x6a)[0x8831ea]
/Percona-Server-5.6/bin/mysqld(_Z26apply_event_and_update_posPP9Log_eventP3THDP14Relay_log_info+0x25e)[0x8ba3ee]
/Percona-Server-5.6/bin/mysqld(handle_slave_sql+0x120b)[0x8bddeb]
/Percona-Server-5.6/bin/mysqld(pfs_spawn_thread+0x143)[0xb03a83]
/lib64/libpthread.so.0[0x3a74a07a51]
/lib64/libc.so.6(clone+0x6d)[0x3a746e893d]


Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 139
Status: NOT_KILLED


You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
170105 10:36:28 mysqld_safe Number of processes running now: 0
170105 10:36:28 mysqld_safe mysqld restarted