elasticsearch 使用go-mysql-elasticsearch 同步Mysql NDB Cluster 日志格式设置
来源:互联网 发布:平面设计软件怎么下载 编辑:程序博客网 时间:2024/06/08 10:19
在使用Elasticsearch 时碰到了这样的问题,记录以备查
使用go-mysql-elasticsearch插件同步时,数据库是 MYSQL,使用的是引擎是 NDB Cluster ;
go-mysql-elasticsearch 使用说明中有如下说明:
Notice
- binlog format must be row.
- binlog row image must be full for MySQL, you may lost some field data if you update PK data in MySQL with minimal or noblob binlog row image. MariaDB only supports full row image.
- Can not alter table format at runtime.
- MySQL table which will be synced should have a PK(primary key), multi columns PK is allowed now, e,g, if the PKs is (a, b), we will use "a:b" as the key. The PK data will be used as "id" in Elasticsearch. And you can also config the id's constituent part with other column.
- You should create the associated mappings in Elasticsearch first, I don't think using the default mapping is a wise decision, you must know how to search accurately.
mysqldump
must exist in the same node with go-mysql-elasticsearch, if not, go-mysql-elasticsearch will try to sync binlog only.- Don't change too many rows at same time in one SQL.
################BINLOG 配置#########
log-bin=mysql-bin
binlog_format = row
binlog-row-image = full
但当更新了数据库表中某些字段时,go-mysql-elasticsearch 同步数据到搜索引擎时出现 除id,和更新的字段外其他字段为空的情况。
eg:
原内容如下图:
当更新第一条的字段 buyers 为1 时,同步后内容如下图:
原因:查看mysql配置
show global variables like 'ndb_%';
| ndb_log_update_as_write | ON
| ndb_log_updated_only | ON
查看MYSQL 配置说明:
Logging Full or Partial Rows (--ndb-log-updated-only Option)
--ndb-log-updated-only
System VariableNamendb_log_updated_only
ScopeGlobalDynamicYesPermitted ValuesTypebooleanDefaultON
For purposes of conflict resolution, there are two basic methods of logging rows, as determined by the setting of the --ndb-log-updated-only
option for mysqld:
Log complete rows
Log only column data that has been updated—that is, column data whose value has been set, regardless of whether or not this value was actually changed. This is the default behavior.
It is usually sufficient—and more efficient—to log updated columns only; however, if you need to log full rows, you can do so by setting --ndb-log-updated-only
to 0
or OFF
.
--ndb-log-update-as-write Option: Logging Changed Data as Updates
--ndb-log-update-as-write
System VariableNamendb_log_update_as_write
ScopeGlobalDynamicYesPermitted ValuesTypebooleanDefaultON
The setting of the MySQL Server's --ndb-log-update-as-write
option determines whether logging is performed with or without the “before” image. Because conflict resolution is done in the MySQL Server's update handler, it is necessary to control logging on the master such that updates are updates and not writes; that is, such that updates are treated as changes in existing rows rather than the writing of new rows (even though these replace existing rows). This option is turned on by default; in other words, updates are treated as writes. (That is, updates are by default written as write_row
events in the binary log, rather than as update_row
events.)
To turn off the option, start the master mysqld with --ndb-log-update-as-write=0
or --ndb-log-update-as-write=OFF
. You must do this when replicating from NDB tables to tables using a different storage engine; see Replication from NDB to other storage engines, and Replication from NDB to a nontransactional storage engine, for more information.
设置 ndb-log-update-as-write=OFF 即可
show global variables like 'ndb_%';
| ndb_log_update_as_write | OFF
| ndb_log_updated_only | ON
重启数据后再同步问题解决
- elasticsearch 使用go-mysql-elasticsearch 同步Mysql NDB Cluster 日志格式设置
- MySQL(9) -- Cluster NDB使用
- LVS & MySQL NDB Cluster
- Mysql ndb cluster初试
- mysql ndb cluster 安装
- MySQL NDB Cluster
- MySQL NDB Cluster实战
- Elasticsearch+mysql 数据同步
- ElasticSearch同步MySql
- elasticsearch mysql数据同步
- go-mysql-elasticsearch实现mysql 与elasticsearch实时同步深入详解
- elasticsearch使用river同步mysql数据
- elasticsearch使用river同步mysql数据
- Mysql Cluster NDB配置介绍
- MySQL Cluster (NDB) Installation Guide
- MySQL NDB Cluster 集群简介
- Elasticsearch+MySQL数据同步示例
- elasticsearch-jdbc实现MySQL同步到ElasticSearch深入详解
- 修改Eclipse格式化代默认长度
- opencv学习系列:目标检测相关
- Python.CVXPY学习指南三
- 记录被计算机虐死的心酸历程之bamm-2.5.0的安装-1
- 2017 Google 开发者大会都有哪些炫酷科技?
- elasticsearch 使用go-mysql-elasticsearch 同步Mysql NDB Cluster 日志格式设置
- 基于Nginx负载均衡方案
- 数据结构实验之排序七:选课名单
- Redis安装
- swift-oc wkwebView使用自定义UIMenuController
- 22. 其他操作流
- Centos下Spark单机版(python)安装配置
- 经典的营销管理模型
- hadoop学习记(2)--HDFS+yarn+MapReduce关系与原理