mysqlfilter-manual

HengWang edited this page on 23 Oct 2012 · 1 revision

 Pages 1

• mysqlfilter manual

Clone this wiki locally

 Clone in Desktop

Mysqlfilter toolkit is based on mysqldump in order to filter the sensitive fields information when I dump the mysql data to developers or PM, the sensitive fields information like password, telephone. These sensitive fields are not important for developers to develop or test, but very important for company, which are the most important resource for the company development and must be protected the security of users.

Usually, in order to filter the sensitive fields information, DBA commonly dump the data used by mysqldump and then import a temp database instance, the next step is cleaning the data by updating the sensitive fields to a insensitive words, finally dump the cleaning data again and send to developers to develop and test. For many companies, they usually build their own platform to provide the cleaning data. But in my mind, I like mysqldump to dump data, and for developers, they prefer to restore the data by sql file. Based on the above purposes, I extended the mysqldump and named mysqlfilter in order to distinguish. Add —sets and —rule options to filter the sensitive fields through the format string like ‘Database.Table.Field=,…’ in —sets option or a rule configure file path in —rule option. Otherwise, I extend the existing —where option because of not supporting filter the given field in the given table of given database to filter the data and add the filter rule of where option into the rule configure file. The rule configure file format as followed.

database <DB_NAME>{  table <TB_NAME>{    filter{      <FD_NAME> = <REPLACE>;      <FD_NAME> = <REPLACE>;    }    where{      <FD_NAME> = <WHERE>;      <FD_NAME> in '(<WHERE>,<WHERE>,...)';      <FD_NAME> like "<REPLACE>";    }  }  ...}...

The above rule configure file has a few notes: 1. The where module support ‘=’, ‘>’, ‘>=’, ‘<’, ‘<=’, ‘like’ and ‘in’ link symbol, but their relationship is ‘and’. 2. The the value of the field (replacement or filter) can use ‘\’’ or ‘\"’ to mark the string, the different is ‘\’’ will be skiped because we just to mean the value is string but not a segment for the data, for example “ in ‘(,,…)’”, the where condition after the link symbol just to build up the sql statement. But the ‘\"’ will be retained used to keep the character type.

How to compile mysqlfilter:
1. Clone the source from github: git clone https://github.com/HengWang/mysqlfilter.git
2. Move the source to the client directory of mysql source and compile the mysql. If the system ask to overwrite the CMakeLists.txt , then say yes to overwrite the file.

How to use mysqlfilter:
The mysqlfilter is based on mysqldump, so the options is the same as mysqldump except —where. The different between mysqlfilter and mysqldump is the mysqlfilter add the —sets and —rule options and extend the —where option.

Welcome you test and use the mysqlfilter toolkit, if you encounter any questions, please contact me. Email:[e-mail]; [e-mail]; [e-mail].

HengWang edited this page on 23 Oct 2012 · 1 revision

 Pages 1

• mysqlfilter manual

Clone this wiki locally

 Clone in Desktop

Mysqlfilter toolkit is based on mysqldump in order to filter the sensitive fields information when I dump the mysql data to developers or PM, the sensitive fields information like password, telephone. These sensitive fields are not important for developers to develop or test, but very important for company, which are the most important resource for the company development and must be protected the security of users.

Usually, in order to filter the sensitive fields information, DBA commonly dump the data used by mysqldump and then import a temp database instance, the next step is cleaning the data by updating the sensitive fields to a insensitive words, finally dump the cleaning data again and send to developers to develop and test. For many companies, they usually build their own platform to provide the cleaning data. But in my mind, I like mysqldump to dump data, and for developers, they prefer to restore the data by sql file. Based on the above purposes, I extended the mysqldump and named mysqlfilter in order to distinguish. Add —sets and —rule options to filter the sensitive fields through the format string like ‘Database.Table.Field=,…’ in —sets option or a rule configure file path in —rule option. Otherwise, I extend the existing —where option because of not supporting filter the given field in the given table of given database to filter the data and add the filter rule of where option into the rule configure file. The rule configure file format as followed.

database <DB_NAME>{  table <TB_NAME>{    filter{      <FD_NAME> = <REPLACE>;      <FD_NAME> = <REPLACE>;    }    where{      <FD_NAME> = <WHERE>;      <FD_NAME> in '(<WHERE>,<WHERE>,...)';      <FD_NAME> like "<REPLACE>";    }  }  ...}...

The above rule configure file has a few notes: 1. The where module support ‘=’, ‘>’, ‘>=’, ‘<’, ‘<=’, ‘like’ and ‘in’ link symbol, but their relationship is ‘and’. 2. The the value of the field (replacement or filter) can use ‘\’’ or ‘\"’ to mark the string, the different is ‘\’’ will be skiped because we just to mean the value is string but not a segment for the data, for example “ in ‘(,,…)’”, the where condition after the link symbol just to build up the sql statement. But the ‘\"’ will be retained used to keep the character type.

How to compile mysqlfilter:
1. Clone the source from github: git clone https://github.com/HengWang/mysqlfilter.git
2. Move the source to the client directory of mysql source and compile the mysql. If the system ask to overwrite the CMakeLists.txt , then say yes to overwrite the file.

How to use mysqlfilter:
The mysqlfilter is based on mysqldump, so the options is the same as mysqldump except —where. The different between mysqlfilter and mysqldump is the mysqlfilter add the —sets and —rule options and extend the —where option.

Welcome you test and use the mysqlfilter toolkit, if you encounter any questions, please contact me. Email:[e-mail]; [e-mail]; [e-mail].

备注：该篇技术文档是与MySQL数据库中的“备份恢复”主题相关的技术主题。


本文转载自：https://github.com/HengWang/mysqlfilter/wiki/mysqlfilter-manual

HengWang edited this page on 23 Oct 2012 · 1 revision

 Pages 1

• mysqlfilter manual

Clone this wiki locally

 Clone in Desktop

Mysqlfilter toolkit is based on mysqldump in order to filter the sensitive fields information when I dump the mysql data to developers or PM, the sensitive fields information like password, telephone. These sensitive fields are not important for developers to develop or test, but very important for company, which are the most important resource for the company development and must be protected the security of users.

Usually, in order to filter the sensitive fields information, DBA commonly dump the data used by mysqldump and then import a temp database instance, the next step is cleaning the data by updating the sensitive fields to a insensitive words, finally dump the cleaning data again and send to developers to develop and test. For many companies, they usually build their own platform to provide the cleaning data. But in my mind, I like mysqldump to dump data, and for developers, they prefer to restore the data by sql file. Based on the above purposes, I extended the mysqldump and named mysqlfilter in order to distinguish. Add —sets and —rule options to filter the sensitive fields through the format string like ‘Database.Table.Field=,…’ in —sets option or a rule configure file path in —rule option. Otherwise, I extend the existing —where option because of not supporting filter the given field in the given table of given database to filter the data and add the filter rule of where option into the rule configure file. The rule configure file format as followed.

database <DB_NAME>{  table <TB_NAME>{    filter{      <FD_NAME> = <REPLACE>;      <FD_NAME> = <REPLACE>;    }    where{      <FD_NAME> = <WHERE>;      <FD_NAME> in '(<WHERE>,<WHERE>,...)';      <FD_NAME> like "<REPLACE>";    }  }  ...}...

The above rule configure file has a few notes: 1. The where module support ‘=’, ‘>’, ‘>=’, ‘<’, ‘<=’, ‘like’ and ‘in’ link symbol, but their relationship is ‘and’. 2. The the value of the field (replacement or filter) can use ‘\’’ or ‘\"’ to mark the string, the different is ‘\’’ will be skiped because we just to mean the value is string but not a segment for the data, for example “ in ‘(,,…)’”, the where condition after the link symbol just to build up the sql statement. But the ‘\"’ will be retained used to keep the character type.

How to compile mysqlfilter:
1. Clone the source from github: git clone https://github.com/HengWang/mysqlfilter.git
2. Move the source to the client directory of mysql source and compile the mysql. If the system ask to overwrite the CMakeLists.txt , then say yes to overwrite the file.

How to use mysqlfilter:
The mysqlfilter is based on mysqldump, so the options is the same as mysqldump except —where. The different between mysqlfilter and mysqldump is the mysqlfilter add the —sets and —rule options and extend the —where option.

Welcome you test and use the mysqlfilter toolkit, if you encounter any questions, please contact me. Email:[e-mail]; [e-mail]; [e-mail].