迁移scm数据库

保证mysql在server端安装成功，查看集群配置，其他库的使用情况是否一致，一般情况下仅仅修正scm库。查看psql中数据

mysql的my.cnf配置修改

[mysqld]datadir=/var/lib/mysqlsocket=/var/lib/mysql/mysql.sockuser=mysql# Recommended in standard MySQL setupsql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLEStransaction-isolation = READ-COMMITTED# Disabling symbolic-links is recommended to prevent assorted security risks;# to do so, uncomment this line:# symbolic-links = 0key_buffer_size = 32Mmax_allowed_packet = 32Mthread_stack = 256Kthread_cache_size = 64query_cache_limit = 8Mquery_cache_size = 64Mquery_cache_type = 1max_connections = 550#expire_logs_days = 10#max_binlog_size = 100M#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system#and chown the specified folder to the mysql user.log_bin=/var/lib/mysql/mysql_binary_logserver-id=1# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.binlog_format = mixedread_buffer_size = 2Mread_rnd_buffer_size = 16Msort_buffer_size = 8Mjoin_buffer_size = 8M# InnoDB settingsinnodb_file_per_table = 1innodb_flush_log_at_trx_commit  = 2innodb_log_buffer_size = 64Minnodb_buffer_pool_size = 4Ginnodb_thread_concurrency = 8innodb_flush_method = O_DIRECTinnodb_log_file_size = 512M[mysqld_safe]log-error=/var/log/mysqld.logpid-file=/var/run/mysqld/mysqld.pidsql_mode=STRICT_ALL_TABLES

---

mysql创建数据库（主要对scm数据库进行迁移，可选可不选）
删除database：
DROP DATABASE scm;
DROP DATABASE amon;
DROP DATABASE rman;
DROP DATABASE metastore;
DROP DATABASE sentry;
DROP DATABASE nav;
DROP DATABASE navms;

创建database：create database amon DEFAULT CHARACTER SET utf8;grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';create database rman DEFAULT CHARACTER SET utf8;grant all on rman.* TO'rman'@'%' IDENTIFIED BY 'rman_password';create database metastore DEFAULT CHARACTER SET utf8;grant all on metastore.*TO 'hive'@'%' IDENTIFIED BY 'hive_password';create database sentry DEFAULT CHARACTER SET utf8;grant all on sentry.* TO'sentry'@'%' IDENTIFIED BY 'sentry_password';create database nav DEFAULT CHARACTER SET utf8;grant all on nav.* TO 'nav'@'%'IDENTIFIED BY 'nav_password';create database navms DEFAULT CHARACTER SET utf8;grant all on navms.* TO 'navms'@'%'IDENTIFIED BY 'navms_password';

---

mysql保证innodb引擎

show global variables like "have_%";show plugins;show variables like "have_dynamic%"; 是否支持动态加载插件show variables like 'plugin_dir'; #找到mysql存放插件的目录INSTALL PLUGIN InnoDB SONAME 'ha_innodb.so';    

http://www.linuxidc.com/Linux/2015-04/116167.htm

---

1.停止服务

1.停止集群2.停止cm的service3.停止bn00-bn04的agent    sudo service cloudera-scm-agent stop

2.备份cm的api

mkdir -p /home/user/deploymentcurl -v -u admin:admin http://bn00:7180/api/v14/cm/deployment > /home/user/deployment

3.对postgre的db和server目录做备份

cd /var/lib/cloudera-scm-server-db/tar -cvf /home/user/cloudera-scm-server-db.tar .cd /etc/cloudera-scm-server/tar -cvf /home/user/cloudera-scm-server.tar .

4.在bn00端停止server，db

sudo service cloudera-scm-server stopsudo service cloudera-scm-server-db stopsudo chkconfig cloudera-scm-server-db off

5.mysql中创建temp

#mysql -h mysql_hostname -u user -pmysql> create database temp;mysql> grant all on *.* to 'temp'@'%' identified by 'temp' with grant option;#/usr/share/cmf/schema/scm_prepare_database.sh mysql -h bn00 -utemp -ptemp --scm-host bn00 scm scm scm   #创建了scm显示信息最后一行如下：All done, your Cloudera Manager database is ready to go!则代表执行成功。删除temp usermysql>drop user 'temp'@'%';

6.在bn00端开启server

sudo service httpd start sudo service cloudera-scm-server start

7.登录UI界面，恢复API，恢复集群

$ curl --upload-file /home/user/deployment -u admin:admin http://bn00:7180/api/v14/cm/deployment?deleteCurrentDeployment=true -H "Content-Type: application/json"

8.刷新UI界面，目前所有服务都在，处于待重启状态。

9.删除旧的guid（所有主机）(主机状况运行不良情况)

rm -rf /var/lib/cloudera-scm-agent/cm_guid

10.主机-所有主机-重新运行主机升级向导，注意使用密钥。

11.查看parcel状态是否正常，进行激活操作。

可能的问题：    parcel处没有任何操作可以进行，卡住了。解决方案：http://node0:7180/static/apidocs/ API文档    1.在http://bn00:7180/api/v14/cm/deployment中找到对应parcel相应的配置以及版本。    2.curl -u admin:admin -X POST http://node0:7180/api/v14/clusters/cluster/parcels/products/CDH/versions/5.7.5-1.cdh5.7.5.p0.3/commands/deactivate    3.curl -u admin:admin -X POST http://bn00:7180/api/v14/clusters/cluster/parcels/products/KAFKA/versions/2.0.1-1.2.0.1.p0.5/commands/deactivate

---

postgresql相关密码查询
http://blog.csdn.net/yujin2010good/article/details/72482007

---

bn00数据库相关配置

Hbase

Hive

Hue

amon

rman

metastore

sentry

nav

navms

（在采用Cloudera-Manager安装cdh时，通常使用内嵌的PostgreSQL数据库。 Cloudera-Manager除了保存CDH集群的配置元数据的scm数据库外，还为Activity Monitor（活动监控）、Service Monitor（服务监控）、Report Manager（报告管理）、Host Monitor（主机监控）、Cloudera Navigator（Cloudera导航）等信息分别创建数据amon、smon、rmon、hmon、nav相应的数据。）
http://blog.csdn.net/yujin2010good/article/details/72482007
http://www.cnblogs.com/jxrichar/p/7240812.html

---

psql的基本命令
>
一、建立数据库连接
接入PostgreSQL数据库: psql -h IP地址 -p 端口 -U 数据库名
>
之后会要求输入数据库密码
>
二、访问数据库
>
1、列举数据库：\l
>
2、选择数据库：\c 数据库名

3、查看该某个库中的所有表：\dt

4、切换数据库：\c interface

5、查看某个库中的某个表结构：\d 表名

6、查看某个库中某个表的记录：select * from apps limit 1;

7、显示字符集：\encoding

8、退出psgl：\q

---

各种坑
>
1.代理节点启动报错（主机状况不良）：
错误信息：Error, CM server guid updated, expected df16790a-2e44-44ec-9db2-8731cc635c61, received b6fecabc-8e32-46be-8a43-5f261064b2c7
解决方法：删除/var/lib/cloudera-scm-agent下cm_guid文件
>
2.oozie缺少ext-2.2：
解决方法：将ext-2.2拷贝至/var/lib/oozie目录
>
3.spark运行报错：Required executor memory (1024+384 MB) is above the max threshold (1024 MB) of this cluster!
Please check the values of ‘yarn.scheduler.maximum-allocation-mb’ and/or ‘yarn.nodemanager.resource.memory-mb’
解决方法：修改服务YARN (MR2 Included)中如下配置项并保存，重启YARN：
最小容器内存(大于1G)：yarn.scheduler.minimum-allocation-mb
最大容器内存(大于1G)：yarn.scheduler.maximum-allocation-mb
容器内存(大于1G)：yarn.nodemanager.resource.memory-mb
>
4.spark运行报错：org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode=”/user”:hdfs:supergroup:drwxr-xr-x
解决方法：将export HADOOP_USER_NAME=hdfs添加到~/.bash_profile中，然后source ~/.bash_profile
>
5.hive执行引擎默认为mapreduce的配置修改：hive.execution.engine由mr改为spark
>
6.