cloudera-manager安装+CDH5.4.1部署+mysql+python+impala安装

三台Red Hat Enterprise Linux Server release 6.3 (Santiago) ，搭建hadoop

[root@master01 cdh_install]# more /etc/hosts127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4::1         localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.5.140 master01192.168.5.141 slave01192.168.5.142 slave02

--解压jdk

tar -xvf jdk-7u7-linux-i586.tar.gz 

--配置jdk环境变量

export JAVA_HOME=/it/jdk1.7.0_07export JRE_HOME=${JAVA_HOME}/jreexport CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/libexport PATH=${JAVA_HOME}/bin:$PATH

--测试java环境

[root@master01 it]# source /etc/profile[root@master01 it]# java -versionjava version "1.7.0_07"Java(TM) SE Runtime Environment (build 1.7.0_07-b10)Java HotSpot(TM) Server VM (build 23.3-b01, mixed mode)

--上传安装包

[root@master01 cdh_install]# mkdir -p cdh_install[root@master01 cdh_install]# cd cdh_install/

[root@master01 cdh_install]# scp * root@slave01:/it/cdh_install/root@slave01's password: CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel                                  100% 1215MB  46.7MB/s   00:26    CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha                              100%   41     0.0KB/s   00:00    cloudera-manager-el6-cm5.4.1_x86_64.tar.gz                            100%  659MB  43.9MB/s   00:15    manifest.json                                                         100%   41KB  41.5KB/s   00:00    Python-2.6.6.tgz                                                      100%   13MB  12.7MB/s   00:00    Python-2.7.10.tgz                                                     100%   16MB  16.0MB/s   00:01    [root@master01 cdh_install]# scp * root@slave02:/it/cdh_install/root@slave02's password: CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel                                  100% 1215MB  40.5MB/s   00:30    CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha                              100%   41     0.0KB/s   00:00    cloudera-manager-el6-cm5.4.1_x86_64.tar.gz                            100%  659MB  47.1MB/s   00:14    manifest.json                                                         100%   41KB  41.5KB/s   00:00    Python-2.6.6.tgz                                                      100%   13MB  12.7MB/s   00:00    Python-2.7.10.tgz                                                     100%   16MB  16.0MB/s   00:01    [root@master01 cdh_install]# 

--下面我们进行CM5.4.1的安装配置，在安装的过程中，官方文档有点累赘，看起来有点费劲（但不可否认，它确实很详细，最权威的文档）

 

--Installation Path C - Manual Installation Using Cloudera Manager Tarballs

--建立目录，拷贝文件(root用户下执行以下命令)sudo mkdir /opt/cloudera-managertar xzf cloudera-manager*.tar.gz -C /opt/cloudera-manageruseradd --system --home=/opt/cloudera-manager/cm-5.4.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scmmkdir -p /var/log/cloudera-scm-headlampmkdir -p /var/log/cloudera-scm-firehosemkdir -p /var/log/cloudera-scm-alertpublishermkdir -p /var/log/cloudera-scm-eventservermkdir -p /var/lib/cloudera-scm-headlampmkdir -p /var/lib/cloudera-scm-firehosemkdir -p /var/lib/cloudera-scm-alertpublishermkdir -p /var/lib/cloudera-scm-eventservermkdir -p /var/lib/cloudera-scm-serverchown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlampmkdir -p /var/cm_logs/cloudera-scm-headlampchown cloudera-scm /var/cm_logs/cloudera-scm-headlampmkdir -p /opt/cloudera/parcel-repochown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repomkdir -p /opt/cloudera/parcelschown cloudera-scm:cloudera-scm /opt/cloudera/parcels

--下面进行mysql的安装配置（rpm包）

--mysql安装

[root@master01 cdh_install]#  rpm -ivh mysql-community-client-5.7.9-1.el6.x86_64.rpm mysql-community-com-embedded-5.7.9-1.el6.x86_64.rpm mysql-community-embedded-devel-5.7.9-1.el6.x86_64.rpm mysql-community-lmunity-server-5.7.9-1.el6.x86_64.rpmwarning: mysql-community-client-5.7.9-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NPreparing...                ########################################### [100%]file /usr/share/mysql/czech/errmsg.sys from install of mysql-community-common-5.7.9-1.el6.x86_64file /usr/share/mysql/danish/errmsg.sys from install of mysql-community-common-5.7.9-1.el6.x86_6file /usr/share/mysql/dutch/errmsg.sys from install of mysql-community-common-5.7.9-1.el6.x86_64

上面mysql安装报错，肯定是mysql有低版本的包，查看mysql已安装的包

[root@master01 cdh_install]# rpm -qa | grep -i mysqlmysql-libs-5.1.61-4.el6.x86_64

--卸载已安装的mysql

[root@master01 cdh_install]# yum -y remove mysql-libs-5.1.61*Loaded plugins: product-id, refresh-packagekit, subscription-managerUpdating certificate-based repositories.Unable to read consumer identitySetting up Remove ProcessResolving Dependencies--> Running transaction check---> Package mysql-libs.x86_64 0:5.1.61-4.el6 will be erased--> Processing Dependency: libmysqlclient.so.16()(64bit) for package: 2:postfix-2.6.6-2.2.el6_1.x86_64--> Processing Dependency: libmysqlclient.so.16(libmysqlclient_16)(64bit) for package: 2:postfix-2.6.6-2.2.el6_1.x86_64--> Processing Dependency: mysql-libs for package: 2:postfix-2.6.6-2.2.el6_1.x86_64--> Running transaction check---> Package postfix.x86_64 2:2.6.6-2.2.el6_1 will be erased--> Processing Dependency: /usr/sbin/sendmail for package: cronie-1.4.4-7.el6.x86_64--> Running transaction check省略。。。Removed:  mysql-libs.x86_64 0:5.1.61-4.el6                                                                      Dependency Removed:  cronie.x86_64 0:1.4.4-7.el6       cronie-anacron.x86_64 0:1.4.4-7.el6  crontabs.noarch 0:1.10-33.el6   postfix.x86_64 2:2.6.6-2.2.el6_1  sysstat.x86_64 0:9.0.4-20.el6       Complete![root@master01 cdh_install]# 

--查看是否已卸载

[root@master01 cdh_install]# rpm -qa | grep -i mysql[root@master01 cdh_install]# 没有了

--重新安装

[root@master01 cdh_install]#  rpm -ivh mysql-community-client-5.7.9-1.el6.x86_64.rpm mysql-community-common-5.7.9-1.el6.x86_64.rpm mysql-community-devel-5.7.9-1.el6.x86_64.rpm mysql-community-embedded-5.7.9-1.el6.x86_64.rpm mysql-community-embedded-devel-5.7.9-1.el6.x86_64.rpm mysql-community-libs-5.7.9-1.el6.x86_64.rpm mysql-community-libs-compat-5.7.9-1.el6.x86_64.rpm mysql-community-server-5.7.9-1.el6.x86_64.rpm warning: mysql-community-client-5.7.9-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEYPreparing...                ########################################### [100%]   1:mysql-community-common ########################################### [ 13%]   2:mysql-community-libs   ########################################### [ 25%]   3:mysql-community-client ########################################### [ 38%]   4:mysql-community-devel  ########################################### [ 50%]   5:mysql-community-embedde########################################### [ 63%]   6:mysql-community-embedde########################################### [ 75%]   7:mysql-community-server ########################################### [ 88%]   8:mysql-community-libs-co########################################### [100%][root@master01 cdh_install]# 

--重设置密码

5.7.9重设置密码有人会问，为什么user没有password字段了。从5.6之后就没有了会在  /var/log/mysqld.log  最后生成一个随机密码，供初始登录。[root@master01 mysql]# more /var/log/mysqld.log 160203 14:40:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql2016-02-03T06:40:45.397816Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).2016-02-03T06:40:45.399265Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.9) starting as process 36627 ...2016-02-03T06:40:45.407902Z 0 [Note] InnoDB: PUNCH HOLE support available2016-02-03T07:28:24.812608Z 0 [Note] /usr/sbin/mysqld: ready for connections.Version: '5.7.9'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)2016-02-03T07:50:54.238157Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).2016-02-03T07:50:54.708935Z 0 [Warning] InnoDB: New log files created, LSN=457902016-02-03T07:50:54.780046Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.2016-02-03T07:50:54.840130Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: da97342a-ca4a-11e5-acfb-000c29c14e9c.2016-02-03T07:50:54.841328Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.2016-02-03T07:50:54.845873Z 1 [Note] A temporary password is generated for root@localhost: C*s!.=y6<-OC[root@master01 mysql]#    mysql -uroot -ppassword:

--或者直接跳过 权限控制，修改密码 （我通常采用这种）

[root@master01 mysql]# mysqld_safe --skip-grant-tables --user=mysql & [1] 37043[root@master01 mysql]# 160203 15:59:11 mysqld_safe Logging to '/var/log/mysqld.log'.160203 15:59:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysqlmysqlWelcome to the MySQL monitor.  Commands end with ; or \g.Your MySQL connection id is 2Server version: 5.7.9 MySQL Community Server (GPL)Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> show databases;+--------------------+| Database           |+--------------------+| information_schema || mysql              || performance_schema || sys                |+--------------------+4 rows in set (0.00 sec)mysql> mysql> use mysqlERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.mysql> SET PASSWORD = PASSWORD('123456'); Query OK, 0 rows affected, 1 warning (0.00 sec)mysql>  FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec)mysql> mysql> mysql> exitBye

--我们登陆试试

[root@master01 mysql]# mysql -uroot -pEnter password: Welcome to the MySQL monitor.  Commands end with ; or \g.Your MySQL connection id is 6Server version: 5.7.9 MySQL Community Server (GPL)Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> exit搞定。

--重启mysqld

[root@master01 mysql]# vi /etc/my.cnf  --调整一些参数[root@master01 mysql]# sudo /sbin/chkconfig mysqld on[root@master01 mysql]# sudo /sbin/chkconfig --list mysqldmysqld         0:off1:off2:on3:on4:on5:on6:off[root@master01 mysql]# sudo service mysqld startStarting mysqld:                                           [  OK  ]

--把jar包拷贝到对应目录下，改为为mysql-connector-java.jar

[root@master01 share]# mkdir -p /usr/share/java[root@master01 share]# sudo cp /it/cdh_install/mysql-connector-java-5.1.38-bin.jar  /usr/share/java/mysql-connector-java.jar[root@master01 share]# cd /usr/share/java[root@master01 java]# lsmysql-connector-java.jar [root@master01 share]#

--建立所需用户

mysql> create database amon DEFAULT CHARACTER SET utf8;Query OK, 1 row affected (0.00 sec)mysql> grant all PRIVILEGES on amon.* TO ' amon'@'%' IDENTIFIED BY ' 123456' WITH GRANT OPTION;Query OK, 1 row affected (0.00 sec) 省略（省略create database ....GRANT ALL PRIVILEGES ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;GRANT ALL PRIVILEGES ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;grant all PRIVILEGES on metastore.* TO 'hive'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;grant all PRIVILEGES on sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456'  WITH GRANT OPTION;grant all PRIVILEGES on nav.* TO 'nav'@'%' IDENTIFIED BY '123456'  WITH GRANT OPTION;grant all PRIVILEGES on navms.* TO 'navms'@'%' IDENTIFIED BY '123456'  WITH GRANT OPTION;）mysql> mysql> flush privileges;Query OK, 0 rows affected (0.00 sec)

--备份mysql

[root@master01 java]# mysqldump  -uroot -p mysql > /tmp/mysql-bakup.sqlEnter password: [root@master01 java]# cd /tmp/[root@master01 tmp]# lshsperfdata_root  mysql-bakup.sql  pulse-ahmL5Yupk46M  ssh-XKnrmy7U7AfMkeyring-NkHAG2   orbit-root       pulse-Nij3doz66iv1  yum.log

----建立scm

[root@master01 schema]# pwd/opt/cloudera-manager/cm-5.4.1/share/cmf/schema[root@master01 schema]# ./scm_prepare_database.sh mysql scm scm 123456JAVA_HOME=/it/jdk1.7.0_07Verifying that we can write to /opt/cloudera-manager/cm-5.4.1/etc/cloudera-scm-serverCreating SCM configuration file in /opt/cloudera-manager/cm-5.4.1/etc/cloudera-scm-serverExecuting:  /it/jdk1.7.0_07/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.4.1/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cloudera-manager/cm-5.4.1/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.Wed Feb 03 17:58:37 CST 2016 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.[                          main] DbCommandExecutor              INFO  Successfully connected to database.All done, your SCM database is configured correctly![root@master01 schema]# 

----把对应的文件拷贝到repo下，并启动cm-server

[root@master01 parcel-repo]# pwd/opt/cloudera/parcel-repo[root@master01 parcel-repo]# cp /it/cdh_install/CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel* manifest.json .[root@master01 parcel-repo]# ls -ltotal 1244320-rw-r--r-- 1 root root 1274129311 Feb  3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel-rw-r--r-- 1 root root         41 Feb  3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha-rw-r--r-- 1 root root      42445 Feb  3 18:04 manifest.json[root@master01 parcel-repo]# ls -ltotal 1244276-rw-r--r-- 1 root root 1274129311 Feb  3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel-rw-r--r-- 1 root root         41 Feb  3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha[root@master01 parcel-repo]# cd /opt/cloudera-manager/cm-5.4.1/etc/init.d/[root@master01 init.d]# lscloudera-scm-agent  cloudera-scm-server  cloudera-scm-server-db[root@master01 init.d]# ls -ltotal 20-rwxr-xr-x 1 1106 4001 5241 May  9  2015 cloudera-scm-agent-rwxr-xr-x 1 1106 4001 6105 May  9  2015 cloudera-scm-server-rwxr-xr-x 1 1106 4001 2894 May  9  2015 cloudera-scm-server-db[root@master01 init.d]# ./cloudera-scm-server startStarting cloudera-scm-server:                              [  OK  ][root@master01 init.d]# 

--查看启动日志

[root@master01 cloudera-scm-server]# tailf cloudera-scm-server.log 2016-02-03 18:02:51,753 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Initializing SearchTemplateManager:2016-02-03T10:02:51.753Z2016-02-03 18:02:51,790 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating entities:2016-02-03T10:02:51.790Z2016-02-03 18:02:51,801 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num entities:1142016-02-03 18:02:51,801 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating documents:2016-02-03T10:02:51.801Z2016-02-03 18:02:51,838 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:1262016-02-03 18:02:51,840 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2016-02-03T10:02:51.839Z2016-02-03 18:02:52,118 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.42016-02-03 18:02:52,120 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:71802016-02-03 18:02:52,120 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.CM-server就搞定了

-- 启动cm-agent 

先检查包 [root@master01 init.d]# rpm -q  chkconfig  python   bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsbchkconfig-1.3.49.3-2.el6.x86_64python-2.6.6-29.el6_2.2.x86_64package bind-utils is not installedpsmisc-22.6-15.el6_0.1.x86_64libxslt-1.1.26-2.el6.x86_64zlib-1.2.3-27.el6.x86_64sqlite-3.6.20-1.el6.x86_64package cyrus-sasl-plain is not installedpackage cyrus-sasl-gssapi is not installedpackage fuse is not installedpackage portmap is not installedpackage fuse-libs is not installedpackage redhat-lsb is not installed[root@master01 init.d]# 

--yum安装所需要的包

[root@master01 init.d]# yum install chkconfig  python   bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsbLoaded plugins: product-id, refresh-packagekit, subscription-manager 

--启动代理

[root@master01 init.d]# ./cloudera-scm-agent startStarting cloudera-scm-agent:                               [  OK  ][root@master01 init.d]# 

--可以登录界面，默认用户密码为admin/admin

这里有140，141，142三台机器，我们这里先安装140=master01  141=slave01，

 

142等安装好后，作为新节点添加。（测试添加节点）