Cloudera Manager 安装 CDH5.x 心得

废话不多说，先展示下这几天捣鼓的成果

• Cloudera Manager 管理配置界面

• Hbase 管理界面及Hbase Web UI

• HDFS 管理界面 及 NameNode Web UI

• Hive 管理界面 及 HiveServer2 Web UI

• Hue 管理界面 及 Web UI

首次登录会提示让创建管理员账号和密码，笔者 admin admin

• Oozie 管理界面 及 Oozie Web UI

# Oozie  Web UI 报错，后期修复Oozie web console is disabled.To enable Oozie web console install the Ext JS library.参考：http://cdh01.aniu.so:11000/oozie/docs/DG_QuickStart.html

• YARN (MR2 Included) 管理界面 及 Web UI

• ResourceManager Web UI
  

• HistoryServer Web UI
  

• Zookeeper 管理界面

笔者这里zookeeper安装的时候选择的默认，因此只安装了一个zookeeper，但个人感觉后期应该需要增加zookeeper的界面数量

下面开始说安装的注事事项

• 1、配置环境要符合要求，要纯净的系统环境

# 笔者环境# CM env192.168.1.137 cdh01.aniu.so CentOS6.9 16G Memory 100G LVM卷 （Manger 节点）192.168.1.148 cdh02.aniu.so CentOS6.9 4G Memory 70G LVM卷192.168.1.149 cdh03.aniu.so CentOS6.9 4G Memory 70G LVM卷192.168.1.150 cdh04.aniu.so CentOS6.9 4G Memory 70G LVM卷建议小白参考笔者的环境配置，主机名可以自定义#对四个节点的系统进行更新，安装开发工具包yum update -y && yum -y groupinstall "Development Tools"

• 2、关闭防火墙、禁用Selinux

# 关闭防火墙/etc/init.d/iptables stop && /etc/init.d/ip6tables stopchkconfig iptables off && chkconfig ip6tables off# 建议采用修改内核参数的方式关闭ip6tables vim /etc/modprobe.d/dist.conf # 编辑此文件，在最后加入：# Disable ipv6alias net-pf-10 offalias ipv6 off# 禁用selinuxsed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/configsetenforce 0  # 不重启临时生效

• 3、内核参数调整

# 内存参数调整sysctl -w vm.swappiness=10 或者 编辑vim /etc/sysctl.conf，在最后加入：vm.swappiness = 10编辑启动项vim /etc/rc.local,最后加入：echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled

注：上面所有操作在所有节点都需要执行

• 4、所有节点间配置免密认证

# CM节点执行ssh-keygen -t rsa -b 2048 # 有确认提示，一直按回车即可cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keyschmod 600 ~/.ssh/authorized_keys# 笔者 hosts.conf# CM node192.168.1.137 cdh01.aniu.so192.168.1.148 cdh02.aniu.so192.168.1.149 cdh03.aniu.so192.168.1.150 cdh04.aniu.so# 同步密钥for ip in $(awk '{print $1}' hosts.conf );do scp ~/.ssh/authorized_keys root@$ip:/root/.ssh ;done或者使用ssh-copy-id [e-mail]-copy-id [e-mail]-copy-id [e-mail]-copy-id [e-mail]# 上面操作也需要在所有节点执行

• 5、使用cloudera-manger repo安装CM

# 在CM节点执行wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.dwget https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/cloudera-cdh5.repo -P /etc/yum.repos.d# yum clean all && yum makecache # 建议执行不强制yum install oracle-j2sdk1.7 -yyum install cloudera-manager-daemons cloudera-manager-server -y# 在其他节点执行wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.dyum install oracle-j2sdk1.7 -y# 配置JAVA_HOME编辑vim /etc/profileexport JAVA_HOME=/usr/java/jdk1.7.0_67-clouderaexport PATH=$JAVA_HOME/bin:$PATHexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar保存退出执行：source /etc/profile 使更改的环境变量生效# 在所有节点执行配置JAVA_HOME的操作

• 6、CM节点安装数据库，或使用已有的数据

# 笔者使用mysql57-community.repo,安装的mysql[mysql57-community]name=MySQL 5.7 Community Serverbaseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/6/$basearch/enabled=1gpgcheck=0gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql#yum install mysql-community-embedded  mysql-community-server  mysql-community-devel mysql-community-client -y# 笔者my.cnf[root@cdh01 yum.repos.d]# cat /etc/my.cnf[client]port        = 3306socket      = /var/lib/mysql/mysql.sock[mysqld]datadir     = /opt/mysqlsocket      = /var/lib/mysql/mysql.sock#skip-grant-tables skip-ssldisable-partition-engine-checkport        = 3306skip-external-lockingkey_buffer_size = 16Mmax_allowed_packet = 1Mtable_open_cache = 64sort_buffer_size = 512Knet_buffer_length = 8Kread_buffer_size = 256Kread_rnd_buffer_size = 512Kmyisam_sort_buffer_size = 8Mthread_cache_size = 8query_cache_size = 8Mtmp_table_size = 16Mperformance_schema_max_table_instances = 500explicit_defaults_for_timestamp = truemax_connections = 500max_connect_errors = 100open_files_limit = 8192log-bin=mysql-binbinlog_format=mixedserver-id   = 1expire_logs_days = 10early-plugin-load = ""default_storage_engine          = InnoDBinnodb_file_per_table           = 1innodb_data_home_dir            = /opt/mysqlinnodb_data_file_path           = ibdata1:1024M;ibdata2:10M:autoextendinnodb_log_group_home_dir       = /opt/mysqlinnodb_buffer_pool_size         = 16Minnodb_log_file_size            = 5Minnodb_log_buffer_size          = 8Minnodb_flush_log_at_trx_commit  = 1innodb_lock_wait_timeout        = 50innodb_log_files_in_group       = 3 innodb_buffer_pool_size         = 12Ginnodb_log_file_size            = 512Minnodb_log_buffer_size          = 256Minnodb_flush_log_at_trx_commit  = 2innodb_lock_wait_timeout        = 150innodb_open_files               = 600innodb_max_dirty_pages_pct      = 50innodb_file_per_table           = 1[mysqldump]quickmax_allowed_packet = 16M[mysql]no-auto-rehash[myisamchk]key_buffer_size = 20Msort_buffer_size = 20Mread_buffer = 2Mwrite_buffer = 2M[mysqlhotcopy]xinteractive-timeoutsymbolic-links=0slow_query_loglong_query_time = 5slow_query_log_file = /var/log/mysql-slow.loglog-error = /var/log/mysqld.logpid-file = /var/run/mysqld/mysqld.pid# 初始化mysql，并设置启动数据库设置root密码/usr/sbin/mysqld --initialize --user=mysql --socket=/var/lib/mysql/mysql.sock # 先执行mysql_secure_installation # 再执行# 创建CM启动用到的数据库mysql -u root -pAniuops123. -e "create database cmf DEFAULT CHARACTER SET utf8;"mysql -u root -pAniuops123. -e "GRANT ALL PRIVILEGES ON `cmf`.* TO 'cmf'@'localhost' IDENTIFIED BY 'Aniunas123.'";"

• 启动cloudera-scm-server，并配置parcel

# 生成db配置文件/usr/share/cmf/schema/scm_prepare_database.sh mysql cmf cmf Aniucmf123.# 启动cloudera-scm-server/etc/init.d/cloudera-scm-server start # 查看启动日志# 配置parcel离线cd /opt/cloudera/parcel-repo/ # 然后下载wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcelwget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha1wget http://archive.cloudera.com/cdh5/parcels/latest/manifest.json# 注：读者根据cloudera当前CDH最新版本更改下载用到的URLmv CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha1 CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha # 强制执行、默认使用本地的parcels包，不更改sha1,cloudera-scm-server启动安装时会去cloudera官网找匹配的parcel安装包重启cloudera-scm-server，查看实时日志/etc/init.d/cloudera-scm-server restarttailf /var/log/cloudera-scm-server/cloudera-scm-server.log 

• 通过CM管理界面安装CDH，注意事项

# CM server启动成功即可通过http://192.168.1.137:7180访问，默认账户密码：admin admin# **重点内容** 下面的话很重要：不要勾选：单用户模式 ，笔者在此模式下安装多次都没成功，有心人可以测试

• 能一次性安装成功的最好，安装不成功建议多试几次，对初始化完成的虚拟机进行快照操作，便于恢复