CDH Hadoop CM方式安装

来源：互联网发布：洛枫网络软件公司编辑：程序博客网时间：2024/04/30 22:44

CDH Hadoop安装

(基于Cloudera Manager半在线方式)

1 CM简介

Cloudera Manager是一个端到端的管理CDH集群的工具。它通过管理CDH集群（组件安装、服务管理、参数监控、问题诊断、性能监控）各个环节从而增强集群管理的质量, 增强规范性,降低运营成本。

2 准备

2.1 环境说明

1 该安装方式依赖如下linux组件:httpd, mysql

2 安装时需要用到其它系统组件,请配置好本地源

3 安装mysql是请确保清理好系统自带的mysql.

4 CM 下载地址见下,需要下载如下的所有文件.详细目录分布见2.2.6说明.

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/RPMS/x86_64/

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/repodata/

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo

5 CDH parcel下载地址http://archive-primary.cloudera.com/cdh5/parcels/5.5.1/

6 CM installer下载地址

http://archive-primary.cloudera.com/cm5/installer/5.5.1/cloudera-manager-installer.bin

注:

1 CM 核心安装文件截图:

2 CDH parcel相关文件截图:

3 CM installer截图:

2.1.1 虚拟机里Linux系统版本

[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux

Red Hat Enterprise Linux Server release 6.5 (Santiago)

2.1.2 JDK版本

[root@localhost native]# java -version

java version "1.7.0_67"

注:官网强调一定是1.7以上版本.

Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit ServerVM (build 24.65-b04, mixed mode)

2.1.3mysql版本

[root@localhost native]# mysql -uroot -proot-e"select version()";

Warning: Using a password onthe command line interface can be insecure.

+-----------+

| version() |

+-----------+

| 5.6.14 |

+-----------+

注:

1 Mysql 驱动为mysql-connector-java-5.1.31-bin.jar

2 该jar需要放置在/usr/share/cmf/lib/内被CM安装时用到

2.2 安装准备

2.2.1 禁用防火墙

service iptables stop

chkconfig iptables off

2.2.2 禁用Selinux

vi /etc/selinux/config

#修改内容见下:

SELINUX=disabled

SELINUXTYPE=targeted

2.2.3 修改主机名

vi/etc/hosts

#加上archive.cloudera.com域名

192.168.56.101 archive.cloudera.com shenl

vi/etc/sysconfig/network

#修改主机名如下:

HOSTNAME=archive.cloudera.com

#重启电脑使得更改生效.

2.2.4 配置本地源

1) 虚拟机加载Linux ISO镜像

2) 找到rom对应设备名

[root@localhost ~]# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sr0 11:0 1 1024M 0 rom

sr1 11:1 1 3.6G 0 rom

sr2 11:2 1 1024M 0 rom

sda 8:0 0 8G 0 disk

?..sda1 8:1 0 500M 0 part /boot

?..sda2 8:2 0 7.5G 0 part

?..VolGroup-lv_root (dm-0) 253:0 0 6.7G 0 lvm /

?..VolGroup-lv_swap (dm-1) 253:1 0 816M 0 lvm [SWAP]

3) 创建目录

[root@localhost ~]#mkdir -p /mnt/cdrom/

4) 挂载镜像

mount -t iso9660 /dev/sr1 /mnt/cdrom

5) 配置linux本地更新源(真实环境这里的应该baseurl应配置成http方式,如http://ios)

vi /etc/yum.repos.d/redhat.repo

#修改内容如下:

[rhel-source]

name=Redhat

baseurl=file:///mnt/cdrom/

enabled=1

gpgcheck=1

gpgkey=file:///mnt/cdrom//RPM-GPG-KEY-redhat-release

6) 更新更新源

yum clean all

yum makecache

yum update list

2.2.5 安装配置mysql

1) 删除系统自带的mysql

rpm -qa | grep -i mysql

MySQL-client-5.1.49-1.glibc23

rpm -e --nodeps MySQL-client-5.1.49-1.glibc23

chkconfig --list | grep -i mysql

mysql 0:off 1:off 2:on 3:on 4:on 5:on 6:off

chkconfig --del mysql

whereis mysql

mysql: /usr/lib/mysql /usr/share/mysql

rm -rf /usr/lib/mysql/

rm -rf /usr/share/mysql

2) 安装Mysql服务、客户端

rpm -ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-shared-compat-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-client-5.6.14-1.el6.x86_64.rpm

3) 启动mysql服务

service mysql start

4) 修改mysql初始密码,先找到安装时的初始密码,在修改成自己的密码

[root@localhost ~]#cat /root/.mysql_secret

# The random password set for the root user at Tue Dec 2221:17:22 2015 (local time):RUmKBqcY

mysql -uroot -p RUmKBqcY

set password=password(‘root’)

注: 如果想修改mysql默认的datadir,假设到/data/mysqldata内,可参考如下方式修改:

#迁移mysql数据目录,做软连接,修改目录的所有者

mv /var/lib/mysql//data/mysqldata/mysql

ln -s /data/mysqldata/mysql/var/lib/

chown -R mysql:mysql/data/mysqldata/mysql

2.2.6 安装httpd,并按照如下的目录结构放置CM和CDH parcel的安装文件.

yum install tree

yum install httpd

service httpd start

cd /var/www/html

注:如果想修改httpd的网站默认主目录,可通过如下方式:

mkdir -p /data/www/

chown -R apache:apache/data/www/

vi /etc/httpd/conf/httpd.conf

#找到DocumentRoot 修改"/var/www/html"成"/data/www"

service httpd restart

报如下错:

Forbidden

You don't have permission to access /cm/ on this server.

vi /etc/httpd/conf/httpd.conf

#找到 <Directory"/var/www/html"> 修改成 /data/www

service httpd restart

#CM的目录安排需要按照如下方式:

[root@archive html]# tree./cm5/

./cm5/

└── redhat

└── 6

└── x86_64

└── cm

├── 5.5.1

│ ├── repodata

│ │ ├── filelists.xml.gz

│ │ ├── filelists.xml.gz.asc

│ │ ├── other.xml.gz

│ │ ├── other.xml.gz.asc

│ │ ├── primary.xml.gz

│ │ ├── primary.xml.gz.asc

│ │ ├── repomd.xml

│ │ └── repomd.xml.asc

│ └── RPMS

│ └── x86_64

│ ├──cloudera-manager-agent-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

│ ├── cloudera-manager-daemons-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

│ ├──cloudera-manager-server-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

│ ├──cloudera-manager-server-db-2-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

│ ├──enterprise-debuginfo-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

│ ├── jdk-6u31-linux-amd64.rpm

│ └──oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

├── cloudera-manager.repo

└── RPM-GPG-KEY-cloudera

8 directories, 17 files

#CDH parcel的目录安排需要按照如下方式:

[root@archive html]#tree ./cdh5/

./cdh5/

└── parcels

└── 5.5.1

├── CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel

├── CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel.sha1

└── manifest.json

2 directories, 3 files

#将CM安装脚本放置/var/www/html(可放置在其它目录,这里为了统一).:

chmod +x cloudera-manager-installer.bin

注:下载repodata文件夹时可以在Window里下载好,执行wget命令

E:\ProgramFiles\wget-1.16.3-win64>wget -r -np -ndhttp://archive-primary.cloude

ra.com/cm5/redhat/6/x86_64/cm/5.5.1/repodata/

3安装

3.1安装中

1) 执行CM安装脚本

./cloudera-manager-installer.bin

#分析并解决CM安装过程中的问题,详细见 3.2安装问题:

cat/var/log/cloudera-manager-installer/2.install-oracle-j2sdk1.7.log

2) 输入CM所在的IP地址和7180端口,进行界面化安装CDH.

#选择存储库时需要点击parcel旁边的更多选项,删除默认定义的路径,写上

http://192.168.56.101/cdh5/parcels/5.5.1

或者

http://archive.cloudera.com/cdh5/parcels/5.5.1

#这时如果选择此Cloudera Manager Server的匹配发行版,会报错,想见3.2安装问题之d).这里建议选择自定义存储库.见下图:

2) 按照提示一步步安装,直至集群安装成功…

注: CM在安装时会自动生成cloudera-manager.repo,请确认里面的url跟自定义存储库里的一致,建议走http协议.cloudera-manager.repo里默认下载地址见下:

https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1

3.2 安装问题

1) 正确配置CM和CDH的目录树结构

2) 正确配置parcel和自定义存储库

a) Loaded plugins: product-id,security, subscription-manager

This system is not registered to Red Hat SubscriptionManagement. You can use subscription-manager to register.

http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/repomd.xml:[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 NotFound"

解决方案:重命名文件名,详细见下:

cd /var/www/html

mv ./cm5/redhat/6/x86_64/cm/5.5.1./cm5/redhat/6/x86_64/cm/5

b) .

解决方案:重命名文件名,详细见如下:

mv ./cm5/redhat/6/x86_64/cm/5 ./cm5/redhat/6/x86_64/cm/5.5.1

c) CMinstaller正常安装相关日志

[root@archive cloudera-manager-installer]# pwd

/var/log/cloudera-manager-installer

[root@archivecloudera-manager-installer]# ll

total 20

-rw-r--r-- 1 root root 0 Jan 4 15:51 0.check-selinux.log

-rw-r--r-- 1 root root 0 Jan 4 15:51 1.install-repo-pkg.log

-rw-r--r-- 1 root root 1647 Jan 4 15:51 2.install-oracle-j2sdk1.7.log

-rw-r--r-- 1 root root 2080 Jan 4 15:53 3.install-cloudera-manager-server.log

-rw-r--r-- 1 root root 0 Jan 4 15:46 3.remove-cloudera-manager-repository.log

-rw-r--r-- 1 root root 3250 Jan 4 15:53 4.install-cloudera-manager-server-db-2.log

-rw-r--r-- 1 root root 1978 Jan 4 15:54 5.start-embedded-db.log

-rw-r--r-- 1 root root 59 Jan 4 15:54 6.start-scm-server.log

d) 按照错误提示修改成b)后仍然报如下错误

BEGIN yum info cloudera-manager-agent
Loaded plugins: product-id, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can usesubscription-manager to register.
https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/repodata/repomd.xml:[Errno 14] PYCURL ERROR 7 - "couldn't connect to host"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository:cloudera-manager. Please verify its path and try again

解决方案:这时需要在群集安装→选择存储库里选择自定义存储库,配置见下图: