Cloudera CM5.3.1&CDH-5.3.0集群安装

Cloudera Hadoop优缺点

优点：

1、友好的监控展示界面

2、集成了Spark-1.2.0、Impala-2.1.0等

缺点：

可控性较Apache版本弱

Cloudera Hadoop官网提供的安装方式

1、Installation Path A - Automated Installation by Cloudera Manager

使用cloudera-manager-installer.bin由Cloudera Manager自动安装
2、Installation Path B - Manual Installation Using Cloudera Manager Packages
手动安装Cloudera提供的各种rpm

3、Installation Path C - Manual Installation Using Cloudera ManagerTarballs

手动安装Cloudera提供的集成包

Tarballs contain both the Cloudera Manager Server and Cloudera Manager Agent in a single file. Download tarballs from the locations listed inCloudera Manager Version and Download Information.

实际安装目录安排

正式安装

一、安装准备

1.1、Install and Configure Databases
Cloudera Manager Server, Cloudera Management Service, and the Hive Metastore data is stored in a database.

service mysql status 

chkconfig mysqld on  设置mysql开机启动

create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
create database metastore DEFAULT CHARACTER SET utf8;
grant all on metastore.* TO 'metastore'@'%' IDENTIFIED BY 'metastore';
create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
create database nav DEFAULT CHARACTER SET utf8;
grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
create database navms DEFAULT CHARACTER SET utf8;
grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';
flush privileges; 

在主节点初始化CM5的数据库：
scm_prepare_database.sh mysql cm -h localhost -u root -p 123456 --scm-host localhost scm scm scm

1.2、(CDH 5 only) On RHEL 5 and CentOS 5, Install Python 2.6 or 2.7

1.3、安装jdk 1.7

1.4、时间同步

所有节点配置NTP服务

1.5、创建用户

每台主机必须手动创建cloudera-scm用户

useradd --system --home=/opt/cloudera-manager/cm-5.3.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

二、正式安装配置

2.1、Cloudera Manager安装

2.1.1、解压缩

mkdir /opt/cloudera-manager

tar xzf cloudera-manager*.tar.gz -C /opt/cloudera-manager
files could extract to /opt/cloudera-manager/cm-5.0/. This full path is needed later and is referred to as tarball_root directory.
export TARBALL_ROOT=/opt/cloudera-manager/cm-5.3.1

将mysql-connector-java-5.1.18-bin.jar拷贝到/opt/cloudera-manager/cm-5.3.1/share/cmf/lib下

2.1.2、配置agent

vi $TARBALL_ROOT/etc/cloudera-scm-agent/config.ini

# Hostname of the CM server.
#server_host=localhost
server_host=bigdata3

2.1.3、配置Cloudera使用的目录

mkdir -p /var/log/cloudera-scm-headlamp
mkdir -p /var/log/cloudera-scm-firehose
mkdir -p /var/log/cloudera-scm-alertpublisher
mkdir -p /var/log/cloudera-scm-eventserver
mkdir -p /var/lib/cloudera-scm-headlamp
mkdir -p /var/lib/cloudera-scm-firehose
mkdir -p /var/lib/cloudera-scm-alertpublisher
mkdir -p /var/lib/cloudera-scm-eventserver
mkdir -p /var/lib/cloudera-scm-server

chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp
chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-firehose
chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-alertpublisher
chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-eventserver
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-headlamp
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-firehose
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-alertpublisher
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-eventserver
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server

2.2、Cloudera Hadoop安装

创建parcel目录

mv /opt/cloudera-manager/cloudera /opt
chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

使用Parcels安装CDH

设置开机启动（不只是下面的步骤，待看官网）
cp $TARBALL_ROOT/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
cp $TARBALL_ROOT/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent

2.3、启动Cloudera Manager

$TARBALLS/etc/init.d/cloudera-scm-server start

$TARBALLS/etc/init.d/cloudera-scm-agent start

三、安装验证

http://bigdata3:7180

netstat -ant | grep 7180

四、其他

1、MySQL--存储元数据

Cloudera Manager Server, Cloudera Management Service, and the Hive Metastore data is stored in a database.

the databases for Cloudera Manager Server, Activity Monitor, Reports Manager, Cloudera Navigator, and Hive metastore
Creating Databases for Activity Monitor, Reports Manager, Hive Metastore Server, Sentry Server, Cloudera Navigator Audit Server, and Cloudera Navigator Metadata Server

2、本地yum源
you must create a custom repository for the RPM or DEB Files.

五、一些网址

【总介绍】
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_intro_primer.html
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_installing_configuring_dbs.html?scroll=concept_vck_m3m_hn_unique_2#concept_mff_xjm_hn_unique_1

http://archive-primary.cloudera.com/cdh5/cdh/5

Cloudera提供的Apache下各种项目的安装包

http://archive.cloudera.com/cdh5/parcels/

CDH的parcel文件

【安装文档（离线Tarballs方式安装）】
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_install_path_c.html

http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v5-latest/Cloudera-Manager-Installation-Guide/cm5ig_install_path_C.html
【Cloudera Manager Version and Download Information（CM各个版本下载信息）】
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_vd.html
可在上面的页面中找到【CM的下载地址】：
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.3.1_x86_64.tar.gz

【Creating and Using a Package Repository】
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_create_local_package_repo.html

【Installation Path A - Automated Installation by Cloudera Manager】
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_install_path_a.html?scroll=cmig_topic_6_5_1_unique_4
【配置MySQL】
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_mysql.html#cmig_topic_5_5

六、问题记录

1、勿忘改变用于存放data的目录权限

chown -R cloudera-scm:cloudera-scm /data/hdfs/data1
chown -R cloudera-scm:cloudera-scm /data/hdfs/data2
chown -R cloudera-scm:cloudera-scm /data/hdfs/data3
chown -R cloudera-scm:cloudera-scm /data/hdfs/data4
chown -R cloudera-scm:cloudera-scm /data/hdfs/data5