Ubuntu16.04 离线安装 Cloudera CDH 5.12.1

这是一篇是关于搭建Ubuntu CDH 的一篇详细教程，CDH hadoop 集群搭建比较麻烦，但是相对于 Apache hadoop在安装hadoop生态的时候 有着极度便利特点，所以这篇文章，带大家手把手上手Ubuntu CDH hadoop

安装准备

1，CDH manager 的下载url，选择cloudera-manager-xenial-cm5.12.1_amd64.tar.gz

2，parcel，sha1，manifest.json的下载url，选择parcel中的对应版本

3，Oracle java8

4，mysql-connector-java-5.1.41-bin.jar

以上为所需提前准备的安装包

更改机器hostname，和hosts

vim /etc/hostname
node1
vim /etc/hosts
10.34.2.1 node1
10.34.2.2 node2
10.34.2.3 node3
10.34.2.4 node4
10.34.2.5 node5

准备CDH需要的目录

mkdir /opt/cloudera-manager
mkdir -p /opt/cloudera/parcel-repo
mkdir -p /opt/cloudera/parcels
mkdir /var/log/cloudera-scm-headlamp
mkdir /var/log/cloudera-scm-firehose
mkdir /var/log/cloudera-scm-alertpublisher
mkdir /var/log/cloudera-scm-eventserver
mkdir /var/lib/cloudera-scm-headlamp
mkdir /var/lib/cloudera-scm-firehose
mkdir /var/lib/cloudera-scm-alertpublisher
mkdir /var/lib/cloudera-scm-eventserver
mkdir /var/lib/cloudera-scm-server

配置ssh无密访问

生成密钥

ssh-keygen

分发到集群每台机器（分发到每台机器，这里取2台为例）

ssh-copy-id root@node1
ssh-copy-id root@node2

将下载的java包解压存放路径

新建路径

mkdir /usr/lib/jvm

解压

tar -zxvf jdk-8u121-linux-x64.tar -C /usr/lib/jvm

配置环境变量

Java1.8 environment variable configuration

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_121/export JRE_HOME=/usr/lib/jvm/jdk1.8.0_121/jreexportCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/jre/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$PATH

关闭防火墙(所有节点)

ufw disable
reboot

配置mysql-connector-java(所有节点)

cp /opt/mysql-connector-java-5.1.41-bin.jar /usr/share/java/

安装Anaconda

wget下载

wget -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/Anaconda3-2.5.0-Linux-x86_64.sh

修改权限

chmod 777 Anaconda3-2.5.0-Linux-x86_64.sh

执行shell脚本

./Anaconda3-2.5.0-Linux-x86_64.sh

添加环境变量

export PATH=/usr/local/anaconda2/bin:$PATH

注:

也可以使用apt install 安装python

apt install python-minimal

ntp的安装与配置

安装ntp

apt install ntp

在配置文件末尾添加

vim /etc/ntp.conf
server 2.cn.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 127.0.0.1
restrict -6 default kod notrap nomodify

重启ntp

service ntp restart

mysql 安装

注：只有一台机器装server，所有机器都要装client

apt install mysql-server
apt install mysql-client
apt install libmysqlclient-dev

修改mysql配置和用户权限等问题

允许mysql允许远程访问，注释掉127.0.0.1

vim /etc/mysql/mysql.conf.d/mysqld.cnf
bind-address = 127.0.0.1

授予权限,集群内其他机器也能访问

password 替换为数据库密码

grant all on . to root@’%’ identified by ‘password’ with grant option;
flush privileges;

重启mysql

service mysql restart

若启动失败，输入指令查看原因

journalctl -xe

关于mysql 一些操作汇总

mysql遇到不明了的问题，可以看这篇转载 Ubuntu 16.04 安装使用MySQL

创建数据库

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
flush privileges;

安装 准备CDH manager 安装程序包

tar -zxvf /opt/all_software_path/cloudera-manager*.tar.gz -C /opt/cloudera-manager

更改代理服务主服务器名称或ip

修改config.ini 的master_name=node1

注：每台机器的agent都要修改指向，node1 为自己的server机器

vim /opt/cloudera-manager/cm-5.12.1/etc/cloudera-scm-agent/config.ini
master_name=node1

验证更改是否成功

grep “server_[host|port]” /opt/cloudera-manager/cm-5.12.1/etc/cloudera-scm-agent/config.ini

mysql jdk的驱动程序

cp /opt/mysql-connector-java-5.1.41-bin.jar /opt/cloudera-manager/cm-5.12.1/share/cmf/lib/

初始化数据库

数据库名称：cdh 用户名:name 密码:password

/opt/cloudera-manager/cm-5.12.1/share/cmf/schema/
scm_prepare_database.sh mysql -uroot -p –scm-host % cdh name password

成功给的反馈

All done, your SCM database is configured correctly

导入安装包

cp /opt/CDH-5.12.1-1.cdh5.12.1.p0.3-xenial.parcel* /opt/cloudera/parcel-repo

cp /opt/all_software_path/manifest.json /opt/cloudera/parcel-repo

mv CDH-5.12.1-1.cdh5.12.1.p0.3-xenial.parcel.sha1 CDH-5.12.1-1.cdh5.12.1.p0.3-xenial.parcel.sha

创建系统用户名

useradd –system –home=/opt/cloudera-manager/cm-5.12.1/run/cloudera-scm-server –no-create-home –shell=/bin/false –comment “Cloudera SCM User” cloudera-scm

更改文件夹权限

chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager
chown -R cloudera-scm:cloudera-scm /opt/cloudera
chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp
chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-firehose
chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-alertpublisher
chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-eventserver
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-headlamp
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-firehose
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-alertpublisher
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-eventserver
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server

安装依赖

apt-get install gedit
apt-get install ant
apt-get install apache2
apt-get install bash
apt install upstart
apt-get install debhelper
apt-get install fuse
apt-get install gcc
apt-get install libfuse2
apt-get install libsasl2-modules
apt-get install libsasl2-modules-gssapi-mit
apt-get install libsqlite3-0
apt-get install libssl-dev
apt-get install libxslt1.1
apt-get install lsb-base
apt-get install make
apt-get install openssl
apt-get install perl
apt-get install postgresql-client
apt-get install postgresql
apt-get install psmisc
apt-get install python-dev
apt-get install python-mysqldb
apt-get install python-psycopg2
apt-get install python-setuptools
apt-get install rpcbind
apt-get install sed
apt-get install swig
apt-get install zlib1g

集群指令

一台server，每台机器都起都得起agent

启动命令

/opt/cloudera-manager/cm-5.12.1/etc/init.d/cloudera-scm-server start
/opt/cloudera-manager/cm-5.12.1/etc/init.d/cloudera-scm-agent start

关闭命令

/opt/cloudera-manager/cm-5.12.1/etc/init.d/cloudera-scm-server stop
/opt/cloudera-manager/cm-5.12.1/etc/init.d/cloudera-scm-agent stop

管理界面

http://server的IP:7180
初始状态下:
user : admin
password : admin

到这一步CDH hadoop平台搭建完成，后续可以进行集群的个性化配置。