Cloudera在Ubuntu的安装部署

1. 安装前的准备

 准备好Ubuntu离线source，配置/etc/apt/source.list
 准备好Cloudera Hadoop离线source，配置/etc/apt/source.list.d/cloudera.list
 同步集群的/etc/hosts
 ssh免密码登录，cloudera-scm-server所在机器能够免密码登录其他cloudera-scm-agent机器或者所有机器的root用户密码相同

2. 系统环境

OS：ubuntu12.04
CDH：cdh3u6
Cloudera manager：4.8.2

3. 数据库的配置

3.1 安装配置MySQL

Cloudera Manager支持三种数据库类型：MySQL、PostgreSQL、Oracle。我们选择使用MySQL。
apt-get install mysql-server
service mysql start
在/etc/mysql/conf.d下创建mysql_cloudera_manager.cnf文件，加入以下内容：
[mysqld]
transaction-isolation=READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0 
key_buffer              = 16M
key_buffer_size         = 32M
max_allowed_packet      = 16M
thread_stack            = 256K
thread_cache_size       = 64
query_cache_limit       = 8M
query_cache_size        = 64M
query_cache_type        = 1
# Important: see Configuring the Databases and Setting max_connections
max_connections         = 550 
# log-bin should be on a disk with enough free space
log-bin=/x/home/mysql/logs/binary/mysql_binary_log 
# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format           = mixed 
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M 
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size          = 64M
innodb_buffer_pool_size         = 4G
innodb_thread_concurrency       = 8
innodb_flush_method             = O_DIRECT
innodb_log_file_size = 512M 
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

3.2 配置innodb

mv /var/lib/mysql/ib_logfile* ~
重启mysql
service mysql restart

3.3 安装mysql-connector

apt-get install libmysql-java

3.4 创建数据库

需要创建的数据库列表：
amon smon ran hmon hive
mysql -u root -p iflytek
create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
create database smon DEFAULT CHARACTER SET utf8;
grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password';
create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';
create database hmon DEFAULT CHARACTER SET utf8;
grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password';
create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';
备份数据库
mysqldump -piflytek amon > /tmp/amon-backup.sql
和其他集群数据库

3.5 安装Cloudera Manager

添加Cloudera Manager的源
创建/etc/apt/source.list.d/cloudera-manager.list文件，并在文件中添加
deb [arch=amd64] http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh precise-cdh4 contrib
deb-src http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh precise-cdh4 contrib
deb http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm precise-cm4 contrib
deb-src http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm precise-cm4 contrib
保存后下载key
curl -s http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/archive.key | sudo apt-key add -
若上述命令执行失败，请自己下载key文件导入。apt-key add [file path]
在Server机器上执行
apt-get install cloudera-manager-daemons cloudera-manager-server 
在agent机器上执行
apt-get install cloudera-manager-agent cloudera-manager-daemons

3.6 为cloudera manager准备数据库

执行/usr/share/cmf/schema/scm_prepare_database.sh database-type [options] database-name username password
如
./scm_prepare_database.sh mysql amon root xxx
./scm_prepare_database.sh mysql smon root xxx
./scm_prepare_database.sh mysql hmon root xxx
./scm_prepare_database.sh mysql rman root xxx
./scm_prepare_database.sh mysql hive root xxx

3.7 安装HDFS

指定一台机器做namendoe
apt-get install namenode
在指定的DataNode机器上执行
apt-get install datanode

3.8 启动cloudera manager Server和agent

在所有机器上创建一些保存日志的目录：
mkdir -p /var/log/cloudera-scm-headlamp
mkdir -p /var/log/cloudera-scm-firehose
mkdir -p /var/log/cloudera-scm-alertpublisher
mkdir -p /var/log/cloudera-scm-eventserver
mkdir -p /var/lib/cloudera-scm-headlamp
mkdir -p /var/lib/cloudera-scm-firehose
mkdir -p /var/lib/cloudera-scm-alertpublisher
mkdir -p /var/lib/cloudera-scm-eventserver
创建完成后修改owner
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-*
chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-*
如果使用tar包安装需要chown到cloudera-scm用户组下。如果使用apt-get方式安装不需要chown。默认使用的是root。
修改所有agent的JDK配置 
在/etc/default/ cloudera-scm-agent中增加
export CMF_AGENT_JAVA_HOME=/usr/local/jdk6
在Server机器上执行：
service cloudera-scm-server start
分别登录agent机器执行：
service cloudera-scm-agent start

3.9 登录web端

登录部署有scm-server机器的http://ServerIP:7180
用户名：admin
密码：admin

3.10 数据库设置

数据库主机名称：localhost，用户名：root，数据库名称上面那5个，密码：xxxx，即MySQL数据库的密码。

4. web端使用

apt-get install oracle-j2sdk1.6

http://www.sjsjw.com/kf_system/article/023067ABA027185.asp

http://blog.csdn.net/ganglia/article/details/21632217

附上另一种离线安装方法：

http://www.tuicool.com/articles/AnuiUra

http://www.mysjtu.com/page/M0/S940/940210.html

http://www.cnblogs.com/thinkCoding/p/3567408.html

官网安装用户手册

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-Installation-Guide/Cloudera-Manager-Installation-Guide.html

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/Cloudera-Manager-Installation-Guide/cmig_install_path_C.html

ubuntu系列：
http://archive.cloudera.com/cm4/cm/4/
http://archive.cloudera.com/search/parcels/latest/
http://archive.cloudera.com/impala/parcels/latest/
http://archive.cloudera.com/cdh4/parcels/latest/
http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm/pool/contrib/o/oracle-j2sdk1.6/