centos7.2 安装hdp2.6.1.0

centos7.2 安装hdp2.6.1步骤
环境：
Operating System: CentOS Linux release 7.2.1511 (Core)
Kernel: Linux 3.10.0-327.el7.x86_64
HDP:2.6.1.0
Ambari:ambari-server-2.5.0.3


下载地址：

ambari-2.5.0.3 tar包下载地址：http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/ambari-2.5.0.3-centos7.tar.gz

HDP-2.6.1.0 tar包下载地址：http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.1.0/HDP-2.6.1.0-centos7-rpm.tar.gz

MD5:HDP-2.6.1.0-centos7-rpm.tar.gz: 48 3F C7 17 19 4A 83 94  82 48 70 7C D4 85 62 14

HDP-UTILS 2.6.10 tar包下载地址：http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz
MD5:HDP-UTILS-1.1.0.21-centos7.tar.gz: B6 1E 3F 3D D9 10 51 8D  86 3D 44 FC 59 15 57 52

0.所有主机安装jdk并替换jce文件(若启用kerberos)
# tar -zxvf /data/jdk-8u111-linux-x64.tar.gz -C /usr/
# unzip /data/UnlimitedJCEPolicyJDK8.zip
# cp -f /data/UnlimitedJCEPolicyJDK8/local_policy.jar /usr/jdk1.8.0_111/jre/lib/security/
# cp -f /data/UnlimitedJCEPolicyJDK8/US_export_policy.jar /usr/jdk1.8.0_111/jre/lib/security/

# vi /etc/profile
末尾添加
export JAVA_HOME=/usr/jdk1.8.0_111
export PATH=$JAVA_HOME/bin:$PATH
#. /etc/profile

1.ssh信任配置(略)

2.ntp配置
2.1 安装并开启ntp
yum install -y ntp
2.2 开启ntp
systemctl is-enabled ntpd
2.3 vi /etc/ntp.conf
注释其他server，添加本地ntp服务器
server x.x.x.x iburst
2.4 
systemctl enable ntpd
systemctl start ntpd
systemctl status ntpd.service 检查服务是否正常

3.主机名配置
hostnamectl set-hostname server_name
查看主机信息
[root@eva0 yum.repos.d]# hostnamectl
   Static hostname: eva0
         Icon name: computer-vm
           Chassis: vm
        Machine ID: b5b4a6e010034d50axxxxxxxxxxxxxxx
           Boot ID: 818af0fc48c64931bxxxxxxxxxxxxxxx
    Virtualization: vmware
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-514.10.2.el7.x86_64
      Architecture: x86-64

4.配置/etc/hosts，不要更改下述行，否则会引起各种问题
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
eva0 x.x.x.x
eva1 x.x.x.x
eva2 x.x.x.x
之后hostname -f检查是否配置成功

5.所有主机关闭防火墙
# systemctl disable firewalld
# service firewalld stop

6.关闭selinux，packagekit(如果已启用)
vi /etc/yum/pluginconf.d/refresh-packagekit.conf
enabled=0

7.更改umask
# umask 0022
# echo umask 0022 >> /etc/profile

8. 建立HDP local repository
8.0 启动http服务
# systemctl start httpd.service
# systemctl status httpd.service

8.1 下载并解压HDP软件包
tar -zxvf HDP-2.6.1.0-centos7-rpm.tar.gz -C /var/www/html
mkdir -p /var/www/html/HDP-UTIL
tar -zxvf HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/HDP-UTIL/
tar -zxvf ambari-2.5.0.3-centos7.tar.gz -C /var/www/html

检查通过http是否可访问到资源
Ambari Base URL
http://1eva0/ambari/centos7/ambari/
HDP Base URL
http://1eva0/HDP/centos7/
HDP-UTILS Base URL
http://1eva0/HDP-UTIL/

8.2 安装createrepo包

yum install createrepo

8.3 创建repo
createrepo /var/www/html/ambari/centos7/
createrepo /var/www/html/HDP/centos7
createrepo /var/www/html/HDP-UTIL
(注意createrepo的目录)

8.4 修改*.repo文件
# more /etc/yum.repos.d/ambari.repo
#VERSION_NUMBER=2.5.1.0
[ambari-2.5.1.0]
name=ambari-2.5.1.0
baseurl=http://eva0ambari/centos7/ambari
gpgcheck=0
#gpgkey=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/RPM-GPG-KEY/RPM-GPG
-KEY-Jenkins
enabled=1
priority=1

# more /etc/yum.repos.d/HDP.repo
#VERSION_NUMBER=2.6.1.0-129
[HDP-2.6.1.0]
name=HDP Version - HDP-2.6.1.0
baseurl=http://eva0HDP/centos7
gpgcheck=0
gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.1.0/RPM-GPG-KEY/RPM-GPG-KEY
-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.21]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.21
baseurl=http://eva0HDP-UTIL
gpgcheck=0
gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.1.0/RPM-GPG-KEY/RPM-GPG-KEY
-Jenkins
enabled=1
priority=1

9 建立centos7 local repository
9.0 挂载软件iso镜像
mkdir -p /var/www/html/os7
mount -o loop /data/centos7.iso /var/www/html/os7

9.1 制作repository
方法一：
mkdir -p /var/www/html/centos7
直接将Packages目录及其下的软件包copy到/var/www/html/centos7
方法二：
mkdir -p /data/centos7
将Packages目录及其下的软件包copy到/data/centos7
ln -s /data/centos7 /var/www/html/centos7 

9.2 创建repo(以方法一为例)
createrepo /var/www/html/centos7

9.3修改repo文件
#vi /etc/yum.repos.d/CentOS-Base.repo
[base]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
baseurl=http://133.37.126.170/centos7/Packages/
gpgcheck=0
#gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enable=1
priority=1
注意enable=1，gpgcheck设置为0

10. 检查当前repository库
先修改repo目录的权限
chmod -R 777 /var/www/html/centos7
chmod -R 777 /var/www/html/ambari
chmod -R 777 /var/www/html/HDP
chmod -R 777 /var/www/html/HDP-UTIL

# yum repolist
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
源标识                               源名称                                      状态
HDP-2.6.1.0            HDP Version - HDP-2.6.1.0                               232
HDP-UTILS-1.1.0.21     HDP-UTILS Version - HDP-UTILS-1.1.0.21                   64
ambari-2.5.0.3         ambari Version - ambari-2.5.0.3                          12
base                   CentOS-7 - Base                                       3,723
repolist: 4,031

重要：检查无误后，将三个repo文件复制到所有节点上
并检查其他节点的/etc/yum.conf文件中是否设置了proxy代理，如果有应暂时取消，以避免可能无法访问到做repository的主机。

11.安装ambari server
选择eva0主机安装：
注意ambari-server依赖postgresql，当前2.5.0.3版本需要postgresql-libs(x86-64) = 9.2.13-1.el7_1，如果os中已经安装不同版本的pg，先卸载再安装指定版本
[root@eva0 ~]# yum erase postgresql-libs-9.2.18-1.el7.x86_64
[root@eva0 ~]# yum install ambari-server

12.安装mysql服务器
安装
yum localinstall  MySQL-server-5.6.14-1.el6.x86_64.rpm
yum localinstall MySQL-devel-5.6.14-1.el6.x86_64.rpm
yum localinstall  MySQL-client-5.6.14-1.el6.x86_64.rpm
配置
vi /etc/my.cnf
[mysqld]
#basedir=/usr/share/mysql
datadir=/data/mysql/data
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
skip-name-resolve
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/lib/mysql/eva0.pid
[client]
port=3306
socket=/var/lib/mysql/mysql.sock

启动
mysql_install_db --user=mysql --defaults-file=/etc/my.cnf
./mysql.server start
mysqladmin -u root password 'yyyyy'

13.设置ambari-server
# ambari-server setup
使用默认用户
Customize user account for ambari-server daemon [y/n] (n)?
使用自行安装的jdk
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
输入JAVA_HOME
Path to JAVA_HOME: /usr/jdk1.8.0_111
至于ambari的元数据存储位置，可以自定义，如选择mysql，也可直接使用默认的pg，测试环境建议直接默认，
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
所有设置采用默认
到此步时不要继续，先完成下述两步骤：
13.1 将mysql的jdbc 的dirver jar放到/usr/share/java目录下，并在ambari.properties文件中添加server.jdbc.driver.path=[path/to/custom_jdbc_driver]属性
(可从https://dev.mysql.com/downloads/connector/j/ 网站下载)
新开窗口执行：
copy driver到指定目录
tar -zxvf /data/mysql-connector-java-5.1.42.tar.gz
cp /data/mysql-connector-java-5.1.4/mysql-connector-java-5.1.42-bin.jar /usr/share/java/
修改ambari配置文件
vi /etc/ambari-server/conf/ambari.properties
添加
server.jdbc.driver.path=/usr/share/java/mysql-connector-java-5.1.42-bin.jar

13.2 运行ambari配置sql脚本
新开窗口执行：
[root@eva0 ~]# mysql -uroot -pyyyyy </var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
注：如果报错，可根据脚本内容先创建db，用户，修改use库的行并取消注释
--------------------------------------
# CREATE DATABASE `ambari` /*!40100 DEFAULT CHARACTER SET utf8 */;
# CREATE USER 'ambari' IDENTIFIED BY 'bigdata';
 USE ambari;
----------------------------------------

13.3 回到原窗口，继续配置过程，即可成功完成配置
Configuring ambari database...
Copying JDBC drivers to server resources...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)?
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

14.启动ambari-server，通过其安装配置hdp软件包
ambari-server start

15启动报错解决：
15.01查看ambari日志定位错误
ava.sql.SQLException: Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED.
实际与mysql配置有关，需要更改bin-log format
查看当前binlogformat
mysql> select @@binlog_format;
+-----------------+
| @@binlog_format |
+-----------------+
| STATEMENT       |
+-----------------+
1 row in set (0.01 sec)

当前服务修改，mysql重启后失效
SET SESSION binlog_format = 'MIXED' ;
SET GLOBAL binlog_format = 'ROW';
永久修改
vi /etc/my.cnf
添加
binlog-format = ROW

15.02 ambari用户无读取db的权限
ERROR [main] DBAccessorImpl:117 - Error while creating database accessor
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Access denied for user 'ambari'@'%' to database 'ambari'
am

# mysql -uroot -p
Enter password:
mysql> grant all privileges on ambari.* to 'ambari'@'%' ;
mysql> flush privileges;

16 登陆ambari web 进行配置
初始密码 admin/admin
若忘记密码，可在后台更改：
mysql> use ambari;
mysql> update users set user_password=PASSWORD('12345') where user_id=1;
mysql> flush privileges;

17配置中可能存在的问题
17.1 配置hive metastore的存储数据库时指定已有mysql数据库，
需先在ambari-server上执行：
# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java-5.1.42-bin.jar
再在mysql中执行以下命令创建hive对应db，用户和赋权
mysql>   create database hive;
mysql> create user hive identified by 'yyyyy';
Query OK, 0 rows affected (0.01 sec)

mysql> grant all privileges on hive.* to 'hive'@'%';
Query OK, 0 rows affected (0.01 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.02 sec)

mysql> update mysql.user set password=PASSWORD('yyyyy') where user='hive';
Query OK, 2 rows affected (0.01 sec)
Rows matched: 3  Changed: 2  Warnings: 0

mysql> flush privileges;

Query OK, 0 rows affected (0.01 sec)

17.2 ambari自动安装hadoop软件时失败，日志显示libtirpc-devel缺失：
如果机器上已经有libtirpc软件包，找到对应版本的 libtirpc-devel安装即可
如果找不到对应版本建议卸载现有libtirpc包，重装版本匹配的libtirpc和libtirpc-devel
# rpm -qa|grep libtirpc
libtirpc-0.2.4-0.6.el7.x86_64

非对应版本无法安装
# rpm -ivh libtirpc-devel-0.2.4-0.8.el7.x86_64.rpm
错误：依赖检测失败：
libtirpc = 0.2.4-0.8.el7 被 libtirpc-devel-0.2.4-0.8.el7.x86_64 需要

卸载现有版本
# yum erase libtirpc-0.2.4-0.6.el7.x86_64

安装匹配版本
# yum localinstall libtirpc-0.2.4-0.8.el7.x86_64.rpm
# yum localinstall libtirpc-devel-0.2.4-0.8.el7.x86_64.rpm

备注：

关闭THP方法：

“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和

“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此设置，

然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置