hdp2.5离线安装笔记

来源：互联网发布：如何申请淘宝号编辑：程序博客网时间：2024/05/16 14:49

说明：

1、由于自己是一个初学者，中间有很多地方不明白、且花费了较多时间，虽然看了很多说明，但大部分说明还是忽略了一些步骤、或者按此实施后，并未解决问题。

2、在成功安装hdp后，写此说明予以共享，希望能对其他新学者提供帮助。

3、由于本人已经完成安装，因此说明中的有些图是借用他人，信息有一点出入，但不影响总体操作。

4、参照了以下文件：

http://www.linuxidc.com/Linux/2016-12/138142.htm

http://www.yp14.cn/2016/12/09/Centos7-2%E5%AE%89%E8%A3%85Ambari2-4-2-HDP2-5-3%E6%90%AD%E5%BB%BAHadoop%E9%9B%86%E7%BE%A4/

http://www.cnblogs.com/roger888/p/5505731.html

http://www.jianshu.com/p/73f9670f71cf

特此感谢！

一、基础环境准备：

1. 环境准备

安装2台集群

192.168.9.202 cluster1.hadoop

192.168.9.107 cluster2.hadoop

如果mysql数据库要自己安装，还需要一台用于安装数据库，否则，就随ambari一起安装。

2. 关闭防火墙和SELinux(先关闭，避免互访、连接端口不通，每台服务器都需要)

$systemctl disable firewalld

$systemctl stop firewalld

临时关闭，不用重启机器：

$setenforce 0

修改配置文件让机器重启也生效：

$cat /etc/sysconfig/selinux

SELINUX=disabled

3. 配置host、设置主机名

Ø 配置host:

[root@cluster1java]# vi /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4localhost4.localdomain4

::1 localhost localhost.localdomainlocalhost6 localhost6.localdomain6

192.168.9.202 cluster1.hadoop

192.168.9.107 cluster2.hadoop

同步到107服务器:

scp/etc/hosts root@192.168.9.107:/etc/

Ø 设置主机名（两台机器均需要设定）：

vi/etc/sysconfig/network

NETWORKING=yes

HOSTNAME=cluster1.hadoop

GATEWAY=192.168.9.1

--然后在终执行 hostname

hostnamecluster1.hadoop

chkconfigiptables off

/etc/init.d/iptablesstop

4. 安装ntp，使时间同步

yum install -y ntp

chkconfig --list ntpd

chkconfig ntpd on

service ntpd start

5. 配置SSH免密码登录

主节点里root用户登录执行如下步骤

$ ssh-keygen

$ cd ~/.ssh/

$ cat id_rsa.pub >>authorized_keys

$ chmod 600 ~/.ssh/authorized_keys

把各个主机上公钥id_rsa.pub文件内容放到各个主机authorized_keys中

scp authorized_keys root@cluster2.hadoop:~/.ssh/authorized_keys

6. 安装JDK1.8，配置环境变量:

Ø 安装：

Wget http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.rpm?AuthParam=1500617988_f05200e439ff9b31b70121ad4012c21a

会安装在/usr/java/jdk1.8.0_141中。

Ø 配置环境变量：

vi /etc/profile

在文件尾部增加如下配置：

export JAVA_HOME=/usr/java/jdk1.8.0_141

exportPATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

exportCLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib安装mysql jdbc驱动

保存后，执行source使变量生效

source /etc/profile

Java –version

$ yum installmysql-connector-java -y

7. 安装mysqljdbc驱动

$ yum install mysql-connector-java –y

8. 检查或修改umask 设置

umask

umask 0022

-- 如果执行umask 返回的已经是 0022 的话，就不需要执行下面的语句了，否则执行下面的语句，永久保存 umask 属性。

echo umask 0022 >> /etc/profile

9. Check the Maximum OpenFile Descriptors

ulimit -Sn

ulimit -Hn

-- 如果上面语句的输出少于 10000，建议重新配置一下：

ulimit -n 10000

二、制作本地源 (在cluster1.hadoop部署)

1. 下载相应的安装包：

ambari-2.4.2.0-centos6.tar.gz

HDP-2.5.3.0-centos6-rpm.tar.gz

HDP-UTILS-1.1.0.21-centos6.tar.gz

2. 将安装包解压到相应的目录下：

Tarxvfz ****** /home/hadoop/

如下：

3. 安装Nginx，做为web资源服务器

Ø 修改nginx.conf

server {

listen 80;

server_name localhost;

root /home/hadoop;

#charset koi8-r;

#access_log logs/host.access.log main;

location / {

root /home/hadoop;

index index.html index.htm;

autoindex on;

autoindex_exact_size off;

autoindex_localtime on;

}

#error_page 404 /404.html;

# redirect server error pages to thestatic page /50x.html

error_page 500 502 503 504 /50x.html;

location = /50x.html {

root html;

}

# proxy the PHP scripts to Apachelistening on 127.0.0.1:80

#location ~ \.php$ {

# proxy_pass http://127.0.0.1;

# pass the PHP scripts to FastCGIserver listening on 127.0.0.1:9000

#location ~ \.php$ {

# root html;

# fastcgi_pass 127.0.0.1:9000;

# fastcgi_index index.php;

# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;

# include fastcgi_params;

# deny access to .htaccess files, ifApache's document root

# concurs with nginx's one

#location ~ /\.ht {

# deny all;

}

Ø 启动Nginx

/usr/local/nginx/sbin/nginx &

在地本电脑浏览器输入：http://192.168.9.202/，如出现如下内容，则表示本地资源制作成功；如未出现，则需要查看nginx的配置是否正确：

4. 制作公共库文件

Ø 下载公共库文件：

ambari.repo:

Wget http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.2.0/ambari.repo

hdp.repo：Wget http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.3.0/hdp.repo

Ø 拷贝到Linux的系统目录/etc/yum.repos.d/，并分别修改其中的内容：

[root@cluster1 yum.repos.d]# viambari.repo

#VERSION_NUMBER=2.4.2.0-136

[Updates-ambari-2.4.2.0]

name=ambari-2.4.2.0- Updates

#baseurl=http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.2.0

baseurl=http://cluster1.hadoop/AMBARI-2.4.2.0/centos6/2.4.2.0-136

gpgcheck=1

#gpgkey=http://public-repo-1.hortonworks.com/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

gpgkey=http://cluster1.hadoop/AMBARI-2.4.2.0/centos6/2.4.2.0-136/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

[root@cluster1 yum.repos.d]# vihdp.repo

#VERSION_NUMBER=2.5.3.0-37

[HDP-2.5.3.0]

name=HDPVersion - HDP-2.5.3.0

#baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0

#baseurl=file:///home/hadoop/HDP-2.5.3.0/centos6

baseurl=http://cluster1.hadoop/HDP-2.5.3.0/centos6

gpgcheck=1

#gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

#gpgkey=file:///home/hadoop/HDP-2.5.3.0/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

gpgkey=http://cluster1.hadoop/HDP-2.5.3.0/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

[HDP-UTILS-1.1.0.21]

name=HDP-UTILSVersion - HDP-UTILS-1.1.0.21

#baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7

#baseurl=file:///home/hadoop/HDP-UTILS-1.1.0.21/repos/centos6

baseurl=http://cluster1.hadoop/HDP-UTILS-1.1.0.21/repos/centos6

gpgcheck=1

#gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

#gpgkey=file:///home/hadoop/HDP-UTILS-1.1.0.21/repos/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

gpgkey=http://cluster1.hadoop/HDP-UTILS-1.1.0.21/repos/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

Ø 验证：

yum clean all

yum list|grep ambari

如果可以看到Ambari的对应版本的安装包列表，说明公共库已配置成功。然后就可以安装Ambari的package了。

结果如下：

Ø 同步公共库文件到其他节点服务器上

scp /etc/yum.repos.d/ambari.repo root@192.168.9.107:/etc/yum.repos.d/

5. 安装插件、并修改配置

yum -y install yum-plugin-priorities

vi/etc/yum/pluginconf.d/priorities.conf

[main]

enabled=1

gpgcheck=0

scp/etc/yum/pluginconf.d/priorities.conf root@192.168.9.107:/etc/yum/pluginconf.d/

三、安装MySql数据库

数据库如果是随平台一起安装，这里将不用管。否则，可以参考

http://www.yp14.cn/2016/12/09/Centos7-2%E5%AE%89%E8%A3%85Ambari2-4-2-HDP2-5-3%E6%90%AD%E5%BB%BAHadoop%E9%9B%86%E7%BE%A4/

中的指引。

四、安装ambari-server：

1. 安装

yum install ambari-server –y

2. 配置

ambari-server setup

在这个交互式的设置中，采用默认配置即可。Ambari会使用Postgres数据库，默认会安装并使用Oracle的JDK。默认设置了Ambari GUI的登录用户为admin/admin。并且指定Ambari Server的运行用户为root。

备注（详细的设置过程）：

[root@hdp1 centos6]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account forambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):ambari
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on allhosts.
WARNING: JCE Policy files are required for configuring Kerberos security. Ifyou plan to use Kerberos,please make sure JCE Unlimited Strength JurisdictionPolicy Files are valid on all hosts.
Path to JAVA_HOME:/opt/java/jdk1.8.0_92
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced databaseconfiguration [y/n] (n)? Y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
==============================================================================
Enter choice (1): 1
Database name (ambari):
Postgres schema (ambari):
Username (ambari): admin
Enter Database Password (bigdata):
Re-enter password:
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Running initdb: This may take upto a minute.
Initializing database: [ OK ]

About to start PostgreSQL
Configuring local database...
Connecting to local database...done.
Configuring PostgreSQL...
Restarting PostgreSQL
Extracting system views...
.ambari-admin-2.2.2.0.460.jar
.....
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup'completed successfully.
[root@hdp1 centos6]#

3. 启动ambari-server（root同样可以启成功）

[root@hdp1 centos6]# su - ambari
[ambari@hdp1 ~]$ ambari-server start
Using python /usr/bin/python
Starting ambari-server
Unable to check PostgreSQL server status when starting without root privileges.
Please do not forget to start PostgreSQL server.
Organizing resource files at /var/lib/ambari-server/resources...
Unable to check firewall status when starting without root privileges.
Please do not forget to disable or adjust firewall if needed
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start....................
Ambari Server 'start' completed successfully.

4. 检验：

后台进程检验

[ambari@hdp1 ~]$ ambari-server status
Using python /usr/bin/python
Ambari-server status
Ambari Server running
Found Ambari Server PID: 4376 at: /var/run/ambari-server/ambari-server.pid
[ambari@hdp1 ~]$

web ui 检验

http://cluster1.hadoop:8080

出现如下界面，表示安装成功

五、安装ambari-agent

1. 碰到的问题

不安装，在网页上进行集群配置时，将不能成功，日志如下：

Command starttime 2017-07-24 19:01:42

scp/usr/lib/python2.6/site-packages/ambari_server/setupAgent.py

host=cluster2.hadoop,exitcode=0

Commandend time 2017-07-24 19:01:43

==========================

Runningsetup agent script...

==========================

Commandstart time 2017-07-24 19:01:43

Host registration aborted. Ambari Agent host cannot reachAmbari Server 'localhost:8080'. Please check the network connectivity betweenthe Ambari Agent host and the Ambari Server

Connectionto cluster2.hadoop closed.

SSHcommand execution finished

host=cluster2.hadoop,exitcode=1

Commandend time 2017-07-24 19:01:49

ERROR:Bootstrap of host cluster2.hadoop fails because previous action finished withnon-zero exit code (1)

ERRORMESSAGE: tcgetattr: Invalid argument

Connectionto cluster2.hadoop closed.

STDOUT:Host registration aborted. Ambari Agent host cannot reach Ambari Server'localhost:8080'. Please check the network connectivity between the AmbariAgent host and the Ambari Server

Connectionto cluster2.hadoop closed.

日志中的意思是agent不能和server建立连接；网上有多种解决方法，其中的一种看似最管用

http://zorro.blog.51cto.com/2139862/1409468

但是按此操作后，然后重启ambari-server还是不行。

如果不恢复此修改，在完成了ambari-agent安装后，将会报如下错误：

Traceback(most recent call last):

File"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py",line 35, in <module>

BeforeAnyHook().execute()

File"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",line 280, in execute

method(env)

File"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py",line 32, in hook

setup_java()

File"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py",line 181, in setup_java

raise Fail(format("Unable to access{java_exec}. Confirm you have copied jdk to this host."))

resource_management.core.exceptions.Fail:Unable to access /usr/jre1.8/bin/java. Confirm you have copied jdk to thishost.

Error:Error: Unable to run the custom hook script ['/usr/bin/python','/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py','ANY', '/var/lib/ambari-agent/data/command-907.json','/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY','/var/lib/ambari-agent/data/structured-out-907.json', 'INFO','/var/lib/ambari-agent/tmp']

很多安装说明上，往往忽略了agent的安装和启动，因此本步一定要执行，且不需要修改/usr/lib/python2.6/site-packages/ambari_server/setupAgent.py；如果因此修改了，一定要恢复，注释掉一样会报错。

2. 安装：

# yuminstall ambari-agent -y

# vi/etc/ambari-agent/conf/ambari-agent.ini

[server]

#改为控制节点的机器名（IP也可）

hostname=cluster1.hadoop

url_port=8440

secured_url_port=8441

scp /etc/ambari-agent/conf/ambari-agent.ini\

root@ cluster2.hadoop:/etc/ambari-agent/conf/ambari-agent.ini

3. 启动agent

#ambari-agent start

VerifyingPython version compatibility...

Usingpython /usr/bin/python

Checkingfor previously running Ambari Agent...

Startingambari-agent

Verifyingambari-agent process status...

AmbariAgent successfully started

Agent PIDat: /run/ambari-agent/ambari-agent.pid

Agent outat: /var/log/ambari-agent/ambari-agent.out

Agent logat: /var/log/ambari-agent/ambari-agent.log

六、网页配置

1、出现登录界面，默认管理员账户登录，账户：admin密码：admin

2、登录成功后出现下面的界面，至此Ambari的安装成功

3、安装安装HDP2.5.3配置集群（点击上面登录成功页面的Launch Install Wizard 按钮进行集群配置）,设置集群名称

4、置HDP安装源（选择HDP2.5 ,并且设置Advanced Repository Options 的信息，本次使用本地源，所以修改对用系统的安装源为本地源地址）

选择useLocal Repository

系统是redhat6，因此remove其它项，分别填写如下资源地址：

HDP-2.5: http://192.168.9.202/HDP-2.5.3.0/centos6/

HDP-UTILS-1.1.0.21：http://192.168.9.202/HDP-UTILS-1.1.0.21/repos/centos6/

5、设置集群机器（下载cluster1.hadoop机器上的id_rsa私钥到本地）

Targethosts中填写：

cluster1.hadoop

cluster2.hadoop

6、Host确认（确认前面配置集群中hosts列表中的机器是否都可用，也可以移除相关机器，集群中机器Success后进行下一步操作）

如果cluster2.hadoop中的ambari-agent没有安装，则cluster2.hadoop将无法配置成功，即会出现“五.安装ambari-agent”中的问题。

7、选择要安装的服务

8、AssignMasters给各节点组件分配资源，这个自己看着来就行

9、Assign Slaves and Clients 配置从节点和客户端。注意最后一列的Client指的是各种客户端，而不是某一种

10、服务的客制化配置
这一步里大部分都可以使用默认项，少部分需要改动和设置，如Hive和Oozie需要指定数据库密码等

红色数字标志表示这一项有需要手动设置的内容。

此外，如果Hive和Oozie使用Mysql的话，还需要先安装MySql,然后创建hive,oozie用户并赋予权限,当然还有hive、oozie数据库

11、显示配置信息

12、开始安装（安装各个服务，并且完成安装后会启动相关服务，安装过程比较长，如果中途出现错误，请根据具体提示或者log进行操作）

13、全部安装成功界面如下

14、安装完成

阅读全文

0 0