Hadoop2.6.5/Spark1.6.3 HA集群构建
来源:互联网 发布:linux文件实时同步ftp 编辑:程序博客网 时间:2024/05/17 02:06
在基于Spark的大数据开发中,我们确定使用Centos6.5,在intellij IDEA中使用Scala语言进行开发。Spark集群包含9台机器,其中,六台作为Worker执行任务,2台作为Master管理集群资源,一台作为Client提交任务,Worker集群的配置和Master、Worker的配置可以不同,具体内容如下:
Worker硬件环境:
内存:1G
硬盘:1T
网络:1Gbit
CPU:1核
Master硬件环境:
内存:2G
硬盘:1T
网络:1Gbit
CPU:1核
软件环境:
操作系统:CentOS 6.5
root用户密码:124077
sparker用户密码:sparker
1 规划部署
一共9台机器,HA架构:2台Master,6台Worker,1台Client
1.1 网络规划:
主机名: sparker001 IP: 192.168.1.101 网关 192.168.1.254
主机名: sparker002 IP: 192.168.1.102 网关 192.168.1.254
主机名: sparker003 IP: 192.168.1.103 网关 192.168.1.254
主机名: sparker004 IP: 192.168.1.104 网关 192.168.1.254
主机名: sparker005 IP: 192.168.1.105 网关 192.168.1.254
主机名: sparker006 IP: 192.168.1.106 网关 192.168.1.254
主机名: sparker007 IP: 192.168.1.107 网关 192.168.1.254
主机名: sparker008 IP: 192.168.1.108 网关 192.168.1.254
主机名: sparker009 IP: 192.168.1.109 网关 192.168.1.254
1.2 修改主机名,配置hosts
(所有节点)
编辑 vi /etc/hosts文件(每台机器均需要编辑)
检查一下(通过拼主机名看可否通信):
2 关闭防火墙 (所有节点)
检查防火墙运行状态: service iptables status永远关闭防火墙-重启后生效 chkconfig iptables off
重启系统后,再次检查防火墙运行状态:
service iptables status
3 关闭SELinux(所有节点)
检查SELinux状态 getenforce
将 SELINUX=enforcing 修改为 disabled vi /etc/selinux/config
重启系统之后,再次检查SELINUX状态,显示为Disabled
4 SSH 免密码登陆 配置
4.1 开启RSA密码验证(所有节点)
vim /etc/ssh/sshd_config
去掉注释,放开这几个配置项
4.2 生成公钥和私钥(所有节点)
说明一下 : 这个公钥私钥的生成需要和 对应的 集群使用用户 目录对应
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
参数说明:-t -P-f
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
进入到master的.ssh目录下面:
cd .ssh
scp authorized_keys
192.168.1.102:~/.ssh/authorized_keys_from_master
分别进入到worker机器的.ssh目录
cat authorized_keys_from_master >> authorized_keys
修改authorized_keys权限
。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。
5 安装JDK(所有节点)
5.1 卸载Centos6.5自带的OpenJDK
查看包:rpm -qa | grep jdk
卸载:rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64rpm -e --nodeps tzdata-java-2013g-1.el6.noarch
5.2 安装Oracle JDK
tar包的下载与解压:略修改 /etc/profile文件:在尾部加上如下几句:
# houjingru 2017 07.09export JAVA_HOME=/home/sparker/cluster/java/jdk1.7.0_80export PATH=$PATH:/home/sparker/cluster/java/jdk1.7.0_80/binexport CLASSPATH=.:/home/sparker/cluster/java/jdk1.7.0_80/jre/lib
立即生效
source /etc/profile
6 安装Zookeeper
6.1 下载解压(略)
6.2 新建zoo.cfg
cd zookeeper-3.4.6cp zoo_sample.cfg zoo.cfg
6.3修改配置文件
# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial # synchronization phase can takeinitLimit=10# The number of ticks that can pass between # sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just # example sakes.dataDir=/home/sparker/cluster/zookeeper-3.4.10/dataserver.1=sparker001:2888:3888server.2=sparker002:2888:3888server.3=sparker009:2888:3888# the port at which the clients will connectclientPort=2181# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the # administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to "0" to disable auto purge feature#autopurge.purgeInterval=1~
6.4 新建data文件夹
/home/sparker/zookeeper/zookeeper3.4.10/下新建data文件夹,然后在该文件夹下新建文件,文件名为myid,向其中加入server.1/2/9中的值,此处填1(只填数字).
6.5 将安装完成的Zookeeper分发到其他机器
[sparker@sparker001 zookeeper]$ scp -r zookeeper-3.4.10 192.168.1.102/home/sparker/cluster/zookeeper/[sparker@sparker001 zookeeper]$ scp -r zookeeper-3.4.10 192.168.1.109:/home/sparker/cluster/zookeeper/
修改102/109上中/data/myid的值.与server后面的值对应。在102中myid修改为2,在109中myid修改为9.
6.6 修改/etc/profile文件
export ZOOKEEPER_HOME=/home/sparker/cluster/zookeeper/zookeeper-3.4.10export PATH=$PATH:$ZOOKEEPER_HOME/bin
立即生效:source /etc/profile
6.7 软件启动
在每一台机器上(sparker001,sparker002,sparker009)上使用命令:zkServer.sh start 启动Zookeeper。
7 Hadoop安装
7.1 安装包下载解压
略
7.2 修改配置文件
配置IO常用的core-site.xml文件:
由于配置的为HDFS的HA,有两个namenode,所以需要给整个集群起一个逻辑名称,此处为hdfs://hjrCluster。使用hadoop.tmp.dir配置了Hadoop的默认临时存储目录。使用ha.zookeeper.quorum配置Zookeeper的三个节点。
vi /home/sparker/cluster/hadoop/hadoop-2.6.5/etc/hadoop/core-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://hjrCluster</value></property><property><name>hadoop.tmp.dir</name><value>/home/sparker/tmp</value></property><property><name>ha.zookeeper.quorum</name><value>sparker001:2181,sparker002:2181,sparker009:2181</value></property><property><name>ha.zookeeper.session-timeout.ms</name><value>1000</value><description>ms</description></property></configuration>
vi /home/sparker/cluster/hadoop/hadoop-2.6.5/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration><property><name>dfs.namenode.name.dir</name><value>/home/sparker/dfs/name</value></property><property><name>dfs.datanode.data.dir</name><value>/home/sparker/cluster/data0/dfs/data,/home/sparker/cluster/data1/dfs/data,/home/sparker/cluster/data2/dfs/data</value></property><property><name>dfs.replication</name><value>3</value></property><property><name>dfs.webhdfs.enabled</name><value>true</value></property><property><name>dfs.nameservices</name><value>hjrCluster</value><description>Logical name for this newnameservice</description></property><!--# dfs.ha.namenodes.[nameservice ID] -为在nameservice中的每一个NameNode# 设置唯一标示符。配置一个逗号分隔的 NameNode ID列表。这将是被DataNode识别为所# 有的NameNode。例如,如果使用“hjrCluster”作为nameservice ID,并且使# 用“nn1”和“nn2”作为NameNodes标示符.--><property><name>dfs.ha.namenodes.hjrCluster</name><value>nn1,nn2</value><description>Unique identifiers for each NameNode in thenameservice</description><!--# fs.namenode.rpc-address.[nameservice ID].[name node ID] - 每个# NameNode监听的完整正确的RPC地址,对于先前配置的NameNode ID,设置全地址和IP端# 口的NameNode进程--><property><name>dfs.namenode.rpc-address.hjrCluster.nn1</name><value>sparker001:8020</value></property><property><name>dfs.namenode.rpc-address.hjrCluster.nn2</name><value>sparker002:8020</value></property><property><name>dfs.namenode.servicerpc-address.hjrCluster.nn1</name><value>sparker001:53310</value></property><property><name>dfs.namenode.servicerpc-address.hjrCluster.nn2</name><value>sparker002:53310</value></property><!--# dfs.namenode.http-address.[nameservice ID].[name node ID] -每个# NameNode监听的完整正确的HTTP地址,和上面设置rpc-address一样,设置NameNode# 的HTTP服务--><property><name>dfs.namenode.http-address.hjrCluster.nn1</name><value>sparker001:50070</value></property><property><name>dfs.namenode.http-address.hjrCluster.nn2</name><value>sparker002:50070</value></property><!--# dfs.namenode.shared.edits.dir - NameNode读/写edits的URL,为JNs服务.这个配置是为了JournalNodes配置,提供edits的共享存储地址,这个地址被活动状态的nameNode写入被备份状态的nameNode读取用来保存活动状态写入的整个文件系统的最新的改变。尽管你必须指定几个JournalNode地址,但是你只要一个配置为这些URL,这个URI应该是这样的“qjournal://host1:port1;host2:port2;host3:port3/journalId”,Journal ID是这个nameservice的唯一标示。它允许一套JournalNodes为多个namesystems提供存储。虽然不是必须的,但是建议重用nameservice ID 作为journal 的标示。例如如果JournalNodes正运行在"node1.example.com", "node2.example.com", 和"node3.example.com",并且集群的nameservice ID是 mycluster,你应该用下面作为设置(JournalNode 默认的端口是8485)--><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://sparker003:8485;sparker004:8485;sparker005:8485;sparker006:8485;sparker007:8485/hjrCluster</value></property><property><name>dfs.client.failover.proxy.provider.hjrCluster</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property><!--# sshfence -利用SSH连接到NameNode服务器并杀死进程,sshfence选项SSH连接到目标节点,并使用fuser杀死进程。为了这个过滤选项可以工作,他需要通过通过SSH的无秘钥登陆到目标节点,那么需要配置一个dfs.ha.fencing.ssh.private-key-files选项,他以逗号分隔.--><property><name>dfs.ha.fencing.methods</name><value>sshfence</value></property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/home/sparker/.ssh/id_rsa</value></property><property><name>dfs.ha.fencing.ssh.connect-timeout</name><value>30000</value></property><!--# dfs.journalnode.edits.dir -JournalNode守护线程报错他本地状态的路径,这是# 一个在JNs主机的绝对路径用来保存JNs使用的edits和其他本地状态。你可能只用一个单# 一的路径为这个配置。如果需要一些冗余的本分可以通过运行多个独立的JNs或者是使用# RAID实现.--><property><name>dfs.journalnode.edits.dir</name><value>/home/sparker/data/hadoop/journaldata</value></property><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property><property><name>ha.failover-controller.cli-check.rpc-timeout.ms</name><value>60000</value></property></configuration>
vi /usr/hadoop/hadoop-2.6.5/etc/hadoop/mapper-site.xml
<configuration>#管理框架<property><name>mapreduce.framework.name</name><value>yarn</value></property>#历史任务地址<property><name>mapreduce.jobhistory.address</name><value>sparker001:10020</value></property><property><name>mapreduce.map.memory.mb</name><value>1536</value></property>#map、reduce任务内存<property><name>mapreduce.map.java.opts</name><value>-Xmx1024M</value></property><property><name>mapreduce.reduce.memory.mb</name><value>3072</value></property><property><name>mapreduce.reduce.java.opts</name><value>-Xmx2560M</value></property>#网页监控历史作业地址<property><name>mapreduce.jobhistory.webapp.address</name><value>sparker001:19888</value></property></configuration>
yarn-site.xml
<configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value></property><property><name>yarn.resourcemanager.address</name><value>sparker001:8032</value></property><property><name>yarn.resourcemanager.scheduler.address</name><value>sparker001:8030</value></property><property><name>yarn.resourcemanager.resource-tracker.address</name><value>sparker001:8031</value></property><property><name>yarn.resourcemanager.admin.address</name><value>sparker001:8033</value></property><property><name>yarn.resourcemanager.webapp.address</name><value>sparker001:8088</value></property></configuration>
vi slaves
vi hadoop-env.sh
vi /etc/profile
source /etc/profile
7.3拷贝到其他节点
scp -r /home/sparker/cluster/hadoop/hadoop-2.6.5 192.168.1.10:2//home/sparker/cluster/hadoopscp -r /home/sparker/cluster/hadoop/hadoop-2.6.5 192.168.1.10:3//home/sparker/cluster/hadoopscp -r /home/sparker/cluster/hadoop/hadoop-2.6.5 192.168.1.10:4//home/sparker/cluster/hadoopscp -r /home/sparker/cluster/hadoop/hadoop-2.6.5 192.168.1.10:5//home/sparker/cluster/hadoopscp -r /home/sparker/cluster/hadoop/hadoop-2.6.5 192.168.1.10:6//home/sparker/cluster/hadoopscp -r /home/sparker/cluster/hadoop/hadoop-2.6.5 192.168.1.10:7//home/sparker/cluster/hadoopscp -r /home/sparker/cluster/hadoop/hadoop-2.6.5 192.168.1.108://home/sparker/cluster/hadoop
7.4 按照配置文件新建文件夹
再/home/sparker/下建立tmp文件夹和/dfs/data,/dfs/name文件夹。
mkdir /home/sparker/tmpmkdir /home/sparker/dfs/datamkdir /home/sparker/dfs/name
7.5 软件启动
(1)zookeeper格式化(仅仅第一步需要做)
hdfs zkfc -formactZK
(2)启动ZKFC
master节点以及standbymaster节点
hadoop-daemon.sh start zkfchadoop-daemon.sh stop zkfc
(3)JournalNode节点启动 《所有节点》
用于主备NN之间同步元数据信息的共享存储系统
所有节点均需要启动
hadoop-daemon.sh start journalnodehadoop-daemon.sh stop journalnode
(4)格式化namenode 【仅仅第一步需要做】
记录问题:如果需要重新格式化namenode,需要删除的目录有:
hdfs-site.xml:core-site.xml:
走起》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》》
hadoop namenode -format
(5)启动NN<此条命令在sparker001上执行>
hadoop-daemon.sh start namenodehadoop-daemon.sh stop namenode
(6)在备NN上同步主NN的元数据信息<此条命令在sparker002上执行>
hdfs namenode -bootstrapStandby
(7)启动备NN<此条命令在sparker002上执行>
在备NN上执行命令:hadoop-daemon.sh start namenode
在主NN节点上执行命令激活NN:hdfs haadmin -transitionToActive nn1
(8) 在主NN上启动Datanod<此条命令在sparker001上执行>
hadoop-daemon.sh start datanodee
7.6 验证查看
8 Spark安装
8.1 Scala安装
(1)下载解压:略
(2)修改/etc/profile文件
export SCALA_HOME=/home/sparker/cluster/scala/scala-2.10.5export PATH=$PATH:$SCALA_HOME/bin
(3)将Scala分发到集群的其他机机器
scp -r /home/sparker/cluster/scala/scala-2.10.5 192.168.1.102:/home/sparker/cluster/scala/ scp -r /home/sparker/cluster/scala/scala-2.10.5 192.168.1.103:/home/sparker/cluster/scala/ scp -r /home/sparker/cluster/scala/scala-2.10.5 192.168.1.104:/home/sparker/cluster/scala/ scp -r /home/sparker/cluster/scala/scala-2.10.5 192.168.1.105:/home/sparker/cluster/scala/ scp -r /home/sparker/cluster/scala/scala-2.10.5 192.168.1.106:/home/sparker/cluster/scala/ scp -r /home/sparker/cluster/scala/scala-2.10.5 192.868.1.107:/home/sparker/cluster/scala/ scp -r /home/sparker/cluster/scala/scala-2.10.5 192.168.1.108:/home/sparker/cluster/scala/
8.2 Spark 安装
Master:sparker001
StandbyMaster:sparker002
Worker:sparker003,sparker004,sparker005,sparker006,sparker007,sparker008
(0)下载解压:略
(1)配置Spark的环境变量
(2)修改spark-1.6.3-bin-hadoop2.6/conf下的slaves文件
cp slaves.template slavesvi slaves
## Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.## A Spark Worker will be started on each of the machines listed below.sparker003sparker004sparer005sparker006sparker007sparker008
(3)修改spark-1.6.3-bin-hadoop2.6/conf下的spark-env.sh
加入Java、Hadoop、Scala等的环境变量,配置worker的内存、CPU核数等,以及加入jdbc,odbc等的jar包路径。具体配置如下:
(4)将Spark分发到其他机器
scp -r /home/sparker/cluster/spark/spark-1.6.3-bin-hadoop2.6 192.168.1.102:/home/sparker/cluster/spark scp -r /home/sparker/cluster/spark/spark-1.6.3-bin-hadoop2.6 192.168.1.103:/home/sparker/cluster/spark scp -r /home/sparker/cluster/spark/spark-1.6.3-bin-hadoop2.6 192.168.1.104:/home/sparker/cluster/spark scp -r /home/sparker/cluster/spark/spark-1.6.3-bin-hadoop2.6 192.168.1.105:/home/sparker/cluster/spark scp -r /home/sparker/cluster/spark/spark-1.6.3-bin-hadoop2.6 192.168.1.106:/home/sparker/cluster/spark scp -r /home/sparker/cluster/spark/spark-1.6.3-bin-hadoop2.6 192.168.1.107:/home/sparker/cluster/spark scp -r /home/sparker/cluster/spark/spark-1.6.3-bin-hadoop2.6 192.168.1.108:/home/sparker/cluster/spark
8.3 Spark 启动
master:cd /home/sparker/cluster/spark/spark-1.6.3/sbin./start-all.sh
StandbyMaster:cd /home/sparker/cluster/spark/spark-1.6.3/sbin./start-master.sh
sparker001:8080
./spark-shell --master spark://sparker001:7077
- Hadoop2.6.5/Spark1.6.3 HA集群构建
- spark1.6+hadoop2.6集群HA搭建与参数解析
- Zookeeper + Hadoop2.6 集群HA + spark1.6完整搭建与所有参数解析
- Spark1.5 +Zookeeper HA 集群搭建
- hadoop2.2+HA 集群搭建
- hadoop2.6.0集群HA搭建
- hadoop2集群搭建+HA配置
- hadoop2.7.0-HA集群部署
- hadoop2.7.1HA集群部署
- hadoop2.7.2HA集群安装
- CENTOS7构建HA集群
- hadoop2.4.1集群安装spark1.1.0
- spark1.2.0+hadoop2.4.0集群环境搭建
- hadoop2.4+spark1.3.0集群安装
- hadoop2的HA集群简单配置
- Hadoop2.2.0集群搭建(HA,联邦,yarn)
- Hadoop2.5+HA+zk3.4.6集群搭建
- Hadoop2.7.0 HA高可靠性集群搭建
- Java设计模式——组合模式(Composite Pattern)
- 理解奈奎斯特定理和香农定理
- 1315: 游程编码
- android集成百度语音使用离线语音是报(-111)错误是解决办法
- Windows学习(011)--内核对象(二)
- Hadoop2.6.5/Spark1.6.3 HA集群构建
- ORB-SLAM(三)地图初始化
- Android入门介绍
- sublime初始vue的代码片段
- 《飞猪规则》 第二章 自由行及跟团游类商品发布规范
- Servlet线程安全性问题总结
- leetcode#551. Student Attendance Record I
- linux常用命令
- springboot中集成jsp,打成jar包可用jsp