HA HADOOP集群和HIVE部署

来源：互联网发布：Java中怎样输出Path 编辑：程序博客网时间：2024/05/17 23:52

HDFS HA HADOOP集群部署

1.集群环境节点分布

JournalNode： bigdatasvr01 , bigdatasvr02，bigdatasvr03
namenode: bigdatasvr02(active)，bigdatasvr03(standby)
datanode: bigdatasvr01, bigdatasvr03

nodemanager: bigdatasvr01, bigdatasvr03

ResourceManager: bigdatasvr02

2.修改主机名

3.设置免密码登录

每台机器上都执行命令:

ssh-keygen -t rsa -P ''

将bigdatasvr02的公钥拷贝到bigdatasvr01 ,bigdatasvr03上

ssh-copy-id hadoop@bigdatasvr01

ssh-copy-id hadoop@bigdatasvr03

至少要保证bigdatasvr02免密码登录到bigdatasvr01 ,bigdatasvr03上

4.设置环境变量

1.设置JDK环境变量

2.设置hadoop环境变量，在/etc/profile.d下新建一个hadoop.sh:

export HADOOP_HOME=/home/hadoop/hadoop

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

使其生效：

source hadoop.sh

5.搭建hadoop集群

用的hadoop是apache原生包hadoop-2.7.1.tar.gz

5.1 修改配置文件

把下面6个文件修改好，然后拷贝到所有节点。

hadoop-env.sh，core-stie.xml，hdfs-site.xml，yarn-site.xml，mapred-site.xml，slaves

5.1.1修改core-stie.xml

<property><name>fs.defaultFS</name><value>hdfs://bigdatasvr02:9000</value></property><property><name>io.file.buffer.size</name><value>131072</value></property><property><name>hadoop.tmp.dir</name><value>file:/home/hadoop/hadoop/tmp</value><description>Abasefor other temporary directories.</description></property>

5.1.2修改hdfs-site.xml

 <property><name>dfs.nameservices</name><value>hadoopcluster</value></property><property><name>dfs.ha.namenodes.hadoopcluster</name><value>nn1,nn2</value></property><property><name>dfs.namenode.rpc-address.hadoopcluster.nn1</name><value>bigdatasvr02:9000</value></property><property><name>dfs.namenode.rpc-address.hadoopcluster.nn2</name><value>bigdatasvr03:9000</value></property><property><name>dfs.namenode.http-address.hadoopcluster.nn1</name><value>bigdatasvr02:50070</value></property><property><name>dfs.namenode.http-address.hadoopcluster.nn2</name><value>bigdatasvr03:50070</value></property><property><name>dfs.namenode.name.dir</name><value>file:/home/hadoop/hadoop/ha/hdfs/name</value><description>allow multiple directory split by ,</description></property><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://bigdatasvr01:8485;bigdatasvr02:8485;bigdatasvr03:8485/hadoopcluster</value></property><property><name>dfs.datanode.data.dir</name><value>file:/home/hadoop/hadoop/ha/hdfs/data</value><description>allow multiple directory split by ,</description></property><property><name>dfs.ha.automatic-failover.enabled</name><value>false</value><description>Whether automatic failover is enabled. See the HDFS HighAvailability documentation for details on automatic HA configuration.</description></property><property><name>dfs.journalnode.edits.dir</name><value>/home/hadoop/hadoop/ha/hdfs/journal</value></property> <property>  <name>dfs.replication</name>  <value>2</value> </property><property> <name>dfs.webhdfs.enabled</name>     <value>true</value> </property>

5.1.3修改mapred-site.xml

 <property>   <name>mapreduce.framework.name</name>   <value>yarn</value> </property>   <property><name>mapreduce.jobhistory.address</name><value>bigdatasvr03:10020</value> </property> <property><name>mapreduce.jobhistory.webapp.address</name><value>bigdatasvr03:19888</value> </property>

5.1.4修改yarn-site.xml

<property><description>The hostname of the RM.</description><name>yarn.resourcemanager.hostname</name><value>bigdatasvr02</value></property>    <property><description>The address of the applications manager interface in the RM.</description><name>yarn.resourcemanager.address</name><value>${yarn.resourcemanager.hostname}:8032</value></property><property><description>The http address of the RM web application.</description><name>yarn.resourcemanager.webapp.address</name><value>${yarn.resourcemanager.hostname}:8088</value></property><property><description>The https adddress of the RM web application.</description><name>yarn.resourcemanager.webapp.https.address</name><value>${yarn.resourcemanager.hostname}:8090</value></property><property><name>yarn.resourcemanager.resource-tracker.address</name><value>${yarn.resourcemanager.hostname}:8031</value></property><property>   <name>yarn.resourcemanager.scheduler.address</name>   <value>${yarn.resourcemanager.hostname}:8030</value></property><property><description>The address of the RM admin interface.</description><name>yarn.resourcemanager.admin.address</name><value>${yarn.resourcemanager.hostname}:8033</value></property><property><description>List of directories to store localized files in. An application's localized file directory will be found in:${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.Individual containers' work directories, called container_${contid}, willbe subdirectories of this.</description><name>yarn.nodemanager.local-dirs</name><value>/home/hadoop/hadoop/ha/yarn/local</value></property><property><description>Whether to enable log aggregation</description><name>yarn.log-aggregation-enable</name><value>true</value></property><property><description>Where to aggregate logs to.</description><name>yarn.nodemanager.remote-app-log-dir</name><value>/home/hadoop/logs</value></property><property><description>Number of CPU cores that can be allocated for containers.</description><name>yarn.nodemanager.resource.cpu-vcores</name><value>4</value></property><property><description>the valid service name should only contain a-zA-Z0-9_ and can not start with numbers</description><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property>