Redhat 6.4-x64 编译 Hadoop-2.7.1并分布式安装
来源:互联网 发布:淘宝卖家一个好评几分 编辑:程序博客网 时间:2024/06/05 19:51
文章中所用到的软件百度网盘,方便下载
http://pan.baidu.com/s/1kTrNXL9
1.系统环境
[root@master ~]# uname -aLinux master 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux或者[root@master /]# cat /etc/redhat-releaseRed Hat Enterprise Linux Server release 6.4 (Santiago)
2.安装前的准备
2.1关闭防火墙
--查看防火墙状态[root@master ~]# service iptables status--停止防火墙[root@master ~]# service iptables stop--永久禁止防火墙[root@master ~]# chkconfig iptables off
2.2查看ssh是否安装
--第一种方式:[root@master ~]# which ssh/usr/bin/ssh--第二种方式[root@master ~]# service sshd statusopenssh-daemon (pid 1983) is running...--第三种方式[root@master ~]# rpm -qa openssh openssl--第四种方式(查看是否启动)[root@master ~]# ps -ef|grep ssh
2.3配置网卡
[root@master ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0HWADDR=08:00:27:C0:18:81TYPE=EthernetUUID=8f07e2a4-702d-4e4e-a7c9-a0d4d1c9a880ONBOOT=yesNM_CONTROLLED=yesBOOTPROTO=noneIPADDR=192.168.8.205NETMASK=255.255.255.0GATEWAY=192.168.8.1DNS1=221.228.255.1
2.4配置hosts
[root@master ~]# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6--添加如下内容192.168.8.205 master192.168.8.223 slave
2.5修改hostname为master
[root@master ~]# cat /etc/sysconfig/networkNETWORKING=yesHOSTNAME=master
2.6创建hadoop用户
--创建用户[root@master ~]# useradd hadoop--配置用户密码[root@master ~]# passwd hadoop
2.7配置ssh无密码登陆
在master机器上操作
--切换hadoop用户[root@master ~]# su - hadoop--生成需要建立ssh的私钥和公钥,连续3次回车建立公钥文件(id_rsa.pub)和私钥文件(id_rsa)[hadoop@master ~]$ ssh-keygen -t rsa--把id_rsa.pub追加到授权的authorized_keys里面去[hadoop@master ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys--修改权限 [hadoop@master ~]$ chmod 600 ~/.ssh/authorized_keys --切换到root用户,修改ssh配置文件[hadoop@master ~]$ exit[root@master ~]# vim /etc/ssh/sshd_config RSAAuthentication yes #启用RSA认证 PubkeyAuthentication yes #启用公钥私钥配对认证方式 AuthorizedKeysFile .ssh/authorized_keys #公钥文件路径
登陆到slave机器上
--切换到hadoop用户[root@slave ~]# su - hadoop --创建.ssh文件夹[hadoop@slave ~]$ mkdir ~/.ssh--切换到maste主机上的hadoop用户下[root@master ~]# su - hadoop --将密钥scp到slave机器上[hadoop@master ~]$ scp ~/.ssh/id_rsa.pub hadoop@slave:~/.ssh--切换到slave机器上的hadoop下[hadoop@slave ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys--授予权限(master和slave都要授权)chmod 700 ~/.ssh/chmod 700 /home/hadoopchmod 600 ~/.ssh/authorized_keys
3.安装必须的软件
3.1安装jdk
--创建jdk目录mkdir /usr/java将下载的jdk-8u60-linux-x64.tar.gz解压到java目录下--配置jdk环境[root@master ~]# vim /etc/profileexport JAVA_HOME=/usr/java/jdk1.8.0_60export PATH=$PATH:$JAVA_HOME/bin--修改生效[root@master ~]# source profile --测试是否安装成功[hadoop@master ~]$ java -versionjava version "1.5.0"gij (GNU libgcj) version 4.4.7 20120313 (Red Hat 4.4.7-3)Copyright (C) 2007 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.这里有点问题,显示的是系统自带的java,替换自己安装的见:http://blog.csdn.net/u011364306/article/details/48375653
3.2安装其他组件
[root@master ~]#yum install svn ncurses-devel gcc* lzo-devel zlib-devel autoconf automake libtool cmake openssl-devel
3.3安装maven
--下载apache-maven,将下载后的文件拷贝到 /usr/local/目录下。apache-maven-3.3.3-bin.tar.gz--解压文件[root@master local]# tar xf apache-maven-3.3.3-bin.tar.gz--给maven做一个软连接[root@master local]# ln -s apache-maven-3.3.3 apache-maven--配置maven环境[root@master ~]# vim /etc/profileexport M2_HOME=/usr/local/apache-maven--在java环境变量后面加上:$M2_HOME/binexport PATH=$PATH:$JAVA_HOME/bin:$M2_HOME/bin--修改生效[root@master ~]# source profile --测试是否安装成功[root@master local]# mvn -vApache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T19:57:37+08:00)Maven home: /usr/local/apache-mavenJava version: 1.8.0_60, vendor: Oracle CorporationJava home: /usr/java/jdk1.8.0_60/jreDefault locale: en_US, platform encoding: UTF-8OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix"
3.4安装ant
--下载apache-ant,将下载后的文件拷贝到 /usr/local/目录下。apache-ant-1.9.6-bin.tar.gz--解压文件[root@master local]# tar xf apache-ant-1.9.6-bin.tar.gz--配置ant环境[root@master ~]# vim /etc/profileexport ANT_HOME=/usr/local/apache-ant-1.9.6export PATH=$PATH:$ANT_HOME/bin--修改生效[root@master ~]# source profile
3.5安装findbugs
--下载findbugs,将下载后的文件拷贝到 /usr/local/目录下。findbugs-3.0.1.tar.gz--解压文件[root@master local]# tar xf findbugs-3.0.1.tar.gz--配置findbugs环境[root@master ~]# vim /etc/profileexport FINDBUGS_HOME=/usr/local/findbugs-3.0.1export PATH=$PATH:$FINDBUGS_HOME/bin--修改生效[root@master ~]# source profile
3.6安装protobuf
--下载protobuf,将下载后的文件拷贝到 /usr/local/目录下。protobuf-2.5.0.tar.gz--解压文件[root@master local]# tar xf protobuf-2.5.0.tar.gz--编译安装[root@master local]# cd protobuf-2.5.0[root@master protobuf-2.5.0]# ./configure --prefix=/usr/local[root@master protobuf-2.5.0]# make && make install
4.编译hadoop源码
--下载hadoop,将下载后的文件拷贝到 /usr/local/目录下。hadoop-2.7.1-src.tar.gz--解压文件[root@master local]# tar xf hadoop-2.7.1-src.tar.gz--编译安装[root@master local]# cd hadoop-2.7.1-src/--这一步执行后会比较久,我虚拟机大概花了近1个小时时间[root@master local]# mvn package -Pdist,native,docs -DskipTests -Dtar
4.1修改maven源
[root@master local]# cd /usr/local/apache-maven/conf[root@master conf]# mv settings.xml settings.xml.bak[root@master conf]# touch settings.xml --maven中央仓库的配置(改成oschina,增加访问速度)<mirror> <id>nexus-osc</id> <mirrorOf>*</mirrorOf> <name>Nexus osc</name> <url>http://maven.oschina.net/content/groups/public/</url> </mirror></mirrors><profiles> <profile> <id>jdk17</id> <activation> <activeByDefault>true</activeByDefault> <jdk>1.7</jdk> </activation> <properties> <maven.compiler.source>1.7</maven.compiler.source> <maven.compiler.target>1.7</maven.compiler.target> <maven.compiler.compilerVersion>1.7</maven.compiler.compilerVersion> </properties> <repositories> <repository> <id>nexus</id> <name>local private nexus</name> <url>http://maven.oschina.net/content/groups/public/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories> <pluginRepositories> <pluginRepository> <id>nexus</id> <name>local private nexus</name> <url>http://maven.oschina.net/content/groups/public/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </pluginRepository> </pluginRepositories> </profile></profiles>
4.2 编译完后,会在目录/usr/local/hadoop-2.7.1-src/hadoop-dist/target下生成
hadoop-2.7.1和编译好的64位hadoop-2.7.1.tar.gz
[root@master conf]# cd /usr/local/hadoop-2.7.1-src/hadoop-dist/target[root@master target]# lltotal 844744drwxr-xr-x. 2 root root 4096 Sep 11 11:25 antrun-rw-r--r--. 1 root root 1867 Sep 11 11:25 dist-layout-stitching.sh-rw-r--r--. 1 root root 640 Sep 11 11:26 dist-tar-stitching.shdrwxr-xr-x. 9 root root 4096 Sep 11 11:25 hadoop-2.7.1-rw-r--r--. 1 root root 285910336 Sep 11 11:27 hadoop-2.7.1.tar.gz-rw-r--r--. 1 root root 2823 Sep 11 11:26 hadoop-dist-2.7.1.jar-rw-r--r--. 1 root root 579067602 Sep 11 11:29 hadoop-dist-2.7.1-javadoc.jardrwxr-xr-x. 2 root root 4096 Sep 11 11:28 javadoc-bundle-optionsdrwxr-xr-x. 2 root root 4096 Sep 11 11:26 maven-archiverdrwxr-xr-x. 2 root root 4096 Sep 11 11:25 test-dir
5.配置hadoop
5.1基础操作
--将文件拷贝到hadoop用户下[root@master /]# cp -r /usr/local/hadoop-2.7.1-src/hadoop-dist/target/hadoop-2.7.1 /home/hadoop/--授予权限[root@master /]# chown -R hadoop.hadoop /home/hadoop/hadoop-2.7.1/--配置hadoop环境[root@master ~]# vim /etc/profileexport HADOOP_HOME=/home/hadoop/hadoop-2.7.1export PATH=$PATH:$HADOOP_HOME/bin--修改生效[root@master ~]# source profile --切换到hadoop用户,创建基础目录[root@master ~]# su - hadoop[hadoop@master ~]$ cd hadoop-2.7.1/[hadoop@master hadoop-2.7.1]$ mkdir -p dfs/name[hadoop@master hadoop-2.7.1]$ mkdir -p dfs/data[hadoop@master hadoop-2.7.1]$ mkdir -p tmp
5.2配置所有slave节点
[hadoop@master hadoop-2.7.1]$ cd etc/hadoop/[hadoop@master hadoop]$ cat slavesslave#slave1#slave2#有多少个slave配置多少个
5.3修改hadoop-env.sh和yarn-env.sh
[hadoop@master hadoop]$ vim hadoop-env.sh export JAVA_HOME=/usr/java/jdk1.8.0_60[hadoop@master hadoop]$ vim yarn-env.shexport JAVA_HOME=/usr/java/jdk1.8.0_60
5.4修改core-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://192.168.8.205:9000</value></property><property><name>io.file.buffer.size</name><value>131702</value></property><property><name>hadoop.tmp.dir</name><value>file:/home/hadoop/hadoop-2.7.1/tmp</value></property><property><name>hadoop.proxyuser.hadoop.hosts</name><value></value></property><property><name>hadoop.proxyuser.hadoop.groups</name><value></value></property></configuration>
5.5修改hdfs-site.xml
<configuration><property><name>dfs.namenode.name.dir</name><value>/home/hadoop/hadoop-2.7.1/dfs/name</value></property><property><name>dfs.datanode.data.dir</name><value>/home/hadoop/hadoop-2.7.1/dfs/data</value></property><property><name>dfs.replication</name><value>3</value></property><property><name>dfs.namenode.secondary.http-address</name><value>192.168.8.205:9001</value></property><property><name>dfs.webhdfs.enabled</name><value>true</value></property></configuration>
5.6修改mapred-site.xml
cp mapred-site.xml.template mapred-site.xml
<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>mapreduce.jobhistory.address</name><value>192.168.8.205:10020</value></property><property><name>mapreduce.jobhistory.webapp.address</name><value>192.168.8.205:19888</value></property></configuration>
5.7配置yarn-site.xml
<configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value></property><property><name>yarn.resourcemanager.address</name><value>192.168.8.205:8032</value></property><property><name>yarn.resourcemanager.scheduler.address</name><value>192.168.8.205:8030</value></property><property><name>yarn.resourcemanager.resource-tracker.address</name><value>192.168.8.205:8031</value></property><property><name>yarn.resourcemanager.admin.address</name><value>192.168.8.205:8033</value></property><property><name>yarn.resourcemanager.webapp.address</name><value>192.168.8.205:8088</value></property><property><name>yarn.nodemanager.resource.memory-mb</name><value>1024</value></property></configuration>
5.8格式化namenode
--切换目录[hadoop@master hadoop]$ cd ../..[hadoop@master hadoop-2.7.1]$ lsbin dfs etc include lib libexec LICENSE.txt logs NOTICE.txt README.txt sbin share tmp--格式化[hadoop@master hadoop-2.7.1]$ ./bin/hdfs namenode -format
5.9启动
[hadoop@master hadoop-2.7.1]$ ./sbin/start-dfs.sh[hadoop@master hadoop-2.7.1]$ ./sbin/start-yarn.sh
5.10检查启动情况
http://192.168.8.205:8088http://192.168.8.205:50070
0 0
- Redhat 6.4-x64 编译 Hadoop-2.7.1并分布式安装
- Hadoop-2.2.0 + hbase-0.98.4-hadoop2 RedHat x64 伪分布式安装小札(及配置文件)
- CentOS7-64bit 编译 Hadoop-2.7.4,并伪分布式安装
- 【烂尾】RedHat Linux 5 x64 安装 Hadoop-2.2.0
- Redhat server Linux 6 x64编译安装wine64
- CentOS-64bit编译Hadoop-2.5.源代码,并进行分布式安装
- Hadoop 2.7.1在centos 6.3下配置变量环境并安装配置伪分布式集群
- centos 6.3下安装Hadoop 2.7.1并配置伪分布式集群
- Hadoop 2.7.1 在centos6.3下配置环境变量并安装配置完全分布式集群
- 自编译 hadoop-2.5.2 x64安装包 下载
- Windows7 x64系统下安装Nodejs并在WebStorm 9.0.1下搭建编译less环境
- Hadoop2.X64 分布式安装指南
- Windows 源码编译Hadoop 2.7.4生成X64
- 2.7.2hadoop分布式安装
- hadoop安装-redhat
- 如何在RedHat里面编译R源码并安装
- hadoop-2.4.1分布式安装
- OpenFoam安装于RedHat 5.7-x64
- linux中sed的用法
- ENC28J60 驱动开发要点
- 开环和闭环传递函数
- 集装箱码头企业IT运维思路
- libcurl库基本使用
- Redhat 6.4-x64 编译 Hadoop-2.7.1并分布式安装
- Android开发_View 硬件加速
- Java 代理模式
- ubuntu中安装gcc和g++编译环境
- 字符串和编码
- 微信第三方平台开发三(消息加解密)
- [LeetCode 231] Power of Two
- 关于O(n^2)复杂度的体验
- KMP算法详解