hadoop学习（1）—— 编译hadoop源码

一、准备工作
操作系统：

centos6.8

远程连接工具：

Xshell、Xftp

软件：

findbugs-1.3.9.tar.gz
protobuf-2.5.0.tar.gz
apache-maven-3.3.9-bin.tar.gz
hadoop-2.6.5-src.tar.gz
eclipse-SDK-4.2.1-linux-gtk-x86_64.tar.gz
jdk-7u25-linux-x64.tar.gz

二、安装步骤
1.修改主机名——临时修改，重启后失效。永久生效，需要更改/etc/network文件。
hostname [HOSTNAME]
2.创建目录，方便管理。

[root@localhost opt]# mkdir /opt/softwares        #下载软件目录    [root@localhost opt]# mkdir /opt/modules          #安装软件目录[root@localhost opt]# mkdir /opt/tools            #下载工具目录[root@localhost opt]# mkdir /opt/datas            #下载数据目录

[root@localhost softwares]# tar -zxvf hadoop-2.6.5-src.tar.gz -C /opt/modules/

3.卸载centos6.8自带的java jdk，系统不同，可能需要卸载的也不一样。

[root@localhost jdk1.8.0_121]# rpm -qa | grep java        java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64tzdata-java-2016c-1.el6.noarchjava-1.7.0-openjdk-1.7.0.99-2.6.5.1.el6.x86_64java_cup-0.10k-5.el6.x86_64java-1.6.0-openjdk-1.6.0.38-1.13.10.4.el6.x86_64[root@localhost jdk1.8.0_121]# rpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64 tzdata-java-2016c-1.el6.noarch java-1.7.0-openjdk-1.7.0.99-2.6.5.1.el6.x86_64 java_cup-0.10k-5.el6.x86_64 java-1.6.0-openjdk-1.6.0.38-1.13.10.4.el6.x86_64

4.安装jdk1.8

[root@localhost jdk1.8.0_121]# tar -zxvf jdk-8u121-linux-x64.tar.gz -C /opt/modules/

5.配置环境变量
编辑/ext/profile文件，添加两行

## SET JAVA_HOMEexport JAVA_HOME=/opt/modules/jdk1.8.0_121export PATH=$PATH:$JAVA_HOME/bin

验证是否成功

[root@hadoop-yarn ~]# source /etc/profile[root@hadoop-yarn ~]# java -versionjava version "1.8.0_121"Java(TM) SE Runtime Environment (build 1.8.0_121-b13)Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

6.安装maven

[root@localhost modules]# tar -zvxf apache-maven-3.5.0-bin.tar.gz -C /opt/modules/

设置环境变量 /etc/profile

## SET MAVEN_HOMEexport MAVEN_HOME=/opt/modules/apache-maven-3.5.0export PATH=$PATH:$MAVEN_HOME/bin[root@hadoop-yarn ~]# source /etc/profile

验证maven是否安装成功，使用mvn

[root@hadoop-yarn ~]# mvn -versionApache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T15:39:06-04:00)Maven home: /opt/modules/apache-maven-3.5.0Java version: 1.8.0_121, vendor: Oracle CorporationJava home: /opt/modules/jdk1.8.0_121/jreDefault locale: en_US, platform encoding: UTF-8OS name: "linux", version: "2.6.32-642.el6.x86_64", arch: "amd64", family: "unix"

在当前用户的家目录下，创建仓库包，拷贝setting文件

[root@hadoop-yarn ~]# mkdir -p .m2/repository[root@hadoop-yarn ~]# cp /opt/modules/apache-maven-3.5.0/conf/setting.xml ~/.m2/repository/

7.安装protobuf-2.5.0

[root@hadoop-yarn softwares]# tar -zxvf protobuf-2.5.0.tar.gz -C /opt/modules/

设置环境变量

## SET PROTOBUF_HOMEexport PROTOBUF_HOME=/opt/modules/protobuf-2.5.0

[root@hadoop-yarn ~]# yum install autoconf automake libtool \cmake ncurses-devel openssl-devel lzo-devel zlib-devel gcc gcc-c++[root@hadoop-yarn ~]./configure[root@hadoop-yarn ~]make[root@hadoop-yarn ~]make install

8.安装findbugs

[root@hadoop-yarn softwares]# tar -zxvf findbugs-1.3.9.tar.gz -C /opt/modules/

## SET FINDBUGS_HOMEexprot FINDBUGS_HOME=/opt/modules/findbugs-1.3.9export PATH=$PATH:$FINDBUGS_HOME/bin

9.编译hadoop

[root@hadoop-yarn hadoop-2.6.5-src]#mvn package -Pdisk,native,docs -DskipTests -Dtar

10.编译完成
在/opt/modules/hadoop-2.6.5-src/hadoop-dist/target目录下可以看到编译好的hadoop-2.6.5.tar.gz文件。

三、注意事项
1.setting.xml不要忘记拷贝
2.编译前最好重启一下
3.如果编译失败，选择jdk,maven版本1.7和3.3.9试试。