How to install Hadoop 2.2.0
来源：互联网 发布：淘宝的管控记录是什么 编辑：程序博客网 时间：2024/06/03 23:48
How to install Hadoop 2.2.0
Share the post "How to install Hadoop 2.2.0 (Pseudo-Distributed Mode)"
Hadoop 2.2.0.
1. hadoop
1
2
[root@ruo91 ~]# useradd hadoop
[root@ruo91 ~]# passwd hadoop
2. SSH 免密钥登陆
1
2
3
4
[root@ruo91~]# su - hadoop
[hadoop@ruo91~]$ssh-keygen-tdsa-P''-f~/.ssh/id_dsa
[hadoop@ruo91~]$cat~/.ssh/id_dsa.pub>>~/.ssh/authorized_keys
[hadoop@ruo91~]$chmod644~/.ssh/authorized_keys
测试成功没有
1
2
3
[hadoop@ruo91 ~]$ ssh localhost
Last login: Thu Oct 17 11:49:09 2013 from localhost.localdomain
Hello?
3. JDK 安装
http://www.oracle.com/technetwork/java/javase/downloads/index.html
1
2
3
4
[hadoop@ruo91~]$curl-LO"http://download.oracle.com/otn-pub/java/jdk/7u55-b13/jdk-7u55-linux-x64.tar.gz"\
-H'Cookie: oraclelicense=accept-securebackup-cookie'
[hadoop@ruo91~]$tarxzvfjdk-7u55-linux-x64.tar.gz
[hadoop@ruo91~]$mvjdk1.7.0_55jdk
配置环境变量(也可是是在/etc/profile里面配置)
1
2
3
4
[hadoop@ruo91 ~]$ nano ~/.bash_profile
# JAVA
export JAVA_HOME=$HOME/jdk
export PATH=$PATH:$JAVA_HOME/bin
4. Hadoop 安装
http://www.apache.org/dyn/closer.cgi/hadoop/common/
1
2
3
[hadoop@ruo91~]$wgethttp://mirror.apache-kr.org/hadoop/core/stable2/hadoop-2.2.0.tar.gz
[hadoop@ruo91~]$tarxzvfhadoop-2.2.0.tar.gz
[hadoop@ruo91~]$mvhadoop-2.2.02.2.0
– Hadoop 环境变量配置
1
2
3
4
5
6
7
8
9
[hadoop@ruo91 ~]$ nano ~/.bash_profile
# Hadoop
export HADOOP_PREFIX="$HOME/2.2.0"
export PATH=$PATH:$HADOOP_PREFIX/bin
export PATH=$PATH:$HADOOP_PREFIX/sbin
export HADOOP_MAPRED_HOME=${HADOOP_PREFIX}
export HADOOP_COMMON_HOME=${HADOOP_PREFIX}
export HADOOP_HDFS_HOME=${HADOOP_PREFIX}
export YARN_HOME=${HADOOP_PREFIX}
– hadoop-env.sh
1
2
3
4
[hadoop@ruo91~]$nano2.2.0/etc/hadoop/hadoop-env.sh
exportJAVA_HOME=$HOME/jdk
exportHADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
exportHADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib/native"
上面的 HADOOP_OPTS没有配置正确会出现 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
– yarn-env.sh
1
2
3
[hadoop@ruo91 ~]$ nano 2.2.0/etc/hadoop/yarn-env.sh
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib/native"
使配置生效
1
[hadoop@ruo91~]$ source~/.bash_profile
5. Hadoop 配置文件
– core-site.xml
1
[hadoop@ruo91 ~]$ nano $HADOOP_PREFIX/etc/hadoop/core-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
<?xmlversion="1.0"encoding="UTF-8"?>
<?xml-stylesheettype="text/xsl"href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
<final>true</final>
</property>
</configuration>
– hdfs-site.xml
1
[hadoop@ruo91 ~]$ nano $HADOOP_PREFIX/etc/hadoop/hdfs-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
<?xmlversion="1.0"encoding="UTF-8"?>
<?xml-stylesheettype="text/xsl"href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hadoop/dfs/name</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hadoop/dfs/data</value>
<final>true</final>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
– mapred-site.xml
1
[hadoop@ruo91 ~]$ nano $HADOOP_PREFIX/etc/hadoop/mapred-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
<?xmlversion="1.0"?>
<?xml-stylesheettype="text/xsl"href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>file:/home/hadoop/mapred/system</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>file:/home/hadoop/mapred/local</value>
<final>true</final>
</property>
</configuration>
– yarn-site.xml
1
[hadoop@ruo91 ~]$ nano $HADOOP_PREFIX/etc/hadoop/yarn-site.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<?xmlversion="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
6. Hadoop 运行
– Name Node format(只要格式一次)
多次格式化后，datanode启动不了
0.20.2版本解决方式:1、查看名称节点上(即在配置文件中配置的hadoop.tmp.dir参数路径)/usr/hadoop0.20.2/hadoop-huanghz/dfs/name/current/文件夹中VERSION文件中的namespaceid；2、在两个数据节点修改上dfs.data.dir配置的路径下current文件夹中的VERSION文件namespaceid与名称节点保持一致2.20版本解决/data/hadoop/dfs/name/current/VERSION用name下面的clusterID，修改datanode的/data/hadoop/dfs/data/current/VERSION 里面的clusterID每次格式化，name下面的VERSION的clusterID会产生一个新的ID，要去修改各个节点的VERSION的clusterID……/dfs/data/currentstorageID=DS-1959445666-10.161.138.100-50010-1386493413986clusterID=CID-64a3a726-29e4-4d80-86a6-035ef33a225bcTime=0storageType=DATA_NODElayoutVersion=-47……/dfs/name/current/VERSION#Fri Jan 03 10:37:48 CST 2014namespaceID=1667984727clusterID=CID-42c6d540-c3ca-44df-95e8-01a6d87effb5cTime=0storageType=NAME_NODEblockpoolID=BP-220196921-10.161.138.100-1388716668863layoutVersion=-47
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
[hadoop@ruo91 ~]$ hdfs namenode -format
13/10/17 23:18:29 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ruo91.yongbok.net/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.2.0
STARTUP_MSG: classpath = /home/hadoop/2.2.0/etc/hadoop:/home/hadoop/2.2.0/share/hadoop/common/lib/jettison-
........
....................
..........................
Blah blah ~
................................
2.2.0.jar:/home/hadoop/2.2.0/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common -r 1529768; compiled by 'hortonmu' on 2013-10-07T06:28Z
STARTUP_MSG: java = 1.7.0_45
************************************************************/
13/10/17 23:18:29 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
13/10/17 23:18:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-060a7239-4bac-4801-a720-9702eb60c341
13/10/17 23:18:31 INFO namenode.HostFileManager: read includes:
HostSet(
)
13/10/17 23:18:31 INFO namenode.HostFileManager: read excludes:
HostSet(
)
13/10/17 23:18:31 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
13/10/17 23:18:31 INFO util.GSet: Computing capacity for map BlocksMap
13/10/17 23:18:31 INFO util.GSet: VM type = 64-bit
13/10/17 23:18:31 INFO util.GSet: 2.0% max memory = 966.7 MB
13/10/17 23:18:31 INFO util.GSet: capacity = 2^21 = 2097152 entries
13/10/17 23:18:31 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
13/10/17 23:18:31 INFO blockmanagement.BlockManager: defaultReplication = 1
13/10/17 23:18:31 INFO blockmanagement.BlockManager: maxReplication = 512
13/10/17 23:18:31 INFO blockmanagement.BlockManager: minReplication = 1
13/10/17 23:18:31 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
13/10/17 23:18:31 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
13/10/17 23:18:31 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
13/10/17 23:18:31 INFO blockmanagement.BlockManager: encryptDataTransfer = false
13/10/17 23:18:31 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
13/10/17 23:18:31 INFO namenode.FSNamesystem: supergroup = supergroup
13/10/17 23:18:31 INFO namenode.FSNamesystem: isPermissionEnabled = false
13/10/17 23:18:31 INFO namenode.FSNamesystem: HA Enabled: false
13/10/17 23:18:31 INFO namenode.FSNamesystem: Append Enabled: true
13/10/17 23:18:32 INFO util.GSet: Computing capacity for map INodeMap
13/10/17 23:18:32 INFO util.GSet: VM type = 64-bit
13/10/17 23:18:32 INFO util.GSet: 1.0% max memory = 966.7 MB
13/10/17 23:18:32 INFO util.GSet: capacity = 2^20 = 1048576 entries
13/10/17 23:18:32 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/10/17 23:18:32 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
13/10/17 23:18:32 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
13/10/17 23:18:32 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
13/10/17 23:18:32 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
13/10/17 23:18:32 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
13/10/17 23:18:32 INFO util.GSet: Computing capacity for map Namenode Retry Cache
13/10/17 23:18:32 INFO util.GSet: VM type = 64-bit
13/10/17 23:18:32 INFO util.GSet: 0.029999999329447746% max memory = 966.7 MB
13/10/17 23:18:32 INFO util.GSet: capacity = 2^15 = 32768 entries
13/10/17 23:18:32 INFO common.Storage: Storage directory /home/hadoop/dfs/name has been successfully formatted.
13/10/17 23:18:32 INFO namenode.FSImage: Saving image file /home/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
13/10/17 23:18:32 INFO namenode.FSImage: Image file /home/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 198 bytes saved in 0 seconds.
13/10/17 23:18:32 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
13/10/17 23:18:32 INFO util.ExitUtil: Exiting with status 0
13/10/17 23:18:32 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ruo91.yongbok.net/127.0.0.1
************************************************************/
– Hadoop (name node, data node, yarn) 启动
1
[hadoop@ruo91~]$start-all.sh
– 查看是否启动成功
1
2
3
4
5
6
7
[hadoop@ruo91~]$jps
32267SecondaryNameNode
32028DataNode
415NodeManager
31905NameNode
452Jps
319ResourceManager
- 通过网页查看
http://localhost:8088/ or http://localhost:50070/
0 0
- How to install Hadoop 2.2.0
- How to Install Hadoop on Ubuntu 13.10
- How to Install Hadoop on Ubuntu 13.10
- How to install Hadoop 2.7 on RedHat 6.5
- How to install BigBlueButton 0.9.0-beta
- How to Install Orabm
- How to Install Cygwin
- how to install CYGWIN
- how to install curl
- how to install mplayer
- HOW TO INSTALL subtitleeditor
- How to TWiki install
- How to install Redmine
- How to install opennebula
- How to install repo
- how to install eclipse
- How to install Jmeter
- How to install statspack
- learning OpenCV(4)
- 解决MyEclipse不能自动提示代码的功能
- JSP隐式对象到servlet API的映射
- 1598 find the most comfortable road
- zigbee中profile,cluster的认识
- How to install Hadoop 2.2.0
- 使用PopoPWindow当做AlertDialog,控制Dialog的背景大小
- Think Python 学习笔记5：条件与递归
- Huffman Codes
- Swift入门教程12-可选链
- ubuntu eclipse 启动图标是个问号
- 将最小值放到数列最前边
- cin.get()
- 各种文件的十六进制文件头
