Linux_Hadoop2.7.3 安装笔记

环境：Vmware Workstation 10，CentOS-7-x86_64-DVD-1511.iso，Xshell 4.0，Master：192.168.216.144，Slave1：192.168.216.145，Slave2：192.168.216.146.

参照Linux_Hadoop1.2.1 安装笔记到无密码登录：

官方安装手册

[hadoop@localhost ~]$ wget http://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz

[hadoop@localhost ~]$ sudo tar -zxvf hadoop-2.7.3.tar.gz -C /usr/local/

[hadoop@localhost ~]$ sudo mv /usr/local/hadoop-2.7.3/ /usr/local/hadoop

[hadoop@localhost ~]$ ll /usr/local/ | grep hadoop

drwxr-xr-x. 9 root root 4096 8月 18 2016 hadoop

[hadoop@localhost ~]$ sudo chown -R hadoop:hadoop /usr/local/hadoop/

[hadoop@localhost ~]$ ll /usr/local/ | grep hadoop

drwxr-xr-x. 9 hadoop hadoop 4096 8月 18 2016 hadoop

[hadoop@localhost ~]$ sudo vim /etc/profile

[hadoop@localhost ~]$ hadoop

-bash: hadoop: 未找到命令

[hadoop@localhost ~]$ source /etc/profile

[hadoop@localhost ~]$ hadoop

Usage: hadoop [–config confdir] [COMMAND | CLASSNAME]

配置文件

[hadoop@localhost ~]$ vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh

[hadoop@localhost ~]$ vim /usr/local/hadoop/etc/hadoop/core-site.xml

[hadoop@localhost ~]$ vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml

如果不想启动 YARN，务必把配置文件 mapred-site.xml 重命名，改成 mapred-site.xml.template，需要用时改回来就行。否则在该配置文件存在，而未开启 YARN 的情况下，运行程序会提示 “Retrying connect to server: 0.0.0.0/0.0.0.0:8032” 的错误，这也是为何该配置文件初始文件名为 mapred-site.xml.template。

[hadoop@localhost ~]$ cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml

[hadoop@localhost ~]$ vim /usr/local/hadoop/etc/hadoop/mapred-site.xml

[hadoop@localhost ~]$ vim /usr/local/hadoop/etc/hadoop/yarn-site.xml

[hadoop@localhost ~]$ vim /usr/local/hadoop/etc/hadoop/slaves

关闭防火墙

注意：下面命令中“–”是两个“-”，csdn显示有误。

[hadoop@localhost ~]$ firewall-cmd –state

running

[hadoop@localhost ~]$ sudo systemctl stop firewalld

[sudo] password for hadoop:

[hadoop@localhost ~]$ firewall-cmd –state

not running

[hadoop@localhost ~]$ hdfs namenode -format

17/06/22 13:26:17 INFO namenode.NameNode: STARTUP_MSG:
/**************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = localhost/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.7.3
。。。。。。
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be5982de4719c1c8af91ccff; compiled by ‘root’ on 2016-08-18T01:41Z
STARTUP_MSG: java = 1.8.0_131
**************************************************/
17/06/22 13:26:17 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/06/22 13:26:17 INFO namenode.NameNode: createNameNode [-format]
17/06/22 13:26:18 WARN common.Util: Path /usr/local/hadoop/name should be specified as a URI in configuration files. Please update hdfs configuration.
17/06/22 13:26:18 WARN common.Util: Path /usr/local/hadoop/name should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-d6a65f14-6165-4fa1-bcd0-7e01903b2c87
17/06/22 13:26:18 INFO namenode.FSNamesystem: No KeyProvider found.
17/06/22 13:26:18 INFO namenode.FSNamesystem: fsLock is fair:true
17/06/22 13:26:18 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
17/06/22 13:26:18 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
17/06/22 13:26:18 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
17/06/22 13:26:18 INFO blockmanagement.BlockManager: The block deletion will start around 2017 六月 22 13:26:18
17/06/22 13:26:18 INFO util.GSet: Computing capacity for map BlocksMap
17/06/22 13:26:18 INFO util.GSet: VM type = 64-bit
17/06/22 13:26:18 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
17/06/22 13:26:18 INFO util.GSet: capacity = 2^21 = 2097152 entries
17/06/22 13:26:18 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
17/06/22 13:26:18 INFO blockmanagement.BlockManager: defaultReplication = 2
17/06/22 13:26:18 INFO blockmanagement.BlockManager: maxReplication = 512
17/06/22 13:26:18 INFO blockmanagement.BlockManager: minReplication = 1
17/06/22 13:26:18 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
17/06/22 13:26:18 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
17/06/22 13:26:18 INFO blockmanagement.BlockManager: encryptDataTransfer = false
17/06/22 13:26:18 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
17/06/22 13:26:18 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
17/06/22 13:26:18 INFO namenode.FSNamesystem: supergroup = supergroup
17/06/22 13:26:18 INFO namenode.FSNamesystem: isPermissionEnabled = true
17/06/22 13:26:18 INFO namenode.FSNamesystem: HA Enabled: false
17/06/22 13:26:18 INFO namenode.FSNamesystem: Append Enabled: true
17/06/22 13:26:18 INFO util.GSet: Computing capacity for map INodeMap
17/06/22 13:26:18 INFO util.GSet: VM type = 64-bit
17/06/22 13:26:18 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
17/06/22 13:26:18 INFO util.GSet: capacity = 2^20 = 1048576 entries
17/06/22 13:26:18 INFO namenode.FSDirectory: ACLs enabled? false
17/06/22 13:26:18 INFO namenode.FSDirectory: XAttrs enabled? true
17/06/22 13:26:18 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
17/06/22 13:26:18 INFO namenode.NameNode: Caching file names occuring more than 10 times
17/06/22 13:26:19 INFO util.GSet: Computing capacity for map cachedBlocks
17/06/22 13:26:19 INFO util.GSet: VM type = 64-bit
17/06/22 13:26:19 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
17/06/22 13:26:19 INFO util.GSet: capacity = 2^18 = 262144 entries
17/06/22 13:26:19 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
17/06/22 13:26:19 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
17/06/22 13:26:19 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
17/06/22 13:26:19 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
17/06/22 13:26:19 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
17/06/22 13:26:19 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
17/06/22 13:26:19 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
17/06/22 13:26:19 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17/06/22 13:26:19 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/06/22 13:26:19 INFO util.GSet: VM type = 64-bit
17/06/22 13:26:19 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
17/06/22 13:26:19 INFO util.GSet: capacity = 2^15 = 32768 entries
17/06/22 13:26:19 INFO namenode.FSImage: Allocated new BlockPoolId: BP-806774403-127.0.0.1-1498109179100
17/06/22 13:26:19 INFO common.Storage: Storage directory /usr/local/hadoop/name has been successfully formatted.
17/06/22 13:26:19 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop/name/current/fsimage.ckpt_0000000000000000000 using no compression
17/06/22 13:26:19 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop/name/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds.
17/06/22 13:26:19 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/06/22 13:26:19 INFO util.ExitUtil: Exiting with status 0
17/06/22 13:26:19 INFO namenode.NameNode: SHUTDOWN_MSG:
/**************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
**************************************************/

Master

[hadoop@localhost ~]$ start-dfs.sh

Starting namenodes on [192.168.216.144]
192.168.216.144: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-localhost.out
192.168.216.146: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-localhost.out
192.168.216.145: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-localhost.out
Starting secondary namenodes [192.168.216.144]
192.168.216.144: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-localhost.out

[hadoop@localhost ~]$ jps

15009 SecondaryNameNode
14818 NameNode
15118 Jps

Slave1和Slave2

[hadoop@localhost ~]$ jps

12660 DataNode
12732 Jps

Master

[hadoop@localhost ~]$ start-yarn.sh

starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-localhost.out
192.168.216.145: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-localhost.out
192.168.216.146: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-localhost.out

[hadoop@localhost ~]$ jps

15009 SecondaryNameNode
14818 NameNode
15173 ResourceManager
15430 Jps

Slave1和Slave2

[hadoop@localhost ~]$ jps

12660 DataNode
12773 NodeManager
12869 Jps

Master

[hadoop@localhost ~]$ mr-jobhistory-daemon.sh start historyserver

starting historyserver, logging to /usr/local/hadoop/logs/mapred-hadoop-historyserver-localhost.out

[hadoop@localhost ~]$ jps

15009 SecondaryNameNode
14818 NameNode
15507 Jps
15173 ResourceManager
15470 JobHistoryServer

---

注意：貌似并未成功

[hadoop@localhost ~]$ hdfs dfsadmin -report

Configured Capacity: 0 (0 B)
Present Capacity: 0 (0 B)
DFS Remaining: 0 (0 B)
DFS Used: 0 (0 B)
DFS Used%: NaN%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

http://192.168.216.144:50070

http://192.168.216.144:8088