Linux_Hadoop1.2.1 安装笔记

环境：Vmware Workstation 10，CentOS-7-x86_64-DVD-1511.iso，Xshell 4.0，Master：192.168.216.141，Slave1：192.168.216.142，Slave2：192.168.216.143.

给普通用户hadoop授予sudo权限

[hadoop@localhost ~]$ su root

密码：

[root@localhost hadoop]# ll /etc/sudoers

-r–r—–. 1 root root 4188 7月 7 2015 /etc/sudoers

[root@localhost hadoop]# chmod u+w /etc/sudoers

[root@localhost hadoop]# ll /etc/sudoers

-rw-r—–. 1 root root 4188 7月 7 2015 /etc/sudoers

[root@localhost hadoop]# vim /etc/sudoers

root ALL=(ALL) ALL
hadoop ALL=(ALL) ALL

[root@localhost hadoop]# chmod u-w /etc/sudoers

[root@localhost hadoop]# ll /etc/sudoers

-r–r—–. 1 root root 4215 6月 20 17:56 /etc/sudoers

[root@localhost hadoop]# exit

exit

---

Master服务器设置ssh无密码登录

[hadoop@localhost ~]$ ssh-keygen -t rsa -P ”

Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory ‘/home/hadoop/.ssh’.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

[hadoop@localhost ~]$ ll ~/.ssh

总用量 8
-rw——-. 1 hadoop hadoop 1675 6月 20 18:02 id_rsa
-rw-r–r–. 1 hadoop hadoop 398 6月 20 18:02 id_rsa.pub

[hadoop@localhost ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[hadoop@localhost ~]$ chmod 600 ~/.ssh/authorized_keys

[hadoop@localhost ~]$ ll ~/.ssh/

authorized_keys的权限必须是600

总用量 16
-rw——-. 1 hadoop hadoop 398 6月 20 18:03 authorized_keys
-rw——-. 1 hadoop hadoop 1675 6月 20 18:02 id_rsa
-rw-r–r–. 1 hadoop hadoop 398 6月 20 18:02 id_rsa.pub

[hadoop@localhost ~]$ sudo vim /etc/ssh/sshd_config

AuthorizedKeysFile .ssh/authorized_keys

[hadoop@localhost ~]$ ssh localhost

The authenticity of host ‘localhost (::1)’ can’t be established.
ECDSA key fingerprint is 1b:1d:06:bb:5c:dd:92:e6:22:ff:03:b3:a7:c0:e7:ed.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘localhost’ (ECDSA) to the list of known hosts.
hadoop@localhost’s password:
Last login: Tue Jun 20 17:44:21 2017 from 192.168.216.1

[hadoop@localhost ~]$ su

密码：

[root@localhost hadoop]# service sshd restart

Redirecting to /bin/systemctl restart sshd.service

[root@localhost hadoop]# exit

exit

[hadoop@localhost ~]$ ssh localhost

The authenticity of host ‘localhost (::1)’ can’t be established.
ECDSA key fingerprint is 1b:1d:06:bb:5c:dd:92:e6:22:ff:03:b3:a7:c0:e7:ed.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘localhost’ (ECDSA) to the list of known hosts.
Last failed login: Tue Jun 20 18:09:37 CST 2017 from localhost on ssh:notty
There were 4 failed login attempts since the last successful login.
Last login: Tue Jun 20 18:06:00 2017 from localhost

将~/.ssh文件夹所有发送给Slave1和Slave2：

[hadoop@localhost ~]$ scp -r ~/.ssh hadoop@192.168.216.142:~

hadoop@192.168.216.142’s password:
id_rsa 100% 1675 1.6KB/s 00:00
id_rsa.pub 100% 398 0.4KB/s 00:00
authorized_keys 100% 398 0.4KB/s 00:00
known_hosts 100% 348 0.3KB/s 00:00

[hadoop@localhost ~]$ scp -r ~/.ssh hadoop@192.168.216.143:~

The authenticity of host ‘192.168.216.143 (192.168.216.143)’ can’t be established.
ECDSA key fingerprint is 64:05:c3:ec:5e:cb:64:d8:41:74:d6:ee:cb:c3:84:59.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘192.168.216.143’ (ECDSA) to the list of known hosts.
hadoop@192.168.216.143’s password:
id_rsa 100% 1675 1.6KB/s 00:00
id_rsa.pub 100% 398 0.4KB/s 00:00
authorized_keys 100% 398 0.4KB/s 00:00
known_hosts 100% 525 0.5KB/s 00:00

接下来可以测试Master，Slave1和Slave2相互免密码登录。
Master服务器

[hadoop@localhost ~]$ ssh 192.168.216.142

Last login: Tue Jun 20 18:24:22 2017 from 192.168.216.142

[hadoop@localhost ~]$ ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno16777736: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9e:9f:16 brd ff:ff:ff:ff:ff:ff
inet 192.168.216.142/24 brd 192.168.216.255 scope global dynamic eno16777736
valid_lft 1148sec preferred_lft 1148sec
inet6 fe80::20c:29ff:fe9e:9f16/64 scope link
valid_lft forever preferred_lft forever

[hadoop@localhost ~]$ exit

登出
Connection to 192.168.216.142 closed.

[hadoop@localhost ~]$ ssh 192.168.216.143

Last login: Tue Jun 20 17:30:54 2017 from 192.168.216.1

[hadoop@localhost ~]$ exit

登出
Connection to 192.168.216.143 closed.

同理从Slave1到Master, Slave2登录；Slave2到Master, Slave1登录。

至此准备阶段结束

---

Master服务器

[hadoop@localhost ~]$ wget http://archive.apache.org/dist/hadoop/core/hadoop-1.2.1/hadoop-1.2.1.tar.gz

[hadoop@localhost ~]$ sudo tar -zxvf hadoop-1.2.1.tar.gz -C /usr/local/

[hadoop@localhost ~]$ sudo mv /usr/local/hadoop-1.2.1/ /usr/local/hadoop

[hadoop@localhost ~]$ ll /usr/local/ | grep hadoop

drwxr-xr-x. 15 root root 4096 7月 23 2013 hadoop

[hadoop@localhost ~]$ sudo chown -R hadoop:hadoop /usr/local/hadoop/

[hadoop@localhost ~]$ ll /usr/local/ | grep hadoop

drwxr-xr-x. 15 hadoop hadoop 4096 7月 23 2013 hadoop

[hadoop@localhost ~]$ sudo vim /etc/profile

[hadoop@localhost ~]$ source /etc/profile

[hadoop@localhost ~]$ hadoop

Usage: hadoop [–config confdir] COMMAND

配置文件

[hadoop@localhost ~]$ vim /usr/local/hadoop/conf/hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_131

[hadoop@localhost ~]$ vim /usr/local/hadoop/conf/core-site.xml

[hadoop@localhost ~]$ vim /usr/local/hadoop/conf/hdfs-site.xml

[hadoop@localhost ~]$ vim /usr/local/hadoop/conf/mapred-site.xml

[hadoop@localhost ~]$ vim /usr/local/hadoop/conf/masters

192.168.216.141

[hadoop@localhost ~]$ vim /usr/local/hadoop/conf/slaves

192.168.216.142
192.168.216.143

[hadoop@localhost ~]$ cd /usr/local/hadoop/ && mkdir datanode tmp

[hadoop@localhost ~]$ chmod 755 /usr/local/hadoop/tmp

[hadoop@localhost ~]$ chmod 755 /usr/local/hadoop/datanode

注意：datanode和tmp目录的权限必须是755，否则datanode启动失败。

将/usr/local/hadoop整个文件夹发送到Slave1和Slave2：

[hadoop@localhost ~]$ scp -r /usr/local/hadoop/ 192.168.216.142:~

[hadoop@localhost ~]$ scp -r /usr/local/hadoop/ 192.168.216.143:~

Slave1和Slave2上操作

[hadoop@localhost ~]$ sudo mv hadoop /usr/local/

[hadoop@localhost ~]$ sudo vim /etc/profile

[hadoop@localhost ~]$ source /etc/profile

[hadoop@localhost ~]$ hadoop

Usage: hadoop [–config confdir] COMMAND

至此hadoop一主两从部署完毕

---

Master，Slave1和Slave2

关闭防火墙
注意：下面命令中“–”是两个“-”，csdn显示有误。

[hadoop@localhost hadoop]$ firewall-cmd –state

running

[hadoop@localhost hadoop]$ sudo systemctl stop firewalld

[sudo] password for hadoop:

[hadoop@localhost hadoop]$ firewall-cmd –state

not running

格式化namenode文件系统

[hadoop@localhost hadoop]$ hadoop namenode -format

17/06/22 14:16:49 INFO namenode.NameNode: STARTUP_MSG:
/**************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = localhost/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by ‘mattf’ on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.8.0_131
**************************************************/
17/06/22 14:16:49 INFO util.GSet: Computing capacity for map BlocksMap
17/06/22 14:16:49 INFO util.GSet: VM type = 64-bit
17/06/22 14:16:49 INFO util.GSet: 2.0% max memory = 1013645312
17/06/22 14:16:49 INFO util.GSet: capacity = 2^21 = 2097152 entries
17/06/22 14:16:49 INFO util.GSet: recommended=2097152, actual=2097152
17/06/22 14:16:49 INFO namenode.FSNamesystem: fsOwner=hadoop
17/06/22 14:16:49 INFO namenode.FSNamesystem: supergroup=supergroup
17/06/22 14:16:49 INFO namenode.FSNamesystem: isPermissionEnabled=false
17/06/22 14:16:49 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
17/06/22 14:16:49 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
17/06/22 14:16:49 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
17/06/22 14:16:49 INFO namenode.NameNode: Caching file names occuring more than 10 times
17/06/22 14:16:50 INFO common.Storage: Image file /usr/local/hadoop/name/current/fsimage of size 112 bytes saved in 0 seconds.
17/06/22 14:16:50 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/hadoop/name/current/edits
17/06/22 14:16:50 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/hadoop/name/current/edits
17/06/22 14:16:50 INFO common.Storage: Storage directory /usr/local/hadoop/name has been successfully formatted.
17/06/22 14:16:50 INFO namenode.NameNode: SHUTDOWN_MSG:
/**************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
**************************************************/
Master

[hadoop@localhost hadoop]$ start-all.sh

starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-namenode-localhost.out
192.168.216.143: datanode running as process 12402. Stop it first.
192.168.216.142: datanode running as process 12458. Stop it first.
The authenticity of host ‘192.168.216.141 (192.168.216.141)’ can’t be established.
ECDSA key fingerprint is 45:3b:37:83:b6:7f:20:55:47:10:b7:ad:63:ae:96:33.
Are you sure you want to continue connecting (yes/no)? yesw^H
192.168.216.141: Warning: Permanently added ‘192.168.216.141’ (ECDSA) to the list of known hosts.
192.168.216.141: secondarynamenode running as process 12610. Stop it first.
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-localhost.out
192.168.216.143: tasktracker running as process 12673. Stop it first.
192.168.216.142: tasktracker running as process 12731. Stop it first.

[hadoop@localhost hadoop]$ jps

12610 SecondaryNameNode
12866 Jps
12758 JobTracker
12442 NameNode

[hadoop@localhost hadoop]$ hadoop dfsadmin -report

Configured Capacity: 37492883456 (34.92 GB)
Present Capacity: 34187776030 (31.84 GB)
DFS Remaining: 34187759616 (31.84 GB)
DFS Used: 16414 (16.03 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

---

Datanodes available: 2 (2 total, 0 dead)

Name: 192.168.216.142:50010
Decommission Status : Normal
Configured Capacity: 18746441728 (17.46 GB)
DFS Used: 8207 (8.01 KB)
Non DFS Used: 1652527089 (1.54 GB)
DFS Remaining: 17093906432(15.92 GB)
DFS Used%: 0%
DFS Remaining%: 91.18%
Last contact: Thu Jun 22 14:19:56 CST 2017

Name: 192.168.216.143:50010
Decommission Status : Normal
Configured Capacity: 18746441728 (17.46 GB)
DFS Used: 8207 (8.01 KB)
Non DFS Used: 1652580337 (1.54 GB)
DFS Remaining: 17093853184(15.92 GB)
DFS Used%: 0%
DFS Remaining%: 91.18%
Last contact: Thu Jun 22 14:19:56 CST 2017

Slave1

[hadoop@localhost hadoop]$ start-all.sh

starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-namenode-localhost.out
192.168.216.143: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-localhost.out
192.168.216.142: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-localhost.out
192.168.216.141: secondarynamenode running as process 12610. Stop it first.
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-localhost.out
192.168.216.142: tasktracker running as process 12731. Stop it first.
192.168.216.143: tasktracker running as process 12673. Stop it first.

[hadoop@localhost hadoop]$ jps

12458 DataNode
12731 TaskTracker
12847 Jps

Slave2

[hadoop@localhost hadoop]$ start-all.sh

starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-namenode-localhost.out
192.168.216.143: datanode running as process 12402. Stop it first.
192.168.216.142: datanode running as process 12458. Stop it first.
192.168.216.141: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-localhost.out
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-localhost.out
192.168.216.142: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-localhost.out
192.168.216.143: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-localhost.out

[hadoop@localhost hadoop]$ jps

12673 TaskTracker
12402 DataNode
12793 Jps

---

http://192.168.216.141:50070

http://192.168.216.141:50030