hadoop2.2.0的分布式centos上的安装(包括linux的相关配置和hadoop安装)

来源：互联网发布：期货软件编辑：程序博客网时间：2024/05/29 12:29

预先的工作：

1.挂载硬盘

1）查看硬盘情况

[root@db1 /]# fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes

255 heads, 63 sectors/track, 1305 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sda1 * 151 1305 9277537+ 83 Linux

/dev/sda2 1 150 1204843+ 82 Linux swap

Partition table entries are not in disk order

Disk /dev/sdb: 5368 MB, 5368709120 bytes

255 heads, 63 sectors/track, 652 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

从查询结果看出，多了一个/dev/sdb的盘

2）用fdisk对/dev/sdb 进行分区

[root@db1 /]# fdisk /dev/sdb

Command (m for help): n

Command action

e extended

p primary partition (1-4)

Partition number (1-4): 1

First cylinder (1-652, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-652, default 652):

Using default value 652

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

3）格式化 /dev/sdb1 分区

[root@db1 /]# mkfs -t ext3 /dev/sdb1

mke2fs 1.35 (28-Feb-2004)

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

655360 inodes, 1309289 blocks

65464 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=1342177280

40 block groups

32768 blocks per group, 32768 fragments per group

16384 inodes per group

Superblock backups stored on blocks:

32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done

Creating journal (8192 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 30 mounts or

180 days, whichever comes first. Use tune2fs -c or -i to override.

4）挂载

[root@db1 /]# mount /dev/sdb1 /mnt

5)验证挂载是否成功

[root@db1 /]# df -k

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/sda1 9131772 7066884 1601012 82% /

none 454256 0 454256 0% /dev/shm

/dev/sdb1 5154852 43040 4849956 1% /backup

6）设置开机自动挂载

vi /etc/fstab

在vi中输入i进入INERT模式，将光标移至文件结尾处并回车，将下面的内容复制/粘贴，然后按Esc键，输入:x保存并退出

/dev/xvdb1 /data ext3 defaults 0 0

2.安装java

1）安装rz,sz上传下载服务器端的文件：

yum install lrzsz

2）安装jdk8

tar -xvf jdk-8u25-linux-x64.tar

3 ) 环境变量设置：

source /etc/profile

# java8

export JAVA_HOME=/mnt/software/jdk/jdk8

export JAVA_BIN=/usr/java/jdk1.6.0_26/bin

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export JAVA_HOME JAVA_BIN PATH CLASSPATH

4)测试：

java

3.ssh无密码登陆

1）.ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop02/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop02/.ssh/id_rsa.
Your public key has been saved in /home/hadoop02/.ssh/id_rsa.pub.
The key fingerprint is:
0f:7e:45:6c:40:7c:9d:46:c5:12:31:09:5c:2c:4a:aa hadoop02@ubuntuserver2
The key's randomart image is:
+--[ RSA 2048]----+
| oo..**B.|
| oo+ B..|
| o o+o . |
| . .o |
| .S . |
| E. o . |
| . o |
| . |
| |
+-----------------+

2）本机访问无密码登陆：

hadoop02@ubuntuserver2:/root$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys

注意：cat 与.ssh之间有空格，id_rsa.pub与>>之间有空格

3）本机访问另外两台无密码登陆：

hadoop02@ubuntuserver2:~/.ssh$ scp id_rsa.pub hadoop01@192.168.1.135:/home/hadoop01

hadoop01@hadoop-master:~$ cat id_rsa.pub >> .ssh/authorized_keys

4.修改机器的名称

1）vim /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=master

NETWORKING_IPV6=no

PEERNTP=no

GATEWAY=112.126.67.247

把HOSTNAME修改为master

2）重启动生效

另外两个机器分别改为slave1和slave2

5. 关闭防火墙

正式的安装：

1. 64位已经编译的hadoop的jar文件进行解压缩

2. 需要修改的配置文件有7个

~/hadoop-2.2.0/etc/hadoop/hadoop-env.sh

~/hadoop-2.2.0/etc/hadoop/yarn-env.sh

~/hadoop-2.2.0/etc/hadoop/slaves

~/hadoop-2.2.0/etc/hadoop/core-site.xml

~/hadoop-2.2.0/etc/hadoop/hdfs-site.xml

~/hadoop-2.2.0/etc/hadoop/mapred-site.xml

~/hadoop-2.2.0/etc/hadoop/yarn-site.xm

hadoop-env.sh:修改：

export JAVA_HOME=/mnt/software/jdk/jdk8

yarn-env.sh ：

export JAVA_HOME=/mnt/software/jdk/jdk8

slaves:

slave1

slave2

core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://master:9000</value>

</property>

<name>io.file.buffer.size</name>

</property>

<name>hadoop.tmp.dir</name>

<value>/mnt/data/hadoop-2.2.0/tmp</value>

</property>

</configuration>

hdfs-site.xml

<name>dfs.namenode.secondary.http-address</name>

<value>master:9001</value>

</property>

<name>dfs.namenode.name.dir</name>

<value>file:/mnt/data/hadoop-2.2.0/dfs/name</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>file:/mnt/data/hadoop-2.2.0/dfs/data</value>

</property>

<name>dfs.replication</name>

</property>

<name>dfs.webhdfs.enabled</name>

</property>

</configuration>

mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.jobhistory.address</name>

<value>master:10020</value>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>master:19888</value>

</property>

</configuration>

yarn-site.xml

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<name>yarn.resourcemanager.address</name>

<value>master:8032</value>

</property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>master:8030</value>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>master:8031</value>

</property>

<name>yarn.resourcemanager.admin.address</name>

<value>master:8033</value>

</property>

<name>yarn.resourcemanager.webapp.address</name>

<value>master:8088</value>

</property>

</configuration>

3.加入有关hadoop的环境变量：

#hadoop

export HADOOP_HOME=/mnt/data/hadoop-2.2.0

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export PATH=$PATH:$HADOOP_HOME/bin

2.安装hbase0.96.0

1.解压缩到服务器

2.配置环境变量：

#hbase

export HBASE_HOME=/mnt/data/hbase-0.96.0-hadoop2

export PATH=$PATH:$HBASE_HOME/bin

source /etc/profiel

3.配置环境变量

hbase-env.sh

# The java implementation to use. Java 1.6 required.

export JAVA_HOME=/mnt/software/jdk/jdk8

# Tell HBase whether it should manage it's own instance of Zookeeper or not.

export HBASE_MANAGES_ZK=true #选择true,就选择了默认hbase内部托管的zookeeper

2)把hadoop上的hdfs-site.xml拷贝到hbase的conf里

3）hdfs-site.xml

<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:9000/hbase</value>
</property>

<property>
<name>hbase.tmp.dir</name>
<value>file:/mnt/data/hbase-0.96.0-hadoop2/tmp</value>
</property>

<property>
<name>hbase.master</name>
<value>hdfs://master:60000</value>
</property>

<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>file:/mnt/data/hbase-0.96.0-hadoop2/zookeeper</value>
</property>

<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
</configuration>

http://wenku.baidu.com/link?url=Wp1xO562r7SK32-HoDOSrtXsCAR1g8HEnNiRpvJBC7uyDGPclOfVHn0_S8mSguZmOQkXzuK5nd8wkqXSJMF6ySBwo7nGaZ_d9J_r-j-7_Ca

http://www.iteblog.com/archives/902

0 0