hadoop2.6伪分布部署笔记

来源：互联网发布：软件申请著作权登记编辑：程序博客网时间：2024/04/30 07:14

在Ubuntu 14.04 LTS下:

创建hadoop用户组和用户

sudo addgroup Hadoop_group

sudo adduser --ingroup Hadoop_group hadoop

管理员权限，我们给hadoop用户添加权限，打开/etc/sudoers文件

在root ALL=(ALL:ALL) ALL下添加hadoop ALL=(ALL:ALL) ALL

切换至Hadoop下

su Hadoop

安装ssh

sudo apt-get install ssh

启动：

sudo /etc/init.d/ssh start

看看是否启动：

ps aux |grep ssh

作为一个安全通信协议（ssh生成密钥有rsa和dsa两种生成方式，默认情况下采用rsa方式），使用时需要密码，因此我们要设置成免密码登录，生成私钥和公钥：

ssh-keygen -t rsa -P ""

Generating public/private rsa key pair.

Enter file in which to save the key(/home/hadoop/.ssh/id_rsa):

Created directory '/home/hadoop/.ssh'.

Your identification has been saved in/home/hadoop/.ssh/id_rsa.

Your public key has been saved in/home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

5e:78:be:21:89:c0:f2:2a:d8:c3:f2:ec:24:22:4f:05hadoop@ubuntu

The key's randomart image is:

+--[ RSA 2048]----+

| |

| E |

| o . |

| .+ S o |

| +. o = |

|=oo . . + o |

|**+. . o |

| =*. . |

+-----------------+

cat/home/hadoop/.ssh/id_rsa.pub>>/home/hadoop/.ssh/authorized_keys

下载最新版hadoop的：

http://apache.fayea.com/hadoop/common/hadoop-2.6.0/

安装hadoop至：/usr/local/hadoop下

profile中配置hadoop

/etc/profile

export HADOOP_HOME=/usr/local/hadoop

export PATH=$HADOOP_HOME/bin:$PATH

export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

source /etc/profile

etc/hadoop/core-site.xml

<name>fs.default.name</name>

<value>hdfs://localhost:9000</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/usr/local/hadoop/tmp</value>

</property>

</configuration>

etc/hadoop/hdfs-site.xml

<value>/usr/local/hadoop/hdfs/name</value>

<description>namenode存储路径</description>

</property>

<value>/usr/local/hadoop/hdfs/data</value>

<description>datanode存储路径</description>

</property>

<name>dfs.replication</name>

</property>

</configuration>

etc/hadoop/mapred-site.xml

<name>mapred.job.tracker</name>

<value>hdfs://localhost:9001</value>

</property>

<name>mapred.local.dir</name>

<value>/usr/local/hadoop/mapred/local</value>

<description>存储mapred自己使用的路径</description>

</property>

<name>mapred.system.dir</name>

<value>/usr/local/hadoop/mapred/system</value>

<description>存储mapred系统级别的路径，可以共享</description>

</property>

</configuration>

Format the filesystem:

$bin/hdfs namenode -format

启动

sbin/start-all.sh 或者：sbin/start-dfs.sh，sbin/start-yarn.sh

关闭

sbin/stop-all.sh 或者相应的：sbin/stop-dfs.sh，sbin/stop-yarn.sh

现在/usr/local/hadoop目录看上去像这个样子，这里好像有点乱：

jps//验证一下

在浏览器查看hadoop信息

http://localhost:50070

hadoop管理页面

http://localhost:8088

其它的：

cat /etc/issue:启动信息

uname -a

Ubuntu 14.04 LTS

Ctrl+Alt+T

vmware-tools复制到桌面解压

Please re-run this program as the superuser.

Execution aborted.

Sudo /home/test/Desktop/vmware-tools-distrib/vmware-install.pl

sudo是superuser do的简写

要确保所有的操作都是在用户hadoop下完成的：

sudo chown -R hadoop:Hadoop_group /usr/local/Hadoop

进入：/usr/local/hadoop/etc/hadoop

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386 /usr/lib/jvm/java-7-openjdk-i386

//export HADOOP_HOME=/usr/local/hadoop

//export PATH=$PATH:/usr/local/hadoop/bin

exportHADOOP_PREFIX=/usr/local/Hadoop

$ mkdir input

$cp etc/hadoop/*.xml input

$bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grepinput output 'dfs[a-z.]+'

$cat output/*

sudo apt-get install openssh-server

启动、停止和重启SSH:

sudo /etc/init.d/ssh start

sudo /etc/init.d/ssh stop

sudo /etc/init.d/ssh restart

或者

sudo start ssh

sudo stop ssh

sudo restart ssh

卸载SSH

先停掉SSH服务：sudo stop ssh

然后：sudo apt-get remove ssh

FATAL namenode.NameNode: Failed to start namenode.
This gives the user hadoop of group hadoop access to the folder.

sudo chown -R hadoop:hadoop_group /usr/local/hadoop

0 0