伪分布模式搭建hadoop

来源：互联网发布：怎么复制图片淘宝店铺编辑：程序博客网时间：2024/05/16 11:31

伪分布模式搭建hadoop，作为练习使用即可

1 免密码ssh登陆

ssh-keygen-t rsa，默认在家目录下的.ssh文件夹下面生成一个id_rsa.pub

cat ~/id_rsa.pub >> ~/.ssh/authorized_keys 注意必须都放到.ssh文件夹下面

2 四个配置文件修改

core-site.xml

<configuration>    <property>            <name>fs.defaultFS</name>            <value>hdfs://127.0.0.1:9000</value>     </property>     <property>            <name>hadoop.tmp.dir</name>            <value>file:/root/hadoop-2.7.3/tmp</value>     </property>     <property>            <name>io.file.buffer.size</name>            <value>131702</value>     </property> </configuration>

hdfs-site.xml

<configuration>    <property>             <name>dfs.namenode.name.dir</name>             <value>file:/root/hadoop-2.7.3/hdfs/name</value>     </property>     <property>             <name>dfs.datanode.data.dir</name>             <value>file:/root/hadoop-2.7.3/hdfs/data</value>     </property>     <property>             <name>dfs.replication</name>             <value>1</value>     </property>     <property>             <name>dfs.namenode.secondary.http-address</name>             <value>127.0.0.1:9001</value>     </property>     <property>             <name>dfs.webhdfs.enabled</name>             <value>true</value>     </property> </configuration>

mapred-site.xml

<configuration>  <property>        <name>mapreduce.framework.name</name>        <value>yarn</value>  </property> <property>        <name>mapreduce.jobhistory.address</name>        <value>127.0.0.1:10020</value>  </property> <property>        <name>mapreduce.jobhistory.webapp.address</name>        <value>127.0.0.1:19888</value>  </property> <property>        <name>mapreduce.tasktracker.map.tasks.maximum</name>        <value>10</value>  </property>  <property>        <name>mapreduce.tasktracker.reduce.tasks.maximum</name>        <value>4</value>  </property></configuration>

yarn-site.xml

<configuration> <!-- Site specific YARN configurationproperties -->   <property>        <name>yarn.nodemanager.aux-services</name>        <value>mapreduce_shuffle</value>    </property>     <property>        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>        <value>org.apache.hadoop.mapred.ShuffleHandler</value>    </property>     <property>        <name>yarn.resourcemanager.admin.address</name>        <value>127.0.0.1:8033</value>    </property>     <property>        <name>yarn.resourcemanager.address</name>        <value>127.0.0.1:8032</value>    </property>     <property>        <name>yarn.resourcemanager.resource-tracker.address</name>        <value>127.0.0.1:8031</value>    </property>     <property>        <name>yarn.resourcemanager.scheduler.address</name>        <value>127.0.0.1:8030</value>    </property>                                           </configuration>

3修改两个sh文件

hadoop-env.sh

export JAVA_HOME=YOUR JAVA HOME

export HADOOP_CONF_DIR=YOUR HADOOP HOME/etc/hadoop

yarn-env.sh

export JAVA_HOME=YOUR JAVA HOME

4格式化并启动hadoop

bin/hadoop namenode -format

sbin/start-all.sh

0 0