基于hadoop2.2的namenode与SecondaryNameNode分开配置在不同的计算机
来源:互联网 发布:传奇mac 编辑:程序博客网 时间:2024/05/23 18:33
前段时间由于配置hadoop做实验,在网上看了许多有关hadoop的配置,但是这些配置多数是将namenode和secondaryNameNode配置在同一台计算机上,这种配置方法如果是做做实验的还可以,如果应用到实际中,存在较大风险,
如果存放namenode的主机出现问题,整个文件系统将被破坏,严重的情况是所有文件都丢失。现在来配置hadoop2.2,将namenode和secondaryNameNode配置在不同的机器上,这样的实用价值更大。
1.假设我们已经配置好了hadoop2.2,这时namenode和secondaryNameNode在同一台机器上,如cloud001;
2.单独配置secondaryNameNode文件,指明secondaryNameNode所在的计算机地址,需要masters文件(类似slaves文件指明从节点),而我下载的hadoop2.2配置文件中不包含masters,
所以我们新建一个masters文件,编辑内容为:
sudo vim masters
cloud002
此处将cloud002作为secondaryNameNode的主机。
3.修改hdfs-site.xml的内容
<!--
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>cloud001:9001</value>
</property>
-->
注释掉之前使namenode与SecondaryNameNode在同一台主机的代码,并新添如下代码
<property>
<name>dfs.http.address</name>
<value>cloud001:50070</value>
<description>
The address and the base port where the dfs namenode web ui will listen on.
If the port is 0 then the server will start on a free port.
</description>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>cloud002:50090</value>
</property>
网上也有说要修改core-site.xml的代码,而我之前配置过,所以现在就不需要再改了。修改的内容如下:
<property>
<name>fs.checkpoint.period</name>
<value>60</value>
<description>The number of seconds between two periodic checkpoints.
</description>
</property>
<property>
<name>fs.checkpoint.size</name>
<value>67108864</value>
</property>
4.以上修改只是在namenode上修改,在集群中的其他节点也要作相应修改,简便方法便是将需要修改的文件覆盖掉。
5.启动hdfs,在hadoop目录下执行:
sbin/start-dfs.sh
xuhui@cloud001:~/hadoop-2.2.0$ sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [cloud001]
cloud001: starting namenode, logging to /home/xuhui/hadoop-2.2.0/logs/hadoop-xuhui-namenode-cloud001.out
cloud004: starting datanode, logging to /home/xuhui/hadoop-2.2.0/logs/hadoop-xuhui-datanode-cloud004.out
cloud002: starting datanode, logging to /home/xuhui/hadoop-2.2.0/logs/hadoop-xuhui-datanode-cloud002.out
Starting secondary namenodes [cloud002]
cloud002: starting secondarynamenode, logging to /home/xuhui/hadoop-2.2.0/logs/hadoop-xuhui-secondarynamenode-cloud002.out
starting yarn daemons
starting resourcemanager, logging to /home/xuhui/hadoop-2.2.0/logs/yarn-xuhui-resourcemanager-cloud001.out
cloud004: starting nodemanager, logging to /home/xuhui/hadoop-2.2.0/logs/yarn-xuhui-nodemanager-cloud004.out
cloud002: starting nodemanager, logging to /home/xuhui/hadoop-2.2.0/logs/yarn-xuhui-nodemanager-cloud002.out
xuhui@cloud001:~/hadoop-2.2.0$ ls
6.检查配置成功与否:
xuhui@cloud001:~/hadoop-2.2.0$ jps
3490 ResourceManager
3123 NameNode
4784 Jps
xuhui@cloud001:~/hadoop-2.2.0$
xuhui@cloud002:~$ jps
7162 SecondaryNameNode
7412 NodeManager
6624 DataNode
7698 Jps
xuhui@cloud002:~$
xuhui@cloud004:~$ jps
6287 NodeManager
6409 Jps
5836 DataNode
xuhui@cloud004:~$
验证结果很显然是配置成功的!
7.关闭hdfs:
xuhui@cloud001:~/hadoop-2.2.0$ sbin/stop-dfs.sh
Stopping namenodes on [cloud001]
cloud001: stopping namenode
cloud004: stopping datanode
cloud002: stopping datanode
Stopping secondary namenodes [cloud002]
cloud002: stopping secondarynamenode
xuhui@cloud001:~/hadoop-2.2.0$
如果存放namenode的主机出现问题,整个文件系统将被破坏,严重的情况是所有文件都丢失。现在来配置hadoop2.2,将namenode和secondaryNameNode配置在不同的机器上,这样的实用价值更大。
1.假设我们已经配置好了hadoop2.2,这时namenode和secondaryNameNode在同一台机器上,如cloud001;
2.单独配置secondaryNameNode文件,指明secondaryNameNode所在的计算机地址,需要masters文件(类似slaves文件指明从节点),而我下载的hadoop2.2配置文件中不包含masters,
所以我们新建一个masters文件,编辑内容为:
sudo vim masters
cloud002
此处将cloud002作为secondaryNameNode的主机。
3.修改hdfs-site.xml的内容
<!--
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>cloud001:9001</value>
</property>
-->
注释掉之前使namenode与SecondaryNameNode在同一台主机的代码,并新添如下代码
<property>
<name>dfs.http.address</name>
<value>cloud001:50070</value>
<description>
The address and the base port where the dfs namenode web ui will listen on.
If the port is 0 then the server will start on a free port.
</description>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>cloud002:50090</value>
</property>
网上也有说要修改core-site.xml的代码,而我之前配置过,所以现在就不需要再改了。修改的内容如下:
<property>
<name>fs.checkpoint.period</name>
<value>60</value>
<description>The number of seconds between two periodic checkpoints.
</description>
</property>
<property>
<name>fs.checkpoint.size</name>
<value>67108864</value>
</property>
4.以上修改只是在namenode上修改,在集群中的其他节点也要作相应修改,简便方法便是将需要修改的文件覆盖掉。
5.启动hdfs,在hadoop目录下执行:
sbin/start-dfs.sh
xuhui@cloud001:~/hadoop-2.2.0$ sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [cloud001]
cloud001: starting namenode, logging to /home/xuhui/hadoop-2.2.0/logs/hadoop-xuhui-namenode-cloud001.out
cloud004: starting datanode, logging to /home/xuhui/hadoop-2.2.0/logs/hadoop-xuhui-datanode-cloud004.out
cloud002: starting datanode, logging to /home/xuhui/hadoop-2.2.0/logs/hadoop-xuhui-datanode-cloud002.out
Starting secondary namenodes [cloud002]
cloud002: starting secondarynamenode, logging to /home/xuhui/hadoop-2.2.0/logs/hadoop-xuhui-secondarynamenode-cloud002.out
starting yarn daemons
starting resourcemanager, logging to /home/xuhui/hadoop-2.2.0/logs/yarn-xuhui-resourcemanager-cloud001.out
cloud004: starting nodemanager, logging to /home/xuhui/hadoop-2.2.0/logs/yarn-xuhui-nodemanager-cloud004.out
cloud002: starting nodemanager, logging to /home/xuhui/hadoop-2.2.0/logs/yarn-xuhui-nodemanager-cloud002.out
xuhui@cloud001:~/hadoop-2.2.0$ ls
6.检查配置成功与否:
xuhui@cloud001:~/hadoop-2.2.0$ jps
3490 ResourceManager
3123 NameNode
4784 Jps
xuhui@cloud001:~/hadoop-2.2.0$
xuhui@cloud002:~$ jps
7162 SecondaryNameNode
7412 NodeManager
6624 DataNode
7698 Jps
xuhui@cloud002:~$
xuhui@cloud004:~$ jps
6287 NodeManager
6409 Jps
5836 DataNode
xuhui@cloud004:~$
验证结果很显然是配置成功的!
7.关闭hdfs:
xuhui@cloud001:~/hadoop-2.2.0$ sbin/stop-dfs.sh
Stopping namenodes on [cloud001]
cloud001: stopping namenode
cloud004: stopping datanode
cloud002: stopping datanode
Stopping secondary namenodes [cloud002]
cloud002: stopping secondarynamenode
xuhui@cloud001:~/hadoop-2.2.0$
阅读全文
0 0
- 基于hadoop2.2的namenode与SecondaryNameNode分开配置在不同的计算机
- 基于hadoop2.2的namenode与SecondaryNameNode分开配置在不同的计算机
- hadoop2.X如何将namenode与SecondaryNameNode分开配置
- hadoop2.X如何将namenode与SecondaryNameNode分开配置
- hadoop2.X如何将namenode与SecondaryNameNode分开配置
- hadoop2.X如何将namenode与SecondaryNameNode分开配置
- NameNode 与 SecondaryNameNode 的关系
- hadoop的namenode和secondnamenode分开部署在不同服务器
- Hadoop的namenode和secondnamenode分开部署在不同服务器
- Hadoop的namenode和secondnamenode分开部署在不同服务器
- hadoop1.x下的NameNode与SecondaryNameNode
- Hadoop的NameNode与SecondaryNameNode,DataNode
- Hadoop中namenode/secondarynamenode/yarn的一些配置
- secondaryNamenode配置与nameNode故障恢复
- Hadoop中namenode和secondarynamenode分开部署
- NameNode跟secondarynamenode的执行过程
- Hadoop之NameNode和SecondaryNameNode的区别
- 将namenode与SecondNamenode分离在不同的主机上
- FlatList测试
- PinPoint使用教程
- 数据结构实验之查找五:平方之哈希表
- MyBatis入门教程
- Protostuff序列化
- 基于hadoop2.2的namenode与SecondaryNameNode分开配置在不同的计算机
- JAVA中上转型对象的详细解释
- 如何相互转换逗号分隔的字符串和List (Java程序员日记 2015/03/06)
- 比较全面的正则表达式使用教程
- CodeForces839B Game of the Rows
- Cookies 和 Session的区别
- Java设计模式
- mysql-python的安装问题
- 大数据和人工智能的相关比赛