hadoop配置 - using the other hard disks
来源:互联网 发布:重新安装centos 编辑:程序博客网 时间:2024/05/22 11:41
I've got a small cluster of 3 nodes with 5 x 2 TB hard drives in each computer.
All are running Ubuntu, have the same hardware specs, and are using Apache Hadoop 1.0.4.
The hard disks are mounted as /media/diskb, /media/diskc, /media/diskd, etc. on each respective computer and are configured as JBOD.
All 3 computers are serving as Data Nodes and Task Trackers, while one is the master Name Node and Secondary Name Node,
the 2nd is the Job Tracker, and the 3rd is a pure slave (DT/TT) node.
In each computer's hdfs-site.xml file, I have listed the mount points, comma-separated, no spaces as values.
For the Name Node:
In mapred-site.xml:
Also, in core-site.xml
(I've played around with changing the temp directory to be assigned to a disk at a time to check permissions, etc and Hadoop works fine)
Permissions for the mounts and ownership of the directories are full for the Hadoop user account.
When I run a map/reduce program, I can see Hadoop create resource folders inside the extra disks on each node under their mapred/local directories,
and I can see the same happening for the data node directories and the configured capacity reported on the administration page (namenode:50070).
All are running Ubuntu, have the same hardware specs, and are using Apache Hadoop 1.0.4.
The hard disks are mounted as /media/diskb, /media/diskc, /media/diskd, etc. on each respective computer and are configured as JBOD.
All 3 computers are serving as Data Nodes and Task Trackers, while one is the master Name Node and Secondary Name Node,
the 2nd is the Job Tracker, and the 3rd is a pure slave (DT/TT) node.
In each computer's hdfs-site.xml file, I have listed the mount points, comma-separated, no spaces as values.
<property> <name>dfs.data.dir</name> <value>/data/dfs/data,/media/diskb/data/dfs/data,/media/diskc/data/dfs/data,..</value></property>
For the Name Node:
<property> <name>dfs.name.dir</name> <value>/data/dfs/name,/media/diskb/data/dfs/name,/media/diskc/data/dfs/name,..</value></property>
In mapred-site.xml:
<property> <name>mapred.local.dir</name> <value>/data/mapred/local,/media/diskb/data/mapred/local,/media/diskc/data/mapred/local,...</value></property>
Also, in core-site.xml
<property> <name>hadoop.tmp.dir</name> <value>/media/diskb/data</value></property>
(I've played around with changing the temp directory to be assigned to a disk at a time to check permissions, etc and Hadoop works fine)
Permissions for the mounts and ownership of the directories are full for the Hadoop user account.
When I run a map/reduce program, I can see Hadoop create resource folders inside the extra disks on each node under their mapred/local directories,
and I can see the same happening for the data node directories and the configured capacity reported on the administration page (namenode:50070).
- hadoop配置 - using the other hard disks
- create Mailboxes using addresses other than the default
- using powershell to check the hard disk information
- find the uuid for disks
- BZOJ1510 [POI2006]Kra-The Disks
- bzoj1510: [POI2006]Kra-The Disks
- bzoj1510 [POI2006]Kra-The Disks
- unable to start the monitor on 4454 ,an other instance is problaly using the same port
- the method of resolve the problem that easyuefi can not select other hard disk as a bootable disk
- Disks
- Artificial Intelligence to optimize performance of Hybrid Hard Disks
- Day3-9.Using other components
- BZOJ 1510 [POI2006]Kra-The Disks 二分
- BZOJ 1510: [POI2006]Kra-The Disks
- bzoj1510[POI2006]Kra-The Disks 二分
- Xenomai: Hard Real Time Driver Example Tutorial with MMAP using the RTDM (Real Time Driver Model)
- Hiding Data from Forensic Imagers – Using the Service Area of a Hard Disk Drive
- On the other hand
- WNR2200可以正常地使用OpenWRT了。
- 用CSS3实现客户端的全文检索
- 基于MINA的网络应用程序架构
- Android 中点击空白处隐藏软键盘
- Python中的切片
- hadoop配置 - using the other hard disks
- 服务生成javacore 和heapdump
- CSS+DIV中的位置常用标签
- 图形与java2D小结与经验
- android 调用自己写的C函数库 代码
- open /root/.ssh/id_rsa failed: Permission denied. Saving the key failed: /root/.ssh/id_rsa.
- 电路设计中用0欧电阻还是磁珠来隔离数字地和模拟地?
- hdu 4759 - Poker Shuffle(巧妙二进制)
- android下对存储的操作