Faster Datanodes with less wait io using df instead of du
来源:互联网 发布:女高中生知乎 编辑:程序博客网 时间:2024/06/01 09:40
I have noticed often that the check Hadoop uses to calculate usage for the data nodes causes a fair amount of wait io on them driving up load.
Every cycle we can get from every spindle we want!
So I came up with a nice little hack to use df instead of du.
Here is basically what I did so you can do it too.
mv /usr/bin/du /usr/bin/bak_du
vi /usr/bin/du
and save this inside of it
#!/bin/sh
mydf=$(df $2 | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $3 }')
echo -e "$mydf\t$2"
then give it execute permission
chmod a+x /usr/bin/du
restart you data node check the log for no errors and make sure it starts back up
viola
Now when Hadoop calls “du -sk /yourhdfslocation” it will be expedient with its results
whats wrong with this?
1) I assume you have nothing else on your disks that you are storing so df is really close to du since almost all of your data is in HDFS
2) If you have more than 1 volume holding your hdfs blocks this is not exactly accurate so you are skewing the size of each vol by only calculating one of them and using that result for the others…. this is simple to fix just parse your df result differently and use the path passed into the second paramater to know which vol to grep in your df result… your first volume is going to be larger anyways most likely and you should be monitoring disk space another way so it is not going to be very harmefull if you just check and report the first volume’s size
3) you might not have your HDFS blocks on your first volume …. see #2 you can just grep the volume you want to report
/*
Joe Stein
http://www.linkedin.com/in/charmalloc
*/
zz: http://allthingshadoop.com/2011/05/20/faster-datanodes-with-less-wait-io-using-df-instead-of-du/
- Faster Datanodes with less wait io using df instead of du
- Using SingleTagSectionHandler Instead Of appSettings
- df du
- du&df
- Practice of using spinlock instead of mutex
- Why Using Sockets instead of Remoting
- Using dmsetup directly instead of cryptsetup
- df du mpstat
- 查看磁盘空间 df, du
- df and du
- du df 命令
- du df区别
- linux df vs du
- du df 命令详解
- du与df
- 磁盘管理 du df
- df 和 du
- df与du
- C++命名规则(Important Knowledge)
- Hibernate 调用函数及过程
- UTF-8规范
- 获取子窗口句柄EnumChildWindows的威力
- Spring从菜鸟到高手(一)实现AOP的基本原理
- Faster Datanodes with less wait io using df instead of du
- winform下UPD通信的简单应用
- Spring从菜鸟到高手(二)AOP的真正实现
- 常见加密算法分,用途,原理以及比较
- Spring从菜鸟到高手(三)依赖注入
- 常见加密算法分类
- Ubuntu 12.04 编译内核步骤 + Fedora 10 编译内核步骤(图)
- SSL算法库的理解与测试
- 现实迷途 第二十五章 重见故人