Linux技巧:一次删除一百万个文件的最快方法
来源:互联网 发布:oracle与mysql的区别 编辑:程序博客网 时间:2024/05/01 02:04
最初的测评
昨天,我看到一个非常有趣的删除一个目录下的海量文件的方法。这个方法来自http://www.quora.com/How-can-someone-rapidly-delete-400-000-files里的Zhenyu Lee。
他没有使用find
或 xargs
,他很有创意的利用了rsync的强大功能,使用rsync –delete
将目标文件夹以一个空文件夹来替换。之后,我做了一个实验来比较各种方法。让我吃惊的是,Lee的方法要比其它的快的多。下面就是我的测评。
环境:
- CPU: Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
- MEM: 4G
- HD: ST3250318AS: 250G/7200RPM
使用 –delete
和 –exclude
,你可以选择性删除符合条件的文件。还有一点,当你需要保留这个目录做其它用处时,这种方法是再适合不过了。
重新测评
几天前,Keith-Winstein在回复Quora上的这个帖子时说我之前的测评无法复制,因为操作的时间持续的太久。我澄清一下,这些数据过大,可能是因为我的计算机在过去的几年里做的事太多,测评中可能存在一些文件系统错误。但我不确定是这些原因。现在好了,我弄了一天比较新的计算机,把测评再做一次。这次我使用/usr/bin/time
,它能提供更详细的信息。下面就是新的结果。
(每次都是1000000个文件。每个文件的体积都是0。)
原始输出
# method 1~/test $ /usr/bin/time -v rsync -a --delete empty/ a/ Command being timed: "rsync -a --delete empty/ a/" User time (seconds): 1.31 System time (seconds): 10.60 Percent of CPU this job got: 95% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:12.42 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 0 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 24378 Voluntary context switches: 106 Involuntary context switches: 22 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0# method 2 Command being timed: "find b/ -type f -delete" User time (seconds): 0.41 System time (seconds): 14.46 Percent of CPU this job got: 52% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:28.51 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 0 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 11749 Voluntary context switches: 14849 Involuntary context switches: 11 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0# method 3find c/ -type f | xargs -L 100 rm~/test $ /usr/bin/time -v ./delete.sh Command being timed: "./delete.sh" User time (seconds): 2.06 System time (seconds): 20.60 Percent of CPU this job got: 54% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:41.69 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 0 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 1764225 Voluntary context switches: 37048 Involuntary context switches: 15074 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0# method 4find d/ -type f | xargs -L 100 -P 100 rm~/test $ /usr/bin/time -v ./delete.sh Command being timed: "./delete.sh" User time (seconds): 2.86 System time (seconds): 27.82 Percent of CPU this job got: 89% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:34.32 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 0 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 1764278 Voluntary context switches: 929897 Involuntary context switches: 21720 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0# method 5~/test $ /usr/bin/time -v rm -rf f Command being timed: "rm -rf f" User time (seconds): 0.20 System time (seconds): 14.80 Percent of CPU this job got: 47% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:31.29 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 0 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 176 Voluntary context switches: 15134 Involuntary context switches: 11 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0
我真的十分好奇为什么Lee的方法要比其它的快,竟然比rm -rf
也要快。如果有人知道,请写在下面,非常感谢。
[英文原文: A faster way to delete millions of files in a directory ]
- Linux技巧:一次删除一百万个文件的最快方法
- Linux技巧:一次删除一百万个文件的最快方法
- Linux技巧:一次删除一百万个文件的最快方法
- Linux技巧:一次删除一百万个文件的最快方法
- linux清除文件内容最快的方法
- linux 删除问题 一次删除多个目录下的相同扩展名的文件
- 删除文件、目录最快的命令
- 多行删除最快的方法
- ORACLE大批量数据最快的删除方法
- linux远程传输文件最快方法
- 一次过删除多个ID方法
- 解决linux无法一次删除许多文件的问题
- AFNetworking 一次上传多个文件的方法
- 寻找最快的大文件拷贝方法
- C# 最快的大文件拷贝方法
- java文件复制最快的方法!
- Java: 复制文件最快、高效率的方法
- SQL 查询表中记录最快的方法,删除表中记录最快的方法.
- GetAdapterInfo得到MAC地址精简版C++源码
- spring事务传播特性,即spring管理事务的策略
- Linux 为什么要用字符 ~ 来表示 home 目录
- ORACLE 使用DBMS_METADATA.GET_DDL获取DDL语句
- C++学习之旅——函数重载代码,必须得看看怎么写的
- Linux技巧:一次删除一百万个文件的最快方法
- 从文档相似度计算看LSH(Locality Sensitive Hashing)
- ThinkPhp上传服务器无法加载模块问题总结
- hdu 4452
- DP练习(初级):ZigZag
- 常见设计模式的解析和实现(C++)之二十一-完结篇
- 返疯乐
- 佐治亚理工学院 计算投资公开课第六周作业 投资策略分析平台
- 国庆节游嵩山