Hadoop HDFS copyMergeFromLocal

在谈到HDFS优化中，其中HDFS擅长处理大文件，而对于小文件常用的优化策略有压缩合并。在此列举小文件合并工具类供参考。

/*** Get all the files in the directories that match the source file pattern* and merge and sort them to only one file on HDFS is kept.* * Also adds a string between the files (useful for adding \n* to a text file)* @param srcf: a file pattern specifying source files* @param dstf: a destination local file/directory* @param endline: if an end of line character is added to a text file * @exception: IOException */public static void copyMergeFromLocal(String srcf, Path dst, boolean endline)            throws IOException {        Configuration conf = new Configuration();        Path srcPath = new Path(srcf);        FileSystem dstFs = srcPath.getFileSystem(conf);        FileSystem srcFs = FileSystem.getLocal(conf);        Path[] srcs = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);        for (Path src : srcs) {            FileUtil.copyMerge(srcFs, src,                    dstFs, dst, false, conf,                    endline ? "\n" : null);        }    }void copyMergeFromLocal(String srcf, Path dst) throws IOException {    copyMergeFromLocal(srcf, dst, false);  }

在HDFS文件上传时，可以设置过滤条件，使小文件自动合并。