hadoop命令distcp注意事项

来源：互联网发布：国家旅游局2016年数据编辑：程序博客网时间：2024/05/01 11:22

hadoop distcp命令用于在两个不同的集群间拷贝数据，它的优点在于将数据拷贝操作作为mapred程序来执行，这样就大大提高了拷贝的速度，使用distcp命令时必须注意以下事项：

1）数据源集群的所有节点必须知道目标集群所有节点ip和host的转换关系

2）目标路径必须存在

3）命令中必须使用主机名，而不是ip地址

测试结果如下：

bin/hadoop distcp hdfs://hadoopmaster:9000/data/dw/vv/20111208/vv_20111208_05_part-00000.lzo hdfs://hadoopmaster2:9000/user/rsync/test111/12/08 17:23:43 INFO tools.DistCp: srcPaths=[hdfs://hadoopmaster:9000/data/dw/vv/20111208/vv_20111208_05_part-00000.lzo]11/12/08 17:23:43 INFO tools.DistCp: destPath=hdfs://hadoopmaster2:9000/user/rsync/test111/12/08 17:23:44 INFO tools.DistCp: sourcePathsCount=111/12/08 17:23:44 INFO tools.DistCp: filesToCopyCount=111/12/08 17:23:44 INFO tools.DistCp: bytesToCopyCount=30.2m11/12/08 17:23:45 INFO mapred.JobClient: Running job: job_201112081643_002711/12/08 17:23:46 INFO mapred.JobClient:  map 0% reduce 0%11/12/08 17:24:08 INFO mapred.JobClient:  map 100% reduce 0%11/12/08 17:24:13 INFO mapred.JobClient: Job complete: job_201112081643_002711/12/08 17:24:13 INFO mapred.JobClient: Counters: 1811/12/08 17:24:13 INFO mapred.JobClient:   Job Counters 11/12/08 17:24:13 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=1676411/12/08 17:24:13 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=011/12/08 17:24:13 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=011/12/08 17:24:13 INFO mapred.JobClient:     Launched map tasks=111/12/08 17:24:13 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=011/12/08 17:24:13 INFO mapred.JobClient:   File Input Format Counters 11/12/08 17:24:13 INFO mapred.JobClient:     Bytes Read=27011/12/08 17:24:13 INFO mapred.JobClient:   File Output Format Counters 11/12/08 17:24:13 INFO mapred.JobClient:     Bytes Written=011/12/08 17:24:13 INFO mapred.JobClient:   FileSystemCounters11/12/08 17:24:13 INFO mapred.JobClient:     HDFS_BYTES_READ=3168254411/12/08 17:24:13 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=2236111/12/08 17:24:13 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=3168212411/12/08 17:24:13 INFO mapred.JobClient:   distcp11/12/08 17:24:13 INFO mapred.JobClient:     Files copied=111/12/08 17:24:13 INFO mapred.JobClient:     Bytes copied=3168212411/12/08 17:24:13 INFO mapred.JobClient:     Bytes expected=3168212411/12/08 17:24:13 INFO mapred.JobClient:   Map-Reduce Framework11/12/08 17:24:13 INFO mapred.JobClient:     Map input records=111/12/08 17:24:13 INFO mapred.JobClient:     Spilled Records=011/12/08 17:24:13 INFO mapred.JobClient:     Map input bytes=17011/12/08 17:24:13 INFO mapred.JobClient:     Map output records=011/12/08 17:24:13 INFO mapred.JobClient:     SPLIT_RAW_BYTES=150