hadoop命令distcp注意事项
来源:互联网 发布:战地3正在同步云数据 编辑:程序博客网 时间:2024/05/16 07:27
地址:http://blog.csdn.net/azhao_dn/article/details/7054286#
hadoop distcp命令用于在两个不同的集群间拷贝数据,它的优点在于将数据拷贝操作作为mapred程序来执行,这样就大大提高了拷贝的速度,使用distcp命令时必须注意以下事项:
1)数据源集群 的所有节点必须 知道目标集群所有节点ip和host的转换关系
2)目标路径必须存在
3)命令中必须使用主机名,而不是ip地址
测试结果如下:
- bin/hadoop distcp hdfs://hadoopmaster:9000/data/dw/vv/20111208/vv_20111208_05_part-00000.lzo hdfs://hadoopmaster2:9000/user/rsync/test1
- 11/12/08 17:23:43 INFO tools.DistCp: srcPaths=[hdfs://hadoopmaster:9000/data/dw/vv/20111208/vv_20111208_05_part-00000.lzo]
- 11/12/08 17:23:43 INFO tools.DistCp: destPath=hdfs://hadoopmaster2:9000/user/rsync/test1
- 11/12/08 17:23:44 INFO tools.DistCp: sourcePathsCount=1
- 11/12/08 17:23:44 INFO tools.DistCp: filesToCopyCount=1
- 11/12/08 17:23:44 INFO tools.DistCp: bytesToCopyCount=30.2m
- 11/12/08 17:23:45 INFO mapred.JobClient: Running job: job_201112081643_0027
- 11/12/08 17:23:46 INFO mapred.JobClient: map 0% reduce 0%
- 11/12/08 17:24:08 INFO mapred.JobClient: map 100% reduce 0%
- 11/12/08 17:24:13 INFO mapred.JobClient: Job complete: job_201112081643_0027
- 11/12/08 17:24:13 INFO mapred.JobClient: Counters: 18
- 11/12/08 17:24:13 INFO mapred.JobClient: Job Counters
- 11/12/08 17:24:13 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=16764
- 11/12/08 17:24:13 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
- 11/12/08 17:24:13 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
- 11/12/08 17:24:13 INFO mapred.JobClient: Launched map tasks=1
- 11/12/08 17:24:13 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
- 11/12/08 17:24:13 INFO mapred.JobClient: File Input Format Counters
- 11/12/08 17:24:13 INFO mapred.JobClient: Bytes Read=270
- 11/12/08 17:24:13 INFO mapred.JobClient: File Output Format Counters
- 11/12/08 17:24:13 INFO mapred.JobClient: Bytes Written=0
- 11/12/08 17:24:13 INFO mapred.JobClient: FileSystemCounters
- 11/12/08 17:24:13 INFO mapred.JobClient: HDFS_BYTES_READ=31682544
- 11/12/08 17:24:13 INFO mapred.JobClient: FILE_BYTES_WRITTEN=22361
- 11/12/08 17:24:13 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=31682124
- 11/12/08 17:24:13 INFO mapred.JobClient: distcp
- 11/12/08 17:24:13 INFO mapred.JobClient: Files copied=1
- 11/12/08 17:24:13 INFO mapred.JobClient: Bytes copied=31682124
- 11/12/08 17:24:13 INFO mapred.JobClient: Bytes expected=31682124
- 11/12/08 17:24:13 INFO mapred.JobClient: Map-Reduce Framework
- 11/12/08 17:24:13 INFO mapred.JobClient: Map input records=1
- 11/12/08 17:24:13 INFO mapred.JobClient: Spilled Records=0
- 11/12/08 17:24:13 INFO mapred.JobClient: Map input bytes=170
- 11/12/08 17:24:13 INFO mapred.JobClient: Map output records=0
- 11/12/08 17:24:13 INFO mapred.JobClient: SPLIT_RAW_BYTES=150
- hadoop命令distcp注意事项
- hadoop命令distcp注意事项
- hadoop命令distcp注意事项
- Hadoop distcp命令
- hadoop distcp 命令
- hadoop distcp 命令
- hadoop分布式复制命令distcp
- hadoop distcp命令的使用
- hadoop distcp(分布式复制)命令
- Hadoop distcp 命令跨集群复制文件
- hadoop distcp
- hadoop distcp
- hadoop distcp
- Hadoop的distcp命令出现Permission denied错误
- Hadoop distcp command error
- Hadoop中的distcp
- Hadoop distcp拷贝
- Hadoop中一个distcp
- 见证海马S7手动型的高配置
- CSS 布局:40个教程、技巧、例子和最佳实践
- repo 的一些用法和理解
- CAS单点登录https协议修改成http协议
- 手机电容触摸屏技术简介
- hadoop命令distcp注意事项
- 怎么提高网站用户体验 降低跳出率
- Hashtable的使用
- Socket简介
- Android基础概念Context的详解
- 关于SQL视图的创建和使用方法
- Spring Scope
- myeclipse10以下安装 activiti 插件
- C++中利用多线程实现定时器