hadoop 2.4.0 使用distcp有关问题解决
来源:互联网 发布:linux 禁ping 编辑:程序博客网 时间:2024/06/05 11:05
hadoop distcp hftp://nn.xxx.xx.com:50070/user/nlp/warehouse/t_m_user_key_action /user/nlp/warehouse/dw1
出现
Caused by: java.io.IOException: Check-sum mismatch between hftp://xxx:50070/foo/yyy.yy and hdfs://dst:8020/foo/xxx.xx
引用
— Distcp using MRv2 (YARN) from a CDH3 cluster to a CDH4 cluster may fail with CRC mismatch errors
Running distcp on a CDH4 YARN cluster with a CDH3 hftp source will fail if the CRC checksum type being used is the CDH4 default (CRC32C). This is because the default checksum type was changed in CDH4 from the CDH3 default of CRC32.
Bug: HADOOP-8060
Severity: Medium
Anticipated Resolution: To be fixed in an upcoming release
Workaround: You can work around this issue by changing the CRC checksum type on the CDH4 cluster to the CDH3 default, CRC32. To do this set dfs.checksum.type to CRC32 in hdfs-site.xml.
在hdfs-site.xml文件里面添加:
<property>
<name>dfs.checksum.type</name>
<value>CRC32</value>
</property>
注意执行命令的集群已经要有另一个集群的所有hosts文件。
0 0
- hadoop 2.4.0 使用distcp有关问题解决
- hadoop distcp使用
- hadoop distcp使用
- hadoop distcp命令的使用
- hadoop distcp
- hadoop distcp
- hadoop distcp
- hadoop集群工具distcp使用笔记
- 使用distcp在hadoop集群之间拷贝文件w
- 使用hadoop distcp从ftp拷贝文件到hdfs
- Hadoop distcp命令
- hadoop命令distcp注意事项
- Hadoop distcp command error
- hadoop distcp 命令
- hadoop命令distcp注意事项
- hadoop distcp 命令
- Hadoop中的distcp
- Hadoop distcp拷贝
- 关于WP开发中的xml问题
- ASP Content Rotator之简介
- fushionCharts3用chart.setDataURL时无法传递多个参数
- Expectation Maximization
- 现有IOS设备唯一标识符方案比较
- hadoop 2.4.0 使用distcp有关问题解决
- 扫盲:什么是加德纳技术成熟度曲线?
- 分区表自动管理
- What is Observer and Observable and when we used these?
- 形状类族的中的纯虚函数
- 基础SQL语句整合
- 第十七篇:曲径通幽处,禅房花木深--初探WDDM驱动学习笔记(四)
- shell编程grep命令详解
- Android 数据库打包随APK发布