hadoop与s3之间互相传输文件

来源:互联网 发布:mac ps cs6 编辑:程序博客网 时间:2024/05/16 16:05
  1. hadoop到s3
    hadoop distcp hdfs:/user/hive/data/SearchRecord/2014-02-16 s3n://*/hadoop/user/hive/data/SearchRecord/2014-02-16 

    需要在core-site.xml中配置好fs.s3n.awsAccessKeyId和fs.s3n.awsSecretAccessKey

    注意s3的路径需要使用s3n

  2. s3到hadoop
    hadoop distcp s3n://*/hadoop/user/hive/data/SearchRecord/2014-02-16/ hdfs:/user/hadoop/test/SearchRecord/2014-02-16
0 0