Hadoop REST API -- WebHDFS(下)
来源:互联网 发布:互助系统源码出售 编辑:程序博客网 时间:2024/04/29 06:42
下面主要介绍几种常用的WebHDFS操作。
5.1 创建
创建一个目录/tmp/webhdfs。
<span style="font-size:18px;">curl-i -X PUT"http://hadoop-master:14000/webhdfs/v1/tmp/webhdfs?user.name=app&op=MKDIRS"HTTP/1.1 200 OKServer: Apache-Coyote/1.1Set-Cookie:hadoop.auth="u=app&p=app&t=simple&e=1393356198808&s=74NhnIdH7WceKqgTW7UJ1ia9h10=";Version=1; Path=/Content-Type: application/jsonTransfer-Encoding: chunkedDate: Tue, 25 Feb 2014 09:23:23 GMT</span>
达到相似的功能,Hadoop 命令:
<span style="font-size:18px;">hdfs dfs -mkdir /tmp/webhdfs</span>
创建一个文件
创建一个文件需要两个步骤:第一步是在namenode运行命令,第二步根据第一步提供的location参数执行PUT操作。
<span style="font-size:18px;">curl -i -X PUT"http://hadoop-master:50070/webhdfs/v1/tmp/webhdfs/webhdfs-test.txt?user.name=app&op=CREATE"HTTP/1.1 307 TEMPORARY_REDIRECTCache-Control: no-cacheExpires: Thu, 01-Jan-1970 00:00:00 GMTDate: Wed, 26 Feb 2014 14:35:29 GMTPragma: no-cacheDate: Wed, 26 Feb 2014 14:35:29 GMTPragma: no-cacheSet-Cookie:hadoop.auth="u=app&p=app&t=simple&e=1393461329238&s=as2cO3rRtvj8psr0jAhMk6YBHRY=";Path=/Location:http://machine-2:50075/webhdfs/v1/tmp/webhdfs/webhdfs-test.txt?op=CREATE&user.name=app&namenoderpcaddress=hadoop-master:9000&overwrite=falseContent-Type: application/octet-streamContent-Length: 0Server: Jetty(6.1.26)</span>
发送数据到指定文件中:
<span style="font-size:18px;">curl -i -X PUT -T webhdfs-test.txt"http://machine-2:50075/webhdfs/v1/tmp/webhdfs/webhdfs-test.txt?op=CREATE&user.name=app&namenoderpcaddress=hadoop-master:9000&overwrite=false"HTTP/1.1 100 Continue HTTP/1.1 201 CreatedCache-Control: no-cacheExpires: Wed, 26 Feb 2014 14:37:51 GMTDate: Wed, 26 Feb 2014 14:37:51 GMTPragma: no-cacheExpires: Wed, 26 Feb 2014 14:37:51 GMTDate: Wed, 26 Feb 2014 14:37:51 GMTPragma: no-cacheLocation:webhdfs://0.0.0.0:50070/tmp/webhdfs/webhdfs-test.txtContent-Type: application/octet-streamContent-Length: 0Server: Jetty(6.1.26)</span>
5.2 读取文件
读取/input文件信息,使用-L参数。
<span style="font-size:18px;">[app@hadoop-master ~]$ curl -i -L"http://hadoop-master:14000/webhdfs/v1/input?op=OPEN&user.name=app"HTTP/1.1 200 OKServer: Apache-Coyote/1.1Set-Cookie:hadoop.auth="u=app&p=app&t=simple&e=1393420279602&s=dSnnx9oOwgGwVV/Q/ZmFyRjbtFU=";Version=1; Path=/Content-Type: application/octet-streamContent-Length: 1358Date: Wed, 26 Feb 2014 03:11:19 GMTApache HBase [1] is an open-source,distributed, versioned, column-orientedstore modeled after Google' Bigtable: ADistributed Storage System forStructured Data by Chang et al.[2] Just as Bigtable leverages the distributeddata storage provided by the Google FileSystem, HBase provides Bigtable-likecapabilities on top of Apache Hadoop[3]. To get started using HBase, the fulldocumentation for this release can befound under the doc/ directory thataccompanies this README. Using abrowser,open the docs/index.html to view theproject home page (or browse to [1]).The hbase 'book' at docs/book.html has a'quick start' section and is where youshould being your exploration of thehbase project. The latest HBase can be downloaded froman Apache Mirror [4]. The source code can be found at [5] The HBase issue tracker is at [6] Apache HBase is made available under theApache License, version 2.0 [7] The HBase mailing lists and archives arelisted here [8]. 1. http://hbase.apache.org2.http://labs.google.com/papers/bigtable.html3. http://hadoop.apache.org4.http://www.apache.org/dyn/closer.cgi/hbase/5.http://hbase.apache.org/docs/current/source-repository.html6.http://hbase.apache.org/docs/current/issue-tracking.html7.http://hbase.apache.org/docs/current/license.html8. http://hbase.apache.org/docs/current/mail-lists.html</span>
5.3 重命名目录
需要修改op的值和添加destination参数,实例如下所示:
<span style="font-size:18px;">[app@hadoop-master~]$ curl -i -X PUT"http://hadoop-master:14000/webhdfs/v1/tmp/webhdfs?op=RENAME&user.name=app&destination=/tmp/webhdfs-new"[app@hadoop-master~]$ curl -i -X PUT"http://hadoop-master:14000/webhdfs/v1/tmp/webhdfs?op=RENAME&user.name=app&destination=/tmp/webhdfs-new"HTTP/1.1 200 OKServer:Apache-Coyote/1.1Set-Cookie: hadoop.auth="u=app&p=app&t=simple&e=1393420506278&s=Oz9HEzxuYvP8kfAY4SWH6h+Gb50=";Version=1; Path=/Content-Type:application/jsonTransfer-Encoding:chunkedDate: Wed, 26 Feb2014 03:15:06 GMT {"boolean":true}</span>
验证是否正确地执行,输入下列命令:
<span style="font-size:18px;">[app@hadoop-master~]$ hdfs dfs -ls /tmpFound 5 itemsdrwx------ - app supergroup 0 2014-01-08 08:02 /tmp/hadoop-yarndrwxr-xr-x - app supergroup 0 2014-02-02 09:43 /tmp/hdfs_outdrwxr-xr-x - app supergroup 0 2014-02-20 22:39 /tmp/hive-appdrwxr-xr-x - app supergroup 0 2014-02-25 04:25 /tmp/jpsdrwxr-xr-x - app supergroup 0 2014-02-25 04:23 /tmp/webhdfs-new</span>
5.4 删除目录
非空的目录删除的话会抛出异常,只有为空的目录才会被删除。
<span style="font-size:18px;">[app@hadoop-master ~]$ curl -i -X DELETE"http://hadoop-master:14000/webhdfs/v1/tmp/webhdfs-new?op=DELETE&user.name=app"HTTP/1.1 200 OKServer: Apache-Coyote/1.1Set-Cookie:hadoop.auth="u=app&p=app&t=simple&e=1393421161052&s=r03LOm2hO91ujcc66wNWyMJnDx4=";Version=1; Path=/Content-Type: application/jsonTransfer-Encoding: chunkedDate: Wed, 26 Feb 2014 03:26:01 GMT {"boolean":true}</span>
核查是否正确地执行
<span style="font-size:18px;">[app@hadoop-master ~]$ hdfs dfs -ls /tmpFound 4 itemsdrwx------ -app supergroup 0 2014-01-0808:02 /tmp/hadoop-yarndrwxr-xr-x -app supergroup 0 2014-02-0209:43 /tmp/hdfs_outdrwxr-xr-x -app supergroup 0 2014-02-2022:39 /tmp/hive-appdrwxr-xr-x -app supergroup 0 2014-02-2504:25 /tmp/jps</span>
总结
WebDFS提供了一个简单、标准的方式来执行Hadoop 文件系统操作,这个客户端不必运行在Hadoop集群本身中。WebHDFS最主要的特性是让客户端通过预定义端口直接地链接namenode 和 datanode。这样的话,规避了好多HDFS proxy的代理层以及预先配置Tomcat的绑定。WebHDFD API是可互相交换的,这样让客户端不需要去打开防火墙端口。
两种常见的异常:
1. HTTP/1.1 405 HTTP method PUT is not supported by this URL
修改hdfs-site.xml文件的权限属性。
<property><name>dfs.permissions</name><value>false</value></property>
2 {"RemoteException":{"exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException","message":"Permission denied: user=dr.who, access=WRITE, inode=\"/qzhang\":Administrator:supergroup:drwxr-xr-x\n\tat
问题2与问题1是一个问题,解决方案同上。
欢迎加入微信公众号
- Hadoop REST API -- WebHDFS(下)
- hadoop webhdfs rest api
- Hadoop REST API -- WebHDFS(上)
- Hadoop web编程--REST API WebHDFS
- html调用hadoop WebHDFS REST API
- Hadoop 第六课 WebHDFS REST API
- WebHDFS REST API
- WebHDFS REST API
- hadoop -webhdfs api 方法
- 利用CURL命令调用WebHDFS REST API与Kerberos机制
- Hadoop web编程--REST API
- Hadoop webHDFS设置和使用说明
- Hadoop webHDFS设置和使用说明
- Hadoop WEBHDFS简单配置,使用
- Hadoop webHDFS设置和使用说明
- WebHdfs API使用和开放WebHdfs使用后权限控制
- hadoop webhdfs 服务开启及使用
- ubuntu使用PHP访问Hadoop,利用WebHDFS
- 看这些国学大学的学术成就与养生之道
- Win8重装为Win7
- 深入Java对象的比较
- “ORA-12541:TNS:无监听程序”错误
- DirectFB+GTK交叉编译记录
- Hadoop REST API -- WebHDFS(下)
- 《俄狄浦斯王》--人伦悲剧还是命运悲剧?
- SQL查询语句优化
- 神经网络
- PAT 1024. Palindromic Number (25)【精华】
- 第七天
- dojo中的dojox/grid/EnhancedGrid表格报错
- 开通我的博客了
- ubuntu下 vi编辑器使用手册