Alex 的 Hadoop 菜鸟教程: 第18课 用Http的方式访问HDFS - HttpFs 教程

原文地址： http://blog.csdn.net/nsrainbow/article/details/43678237  最新课程请关注原作者博客，获得更好的显示体验

声明

• 本文基于Centos 6.x + CDH 5.x

HttpFs 有啥用

HttpFs可以干这两件事情

• 通过HttpFs你可以在浏览器里面管理HDFS上的文件
• HttpFs还提供了一套REST 风格的API可以用来管理HDFS

其实很简单的一个东西嘛，但是很实用

安装HttpFs

在集群里面找一台可以访问hdfs的机器安装HttpFs

$ sudo yum install hadoop-httpfs

配置

编辑/etc/hadoop/conf/core-site.xml

<property>  <name>hadoop.proxyuser.httpfs.hosts</name>  <value>*</value>  </property>  <property>  <name>hadoop.proxyuser.httpfs.groups</name>  <value>*</value>  </property>  

这边是定义可以使用httpfs的用户组和host，写*就是不限制

配置好之后重启hadoop

启动HttpFs

$ sudo service hadoop-httpfs start

使用HttpFs

打开浏览器访问 http://host2:14000/webhdfs/v1?op=LISTSTATUS&user.name=httpfs 可以看到

{"FileStatuses": {"FileStatus": [{"pathSuffix": "hbase","type": "DIRECTORY","length": 0,"owner": "hbase","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423446940595,"blockSize": 0,"replication": 0},{"pathSuffix": "tmp","type": "DIRECTORY","length": 0,"owner": "hdfs","group": "hadoop","permission": "1777","accessTime": 0,"modificationTime": 1423122488037,"blockSize": 0,"replication": 0},{"pathSuffix": "user","type": "DIRECTORY","length": 0,"owner": "hdfs","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423529997937,"blockSize": 0,"replication": 0},{"pathSuffix": "var","type": "DIRECTORY","length": 0,"owner": "hdfs","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1422945036465,"blockSize": 0,"replication": 0}]}}

这个 &user.name=httpfs 表示用默认用户 httpfs 访问，默认用户是没有密码的。

webhdfs/v1 这是HttpFs的根目录

访问 http://host2:14000/webhdfs/v1/user?op=LISTSTATUS&user.name=httpfs 可以看到

{"FileStatuses": {"FileStatus": [{"pathSuffix": "cloudera","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423472508868,"blockSize": 0,"replication": 0},{"pathSuffix": "hdfs","type": "DIRECTORY","length": 0,"owner": "hdfs","group": "hadoop","permission": "700","accessTime": 0,"modificationTime": 1422947019504,"blockSize": 0,"replication": 0},{"pathSuffix": "history","type": "DIRECTORY","length": 0,"owner": "mapred","group": "hadoop","permission": "1777","accessTime": 0,"modificationTime": 1422945692887,"blockSize": 0,"replication": 0},{"pathSuffix": "hive","type": "DIRECTORY","length": 0,"owner": "hive","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423123187569,"blockSize": 0,"replication": 0},{"pathSuffix": "hive_people","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423216966453,"blockSize": 0,"replication": 0},{"pathSuffix": "hive_people2","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423222237254,"blockSize": 0,"replication": 0},{"pathSuffix": "impala","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423475272189,"blockSize": 0,"replication": 0},{"pathSuffix": "root","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "700","accessTime": 0,"modificationTime": 1423221719835,"blockSize": 0,"replication": 0},{"pathSuffix": "spark","type": "DIRECTORY","length": 0,"owner": "spark","group": "spark","permission": "755","accessTime": 0,"modificationTime": 1423530243396,"blockSize": 0,"replication": 0},{"pathSuffix": "sqoop","type": "DIRECTORY","length": 0,"owner": "hdfs","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423127462911,"blockSize": 0,"replication": 0},{"pathSuffix": "test_hive","type": "DIRECTORY","length": 0,"owner": "root","group": "hadoop","permission": "755","accessTime": 0,"modificationTime": 1423215687891,"blockSize": 0,"replication": 0}]}}

很奇怪的是HttpFs的文档很少，更具体的命令要去 WebHDFS的文档里面看 WebHDFS REST API  

支持的命令

Operations

• HTTP GET
  • OPEN (see FileSystem.open)
  • GETFILESTATUS (see FileSystem.getFileStatus)
  • LISTSTATUS (see FileSystem.listStatus)
  • GETCONTENTSUMMARY (see FileSystem.getContentSummary)
  • GETFILECHECKSUM (see FileSystem.getFileChecksum)
  • GETHOMEDIRECTORY (see FileSystem.getHomeDirectory)
  • GETDELEGATIONTOKEN (see FileSystem.getDelegationToken)
• HTTP PUT
  • CREATE (see FileSystem.create)
  • MKDIRS (see FileSystem.mkdirs)
  • RENAME (see FileSystem.rename)
  • SETREPLICATION (see FileSystem.setReplication)
  • SETOWNER (see FileSystem.setOwner)
  • SETPERMISSION (see FileSystem.setPermission)
  • SETTIMES (see FileSystem.setTimes)
  • RENEWDELEGATIONTOKEN (see DistributedFileSystem.renewDelegationToken)
  • CANCELDELEGATIONTOKEN (see DistributedFileSystem.cancelDelegationToken)
• HTTP POST
  • APPEND (see FileSystem.append)
• HTTP DELETE
  • DELETE (see FileSystem.delete)

建立文件夹

尝试建立一个叫 abc 的文件夹

[root@host2 hadoop-httpfs]# curl -i -X PUT "http://host2:14000/webhdfs/v1/user/abc?op=MKDIRS&user.name=httpfs"HTTP/1.1 200 OKServer: Apache-Coyote/1.1Set-Cookie: hadoop.auth="u=httpfs&p=httpfs&t=simple&e=1423573951025&s=Ab44ha1Slg1f4xCrK+x4R/s1eMY="; Path=/; Expires=Tue, 10-Feb-2015 13:12:31 GMT; HttpOnlyContent-Type: application/jsonTransfer-Encoding: chunkedDate: Tue, 10 Feb 2015 03:12:36 GMT{"boolean":true}

然后用服务器上的hdfs dfs -ls 命令看下结果

[root@host2 conf]# hdfs dfs -ls /userFound 12 itemsdrwxr-xr-x   - httpfs hadoop          0 2015-02-10 11:12 /user/abcdrwxr-xr-x   - root   hadoop          0 2015-02-09 17:01 /user/clouderadrwx------   - hdfs   hadoop          0 2015-02-03 15:03 /user/hdfsdrwxrwxrwt   - mapred hadoop          0 2015-02-03 14:41 /user/historydrwxr-xr-x   - hive   hadoop          0 2015-02-05 15:59 /user/hivedrwxr-xr-x   - root   hadoop          0 2015-02-06 18:02 /user/hive_peopledrwxr-xr-x   - root   hadoop          0 2015-02-06 19:30 /user/hive_people2drwxr-xr-x   - root   hadoop          0 2015-02-09 17:47 /user/impaladrwx------   - root   hadoop          0 2015-02-06 19:21 /user/rootdrwxr-xr-x   - spark  spark           0 2015-02-10 09:04 /user/sparkdrwxr-xr-x   - hdfs   hadoop          0 2015-02-05 17:11 /user/sqoopdrwxr-xr-x   - root   hadoop          0 2015-02-06 17:41 /user/test_hive

可以看到建立了一个属于 httpfs 的文件夹 abc

打开文件

从后台上传一个文本文件 test.txt  到 /user/abc 目录下，内容是

Hello World!

用httpfs访问

[root@host2 hadoop-httpfs]# curl -i -X GET "http://host2:14000/webhdfs/v1/user/abc/test.txt?op=OPEN&user.name=httpfs"HTTP/1.1 200 OKServer: Apache-Coyote/1.1Set-Cookie: hadoop.auth="u=httpfs&p=httpfs&t=simple&e=1423574166943&s=JTxqIJUsblVBeHVuTs6JCV2UbBs="; Path=/; Expires=Tue, 10-Feb-2015 13:16:06 GMT; HttpOnlyContent-Type: application/octet-streamContent-Length: 13Date: Tue, 10 Feb 2015 03:16:07 GMTHello World!