Alluxio安装(以HDFS作为底层文件系统)

上一篇我们介绍了如何通过源码实现 Alluxio编译 的整个过程，接下来，这篇文章，将带你一起把我们编译好的安装包（或者是从官方直接下载的安装包——hadoop版本号需要跟你的基础环境一致）部署安装在我们的服务器上。

其他更多关于Alluxio的使用和交流，请加QQ群讨论：

Alluxio-China 452894882

一、准备工作
1. 编译好的二进制安装文件包
2. 已安装了JDK_1.7或者以上版本的java运行时环境
3. 运行正常且版本号与我们的Alluxio安装包对应的版本一致的Hadoop基础环境


二、执行配置&安装
1. 先将我们前面编译好的二进制安装包文件上传到服务器上，然后解压缩
[op1@HOST_NAME_01 alluxio]$ ls -l
total 235260
-rw------- 1 op1 op1 240902504 Jan 17 11:59 alluxio-1.4.0.bin.tar.gz
tar -zxvf alluxio-1.4.0.bin.tar.gz 
ln -s alluxio current


2. 创建HDFS数据目录，用于存放Alluxio数据文件
[op1@HOST_NAME_01 current]$ hdfs dfs -mkdir -p /alluxio/data_demo
...
[op1@HOST_NAME_01 current]$ hdfs dfs -ls /alluxio/
...
drwxr-xr-x   - op1 supergroup          0 2017-01-17 12:25 /alluxio/data_demo


3. 初始化配置
[op1@HOST_NAME_01 alluxio]$ cd current
[op1@HOST_NAME_01 current]$ ./bin/alluxio bootstrapConf HOST_NAME_01 hdfs
/home/op1/alluxio/current/bin/../conf/alluxio-env.sh is created.


mv conf/alluxio-site.properties.template conf/alluxio-site.properties
修改conf/alluxio-site.properties文件，将底层存储系统的地址设置为HDFS namenode的地址以及你想挂载到Alluxio根目录下的HDFS目录。
例如，若你的HDFS namenode是在本地默认端口运行，并且HDFS的根目录已经被映射到Alluxio根目录，则该地址为hdfs://HOST_NAME_01:9000；
若只有/alluxio/data这一个HDFS目录被映射到Alluxio根目录，则该地址为hdfs://HOST_NAME_01:9000/alluxio/data_demo。
[op1@HOST_NAME_01 current]$ vi conf/alluxio-site.properties
#
# The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
# (the "License"). You may not use this work except in compliance with the License, which is
# available at www.apache.org/licenses/LICENSE-2.0
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied, as more fully set forth in the License.
#
# See the NOTICE file distributed with this work for information regarding copyright ownership.
#

# Site specific configuration properties for Alluxio
# Details about all configuration properties http://www.alluxio.org/documentation/en/Configuration-Settings.html

# Common properties
# alluxio.master.hostname=localhost
# alluxio.underfs.address=${alluxio.work.dir}/underFSStorage
alluxio.underfs.address=hdfs://HADOOP_MASTER_HOST_NAME:9000/alluxio/data_demo

# Security properties
# alluxio.security.authorization.permission.enabled=true
# alluxio.security.authentication.type=SIMPLE

# Worker properties
# alluxio.worker.memory.size=1GB
# alluxio.worker.tieredstore.levels=1
# alluxio.worker.tieredstore.level0.alias=MEM
# alluxio.worker.tieredstore.level0.dirs.path=/mnt/ramdisk

# User properties
# alluxio.user.file.readtype.default=CACHE_PROMOTE
# alluxio.user.file.writetype.default=MUST_CACHE

三、启动&确认

1.启动

./bin/alluxio format
./bin/alluxio-start.sh all

至此，一个Alluxio master和一个Alluxio worker已经在本地启动完成，可以在浏览器中访问http://host_ip:19999查看Master Web UI,如下图。

2. 测试&验证

接下来，运行一个官方的示例程序

[op1@HOST_NAME_01 current]$ hdfs dfs -ls /alluxio/data_demo

...

[op1@HOST_NAME_01 current]$ ./bin/alluxio runTests

2017-01-17 12:43:49,794 INFO  type (MetricsSystem.java:startSinksFromConfig) - Starting sinks with config: {}.
2017-01-17 12:43:49,885 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.4.0) is trying to connect with FileSystemMasterClient master @ SVR3404HW1288/10.2.26.40:19998
2017-01-17 12:43:49,906 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemMasterClient master @ SVR3404HW1288/10.2.26.40:19998
runTest Basic CACHE_PROMOTE MUST_CACHE
2017-01-17 12:43:50,107 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.4.0) is trying to connect with BlockMasterClient master @ SVR3404HW1288/10.2.26.40:19998
2017-01-17 12:43:50,110 INFO  type (AbstractClient.java:connect) - Client registered with BlockMasterClient master @ SVR3404HW1288/10.2.26.40:19998
2017-01-17 12:43:50,182 INFO  type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.BlockWorkerClientService$Client@109f9bf6
2017-01-17 12:43:50,215 INFO  type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.BlockWorkerClientService$Client@5d50b632
2017-01-17 12:43:50,305 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_MUST_CACHE took 329 ms.
2017-01-17 12:43:50,414 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_MUST_CACHE took 109 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE MUST_CACHE
2017-01-17 12:43:50,435 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_MUST_CACHE took 10 ms.
2017-01-17 12:43:50,449 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_MUST_CACHE took 13 ms.
Passed the test!
runTest Basic CACHE_PROMOTE CACHE_THROUGH
2017-01-17 12:43:50,501 INFO  type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.FileSystemWorkerClientService$Client@3018fc1a
2017-01-17 12:43:50,527 INFO  type (ThriftClientPool.java:createNewResource) - Created a new thrift client alluxio.thrift.FileSystemWorkerClientService$Client@24834237
2017-01-17 12:43:50,845 INFO  type (NettyChannelPool.java:createNewResource) - Created netty channel with netty bootstrap Bootstrap(group: EpollEventLoopGroup, channelFactory: EpollSocketChannel.class, options: {SO_KEEPALIVE=true, TCP_NODELAY=true, ALLOCATOR=PooledByteBufAllocator(directByDefault: true)}, handler: alluxio.client.netty.NettyClient$1@7ce489c6, remoteAddress: svr3404hw1288.hadoop.fat.qa.nt.ctripcorp.com/10.2.26.40:29999).
2017-01-17 12:43:51,199 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_CACHE_THROUGH took 750 ms.
2017-01-17 12:43:51,208 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_CACHE_THROUGH took 9 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE CACHE_THROUGH
2017-01-17 12:43:51,272 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_CACHE_THROUGH took 44 ms.
2017-01-17 12:43:51,281 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_CACHE_THROUGH took 9 ms.
Passed the test!
runTest Basic CACHE_PROMOTE THROUGH
2017-01-17 12:43:51,324 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_THROUGH took 41 ms.
2017-01-17 12:43:51,479 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_THROUGH took 155 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE THROUGH
2017-01-17 12:43:51,535 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_THROUGH took 38 ms.
2017-01-17 12:43:51,583 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_THROUGH took 47 ms.
Passed the test!
runTest Basic CACHE_PROMOTE ASYNC_THROUGH
2017-01-17 12:43:51,619 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_ASYNC_THROUGH took 35 ms.
2017-01-17 12:43:51,630 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_ASYNC_THROUGH took 11 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE ASYNC_THROUGH
2017-01-17 12:43:51,650 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_ASYNC_THROUGH took 11 ms.
2017-01-17 12:43:51,662 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_PROMOTE_ASYNC_THROUGH took 12 ms.
Passed the test!
runTest Basic CACHE MUST_CACHE
2017-01-17 12:43:51,677 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_MUST_CACHE took 14 ms.
2017-01-17 12:43:51,690 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_MUST_CACHE took 13 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE MUST_CACHE
2017-01-17 12:43:51,707 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_MUST_CACHE took 8 ms.
2017-01-17 12:43:51,716 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_MUST_CACHE took 9 ms.
Passed the test!
runTest Basic CACHE CACHE_THROUGH
2017-01-17 12:43:51,773 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_CACHE_THROUGH took 56 ms.
2017-01-17 12:43:51,784 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_CACHE_THROUGH took 11 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE CACHE_THROUGH
2017-01-17 12:43:51,844 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_CACHE_THROUGH took 42 ms.
2017-01-17 12:43:51,856 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_CACHE_THROUGH took 11 ms.
Passed the test!
runTest Basic CACHE THROUGH
2017-01-17 12:43:51,906 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_THROUGH took 49 ms.
2017-01-17 12:43:51,948 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_THROUGH took 42 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE THROUGH
2017-01-17 12:43:52,003 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_THROUGH took 36 ms.
2017-01-17 12:43:52,054 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_THROUGH took 51 ms.
Passed the test!
runTest Basic CACHE ASYNC_THROUGH
2017-01-17 12:43:52,070 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_ASYNC_THROUGH took 14 ms.
2017-01-17 12:43:52,080 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_ASYNC_THROUGH took 10 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE ASYNC_THROUGH
2017-01-17 12:43:52,100 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_CACHE_ASYNC_THROUGH took 12 ms.
2017-01-17 12:43:52,129 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_CACHE_ASYNC_THROUGH took 28 ms.
Passed the test!
runTest Basic NO_CACHE MUST_CACHE
2017-01-17 12:43:52,144 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_MUST_CACHE took 13 ms.
2017-01-17 12:43:52,152 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_MUST_CACHE took 8 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE MUST_CACHE
2017-01-17 12:43:52,167 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_NO_CACHE_MUST_CACHE took 8 ms.
2017-01-17 12:43:52,176 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_NO_CACHE_MUST_CACHE took 8 ms.
Passed the test!
runTest Basic NO_CACHE CACHE_THROUGH
2017-01-17 12:43:52,234 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_CACHE_THROUGH took 56 ms.
2017-01-17 12:43:52,243 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_CACHE_THROUGH took 8 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE CACHE_THROUGH
2017-01-17 12:43:52,303 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_NO_CACHE_CACHE_THROUGH took 44 ms.
2017-01-17 12:43:52,314 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_NO_CACHE_CACHE_THROUGH took 10 ms.
Passed the test!
runTest Basic NO_CACHE THROUGH
2017-01-17 12:43:52,362 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_THROUGH took 47 ms.
2017-01-17 12:43:52,394 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_THROUGH took 32 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE THROUGH
2017-01-17 12:43:52,444 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_NO_CACHE_THROUGH took 34 ms.
2017-01-17 12:43:52,476 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_NO_CACHE_THROUGH took 32 ms.
Passed the test!
runTest Basic NO_CACHE ASYNC_THROUGH
2017-01-17 12:43:52,492 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_ASYNC_THROUGH took 15 ms.
2017-01-17 12:43:52,499 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_ASYNC_THROUGH took 7 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE ASYNC_THROUGH
2017-01-17 12:43:52,517 INFO  type (BasicNonByteBufferOperations.java:write) - writeFile to file /default_tests_files/BasicNonByteBuffer_NO_CACHE_ASYNC_THROUGH took 10 ms.
2017-01-17 12:43:52,526 INFO  type (BasicNonByteBufferOperations.java:read) - readFile file /default_tests_files/BasicNonByteBuffer_NO_CACHE_ASYNC_THROUGH took 9 ms.
Passed the test!

测试程序运行结束后，就能在hdfs的数据目录下看到该测试程序生成是数据目录和文件

[op1@HOST_NAME_01 current]$ hdfs dfs -ls /alluxio/data_demo
...
Found 1 items
drwxr-xr-x   - op1 op1          0 2017-01-17 12:43 /alluxio/data_demo/default_tests_files

在Master的WebUI界面上也能看到上述测试程序生成的数据文件及目录

四、停止Alluxio

./bin/alluxio-stop.sh all