Tachyon 0.7.1伪分布式集群安装与测试
来源:互联网 发布:网络高清图片大全 编辑:程序博客网 时间:2024/06/05 06:18
Tachyon是一个高容错的分布式文件系统,允许文件以内存的速度在集群框架中进行可靠的共享,就像Spark和 MapReduce那样。通过利用信息继承,内存侵入,Tachyon获得了高性能。Tachyon工作集文件缓存在内存中,并且让不同的 Jobs/Queries以及框架都能内存的速度来访问缓存文件。因此,Tachyon可以减少那些需要经常使用的数据集通过访问磁盘来获得的次数。
源码下载
源码地址:https://github.com/amplab/tachyon
git clone https://github.com/amplab/tachyon.git
编译
进入 TACHYON_HOME
git tagv0.1.0v0.2.0v0.2.1v0.3.0v0.4.0v0.4.1v0.4.1-thriftv0.5.0v0.6.0v0.6.1v0.6.2v0.6.3v0.6.4v0.7.0v0.7.0-rc1v0.7.1
切换到v0.7.1版本
git checkout v0.7.1 Note: checking out 'v0.7.1'.You are in 'detached HEAD' state. You can look around, make experimentalchanges and commit them, and you can discard any commits you make in thisstate without impacting any branches by performing another checkout.If you want to create a new branch to retain commits you create, you maydo so (now or later) by using -b with the checkout command again. Example: git checkout -b new_branch_nameHEAD 目前位于 a2b69c9... [maven-release-plugin] prepare release v0.7.1
执行如下命令:
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"mvn clean package -Dhadoop.version=2.3.0-cdh5.1.0 -Dspark.version=1.4.1 -DskipTests
配置
- 创建journal目录
mkdir -p /home/cluster/apps/tachyon/journal - 创建ramdisk
mkdir -p /home/cluster/apps/tachyon/ramdisk - 将tachyon_home/conf/tachyon-env.sh.template 修改为tachyon_home/conf/tachyon-env.sh
mv conf/tachyon-env.sh.template conf/tachyon-env.sh - 编辑tachyon_home/conf/tachyon-env.sh 增加如下内容
export JAVA_HOME=/home/cluster/share/java1.7export HADOOP_HOME=/home/cluster/apps/hadoopexport TACHYON_HOME=/home/cluster/apps/tachyonexport TACHYON_RAM_FOLDER=$TACHYON_HOME/ramdiskexport TACHYON_MASTER_ADDRESS=master
- 修改tachyon日志存放路径:
tachyon_home/下创建logs目录
mkdir logs
vim conf/log4j.properties 增加如下内容:
tachyon.logs.dir=/home/cluster/apps/tachyon/logs - 将tachyon_home/conf/core-site.xml.template 修改为tachyon_home/conf/core-site.xml
mv conf/core-site.xml.template conf/core-site.xml
编辑 conf/core-site.xml,增加如下内容:
<configuration>
<property>
<name>fs.tachyon.impl</name>
<value>tachyon.hadoop.TFS</value>
</property>
</configuration> - 格式化Tachyon
bin/tachyon formatConnecting to localhost as wangyue...Formatting Tachyon Worker @ wangyue-umConnection to localhost closed.Formatting Tachyon Master @ master
启动
bin/tachyon-start.sh localKilled 0 processes on wangyue-umKilled 0 processes on wangyue-umConnecting to localhost as wangyue...Killed 0 processes on wangyue-umConnection to localhost closed.[sudo] password for wangyue: Formatting RamFS: /home/cluster/apps/tachyon/ramdisk (1gb)Starting master @ masterStarting worker @ wangyue-umjps下:21025 ZeppelinServer22214 Worker21987 Master21360 TachyonWorker32212 RemoteInterpreterServer23361 NameNode29603 RemoteInterpreterServer21324 TachyonMaster23493 DataNode21412 Jps27005 RemoteInterpreterServer已经启动TachyonMaster & TachyonWorker
tachyon-start.sh local命令将同时在本地启动Master和Worker进程。需要注意的是,运行tachyon-start.sh local命令一定要拥有切换root的密码,否者会无法启动。这是因为RamFS的格式化需要root权限。
测试
webui:http://master:19999
/home/cluster/apps/tachyon$ bin/tachyon runTest Basic CACHE_THROUGH/default_tests_files/BasicFile_CACHE_THROUGH has been removed2015-09-09 20:00:17,911 INFO (MasterClient.java:connect) - Tachyon client (version 0.7.1) is trying to connect with master @ master/127.0.0.1:199982015-09-09 20:00:17,938 INFO (MasterClient.java:connect) - User registered with the master @ master/127.0.0.1:19998; got UserId 52015-09-09 20:00:17,964 INFO (CommonUtils.java:printTimeTakenMs) - createFile with fileId 3 took 57 ms.2015-09-09 20:00:17,999 INFO (WorkerClient.java:connect) - Trying to get local worker host : wangyue-um.local2015-09-09 20:00:18,013 INFO (WorkerClient.java:connect) - Connecting local worker @ wangyue-um.local/10.32.24.78:299982015-09-09 20:00:18,098 INFO (BlockOutStream.java:get) - Writing with local stream. tachyonFile: /default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, opType: CACHE_THROUGH2015-09-09 20:00:18,136 INFO (CommonUtils.java:createBlockPath) - Folder /home/cluster/apps/tachyon/ramdisk/tachyonworker/5 was created!2015-09-09 20:00:18,140 INFO (LocalBlockOutStream.java:<init>) - /home/cluster/apps/tachyon/ramdisk/tachyonworker/5/3221225472 was created! tachyonFile: /default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, blockId: 3221225472, blockCapacityByte: 5368709122015-09-09 20:00:18,203 INFO (CommonUtils.java:printTimeTakenMs) - writeFile to file /default_tests_files/BasicFile_CACHE_THROUGH took 238 ms.2015-09-09 20:00:18,246 INFO (CommonUtils.java:printTimeTakenMs) - readFile file /default_tests_files/BasicFile_CACHE_THROUGH took 43 ms.Passed the test!执行 bin/tachyon runTests 循环测试
停止
wangyue@wangyue-um:/home/cluster/apps/tachyon$ bin/tachyon-stop.sh Killed 1 processes on wangyue-umKilled 1 processes on wangyue-umConnecting to localhost as wangyue...Killed 0 processes on wangyue-umConnection to localhost closed.执行jps:21025 ZeppelinServer22214 Worker21987 Master32212 RemoteInterpreterServer23361 NameNode29603 RemoteInterpreterServer27469 Jps23493 DataNode27005 RemoteInterpreterServer没有tachyon相关进程
尊重原创,拒绝转载,http://blog.csdn.net/stark_summer/article/details/48321605
0 0
- Tachyon 0.7.1伪分布式集群安装与测试
- spark伪分布式安装与测试
- HBase伪分布式集群安装及代码测试
- zookeeper伪分布式集群安装
- ZooKeeper伪分布式集群安装
- zookeeper伪分布式集群安装
- ZooKeeper伪分布式集群安装
- ZooKeeper伪分布式集群安装
- Hadoop伪分布式与集群式安装配置
- Hadoop伪分布式与集群式安装配置
- Hadoop伪分布式与集群式安装配置
- hadoop伪分布式集群搭建与安装(ubuntu系统)
- 4. Tachyon安装-Tachyon集群模式搭建
- zookeeper安装和伪分布式集群安装
- hadoop2.2.0集群测试搭建(伪分布式集群测试)
- Alluxio(tachyon)集群安装部署
- ZooKeeper伪分布式集群安装及使用
- ZooKeeper伪分布式集群安装及使用
- spring MethodInterceptor 与 反射
- 深拷贝,浅拷贝
- JackJson反序列化为何一定需要TypeReference
- httpclient超时设置
- 详解温度传感器tmp275驱动
- Tachyon 0.7.1伪分布式集群安装与测试
- 关于算法的知识要点笔记
- HDU-3081 Marriage Match II
- Google反编译新工具——Enjarify
- eth0: ERROR while getting interface flags: No such device
- 网络编程
- UVA - 10319 Manhattan
- 单链表例题 多项式加法乘法
- 线程进程