Ubuntu Linux64 安装配置Spark1.6.1
来源:互联网 发布:linux中cd的用法 编辑:程序博客网 时间:2024/06/16 06:15
Ubuntu Linux64 安装配置Spark1.6.1
ubuntuu机器:
-> uname -a
Linux xxx 4.1.5-x86_64 #7 SMPMon Aug 24 13:46:31 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
-> getconf LONG_BIT
64
1、安装jvm
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
jdk-8u77-linux-x64.tar.gz
-> tar -zxf jdk-8u77-linux-x64.gz
cd jdk1.8.0_77
配置:
-> vi ~/.bashrc
export JAVA_HOME=xxx/jdk1.8.0_77
exportJRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
-> source ~/.bashrc
检查jvm安装是否正常
-> java -version
2、安装scala
http://www.scala-lang.org/
-> tar -zxf scala-2.12.0-M3.tgz
-> vi ~/.bashrc
export SCALA_HOME=xxx/scala-2.12.0-M3/bin
export PATH=$JAVA_HOME/bin:$SCALA_HOME:$HADOOP_HOME/bin:$SPARK_HOME/bin:$PATH
-> source ~/.bashrc
测试是否安装成功:
-> scala
Welcome to Scala 2.12.0-M3(Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77).
Type in expressions forevaluation. Or try :help.
scala>
3、安装hadoop
hadoop-2.6.0.tar.gz:
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.0/
-> tar -zxf hadoop-2.6.0.tar.gz
-> vi ~/.bashrc
export SCALA_HOME=xxx/scala-2.12.0-M3/bin
export PATH=$JAVA_HOME/bin:$SCALA_HOME:$HADOOP_HOME/bin:$SPARK_HOME/bin:$PATH
-> source ~/.bashrc
export HADOOP_HOME=xx/hadoop-2.6.0
exportLD_LIBRARY_PATH=$HADOOP_HOME/lib
exportPATH=$JAVA_HOME/bin:$SCALA_HOME:$HADOOP_HOME/bin:$SPARK_HOME/bin:$PATH
-> source ~/.bashrc
4、安装spark
http://spark.apache.org/downloads.html
-> tar -zxf spark-1.6.1-bin-hadoop2.6.tgz
-> source ~/.bashrc
export SPARK_HOME=xx/spark/spark-1.6.1-bin-hadoop2.6
PATH=$JAVA_HOME/bin:$SCALA_HOME:$HADOOP_HOME/bin:$SPARK_HOME/bin:$PATH
-> source ~/.bashrc
-> cd spark-1.6.1-bin-hadoop2.6
-> tree -L 2 //查看2级目录树
-> ./bin/ pyspark
★出现错误: "unable to load native-hadoop library for your platform"
解决:更新hadoop-native库:
http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-2.6.0.tar
-> tar -xhadoop-native-64-2.4.0.tar -C xx/hadoop/lib
★出现错误:"java.net.UnknownHostException: xx: xx: unknown erro"
解决:hosts配置错误
-> vi /etc/hosts
增加如:
127.0.0.1 xx
5、使用scala测试spark
-> bin/spark-shell
- Ubuntu Linux64 安装配置Spark1.6.1
- ubuntu安装spark1.6
- spark1.6.1及scala2.11.8安装配置
- ubuntu linux64 bashrc文件 环境配置
- 最新版spark1.1.0集群安装配置
- Spark1.5.2 on Hadoop2.4.0 安装配置
- Spark1.4.1单机版安装配置
- Spark1.3.0安装配置及WordCount示例
- java JDK安装 linux64
- linux64下安装swftools
- Ubuntu kylin 14.04下的spark1.0.1安装
- CentOS And Ubuntu linux64位服务器 安装JDK gzip:stdin not in gzip format
- Spark1.6.0 on Hadoop-2.6.3 安装配置
- Spark1.6.0 on Hadoop-2.6.3 安装配置
- HaDoop2.6下安装并配置Spark1.6.3
- InstantClient linux64位安装指南
- linux64 CentOS下安装 FMS5
- linux64下安装Go语言
- Android权限监控拦截动态实现
- 求平均数的方法以及数的移位
- C++书籍
- 20160325servlet学习笔记表单中各种类型的信息传递与获取 请求转发
- 各种开发环境代码对齐方式
- Ubuntu Linux64 安装配置Spark1.6.1
- android Fragment与Activity交互,互相发数据(附图详解)
- 把bitmap保存到手机相册
- IOS删除沙盒所有同类型的文件
- 你真的会写单例模式吗——Java实现
- 如何为Swift进行宏定义
- 如何根据树的两个遍历顺序确定另外一个遍历顺序
- git commit 撤销
- [LeetCode]Binary Tree Paths