零基础入门深度学习之-002-徒手搭建Spark

摘要

深度学习是机器学习的一个分支，机器学习的入门神器是Spark

提前准备

Ubuntu 14 (自带 Python 2.7)
py4j-0.10.4.tar.gz
spark-2.1.0-bin-hadoop2.7.tgz
jdk-8u121-linux-x64.tar.gz

各种解压略过

设置环境变量

nano ~/.bashrcexport JAVA_HOME=~/jdk/jdk1.8.0_121export JRE_HOME=${JAVA_HOME}/jreexport CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/libexport PATH=${JAVA_HOME}/bin:$PATHexport SPARK_HOME=~/spark/spark-2.1.0-bin-hadoop2.7export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATHexport PATH=${SPARK_HOME}/bin:$PATH# export PYSPARK_PYTHON=python3

安装 py4j

tar xf py4j-0.10.4.tar.gzcd py4j-0.10.4/sudo python setup.py install

run following code in python shell

from pyspark import SparkConf, SparkContextconf = SparkConf().setMaster("local").setAppName("My App")sc = SparkContext(conf = conf)print (sc)