Mac配置单节点Spark环境

Last login: Thu Apr 28 16:15:47 from ::1

1、下载spark并解压到指定路径下

spirit:opt spirit$ cd /opt

spirit:opt spirit$ cd spark/
spirit:spark spirit$ ls
CHANGES.txt NOTICEREADME.md bindata exampleslicenses sbin
LICENSE R RELEASE conf ec2 lib python

2、配置~/.bash_profile,主要添加$SCALA_HOME和$SPARK_HOME导出，并将两者的bin子目录添加到$PATH中

spirit:spark spirit$ echo $PATH

/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/go/bin:/opt/maven/bin:/opt/ant/bin:/opt/brew/bin:/opt/spark/bin:/usr/local/Cellar/scala/2.11.7/bin

3、配置spark-env文件，添加导出$SCALA_HOME, $SPARK_MASTER_IP,$SPARK_WORKER_MEMORY三个变量

spirit:spark spirit$ cd conf/
spirit:conf spirit$ cp spark-env.sh.template spark-env.sh          

spirit:conf spirit$ vim spark-env.sh

4、配置ssh无秘钥登陆

spirit:conf spirit$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/spirit/.ssh/id_rsa): 
/Users/spirit/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Your identification has been saved in /Users/spirit/.ssh/id_rsa.
Your public key has been saved in /Users/spirit/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:6ka9M7CaxinbxlrZ+3omdjauV1ve4t9ROld/2T0QkeY spirit@spirit
The key's randomart image is:
+---[RSA 2048]----+
|             ..  |
|             o.  |
|            o.   |
|             E.  |
|       .S    .  o|
|     oo.. . . ..B|
|   ooooo o + .o+*|
|  .o*o=.@ . o .++|
|  o*++=%o+ ..o. .|
+----[SHA256]-----+

spirit:conf spirit$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

5、ssh登陆

spirit:conf spirit$ ssh localhost

Last login: Thu Apr 28 16:44:16 2016

6、通过$SPARK_HOME/sbin/start-all.sh启动spark运行环境，并利用jps检查进程

spirit:~ spirit$ cd /opt/spark/
spirit:spark spirit$ ./sbin/start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /opt/spark/logs/spark-spirit-org.apache.spark.deploy.master.Master-1-spirit.out
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-spirit-org.apache.spark.deploy.worker.Worker-1-spirit.out
spirit:spark spirit$ jps
2170 Jps
2125 Master
2157 Worker

7、$SPARK_HOME/bin/spark-shell.sh 登陆集群

spirit:spark spirit$ ./bin/spark-shell 

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.1
      /_/


Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_20)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
16/04/28 16:50:40 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/04/28 16:50:41 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/04/28 16:50:45 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/04/28 16:50:45 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/04/28 16:51:16 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/04/28 16:51:17 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/04/28 16:51:20 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/04/28 16:51:20 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
SQL context available as sqlContext.

scala> 

8、运行测试进行验证

scala> sc

res2: org.apache.spark.SparkContext = org.apache.spark.SparkContext@56881196