Spark-yarn环境部署

    基本环境: ubuntu 12.04 32位

一. 安装yarn伪分布式集群

     1. 创建新用户

         (1) 添加用户: sudo useradd -m hadoop -s /bin/bash         (2) 修改密码: sudo passwd hadoop         (3) 添加sudo权限: sudo adduser hadoop sudo         (4) 注销选择hadoop用户登录

     2. 系统环境配置

         (1) sudo apt-get update         (2) 安装vim: sudo apt-get install vim         (3) 安装ssh: sudo apt-get install openssh-server

(4) 设置ssh免登录cd ~/.sshrm ./id_rsa*ssh-keygen -t rsa  (一路回车即可)cat ./id_rsa.pub >> ./authorized_keys

     3. 安装java环境

         (1) 安装jre和jdk: sudo apt-get install openjdk-7-jre openjdk-7-jdk         (2) 设置环境变量JAVA_HOME: dpkg -L openjdk-7-jdk | grep '/bin/javac'              该命令会输出一个路径，除去路径末尾的 “/bin/javac”         (3) vim ~/.bashrc              添加一行export JAVA_HOME=...         (4) source ~/.bashrc         (5) check java 版本: java -version

     4. 安装hadoop2

         (1) 从http://mirror.bit.edu.cn/apache/hadoop/common/下载最新的稳定版本的hadoop，例如hadoop-2.7.3/hadoop-2.7.3.tar.gz         (2) 安装hadoop:               sudo tar -zxf hadoop-2.7.3.tar.gz -C /usr/local               cd /usr/local               sudo mv hadoop-2.7.3 hadoop               sudo chown -R hadoop hadoop

     5. 配置hadoop

       cd /usr/local/hadoop/etc/hadoop进入hadoop配置目录，如果没有hadoop-env.sh或yarn-env.sh需要从后缀名为hadoop-env.sh.template复制一份       1). 在hadoop-env.sh中配置JAVA_HOME       2).在yarn-env.sh中配置JAVA_HOME

3). 修改core-site.xml<configuration>        <property>             <name>hadoop.tmp.dir</name>             <value>file:/usr/local/hadoop/tmp</value>             <description>Abase for other temporary directories.</description>        </property>        <property>             <name>fs.defaultFS</name>             <value>hdfs://localhost:9000</value>        </property></configuration>

4). hdfs-site.xml<configuration>        <property>             <name>dfs.replication</name>             <value>1</value>        </property>        <property>             <name>dfs.namenode.name.dir</name>             <value>file:/usr/local/hadoop/tmp/dfs/name</value>        </property>        <property>             <name>dfs.datanode.data.dir</name>             <value>file:/usr/local/hadoop/tmp/dfs/data</value>        </property></configuration>

5). mapred-site.xml<configuration>    <property>        <name>mapreduce.framework.name</name>        <value>yarn</value>    </property></configuration>

6). yarn-site.xml<configuration>    <property>        <name>yarn.nodemanager.aux-services</name>        <value>mapreduce_shuffle</value>    </property></configuration>

       6. 启动hadoop

(1) namenode格式化    ./bin/hdfs namenode -format(2) 修改~/.bashrc添加以下两行，并执行source ~/.bashrc    export HADOOP_HOME=/usr/local/hadoop     export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native(3) ./libexec/hadoop-config.sh 找到JAVA_HOME，在前面添加    export JAVA_HOME=...(4) 开启 NameNode 和 DataNode 守护进程    ./sbin/start-dfs.sh(5) jps命令查看是否成功启动    若成功启动则会列出如下进程: “NameNode”、”DataNode” 和 “SecondaryNameNode”（如果 SecondaryNameNode 没有启动，请运行 sbin/stop-dfs.sh 关闭进程，然后再次尝试启动尝试）。如果没有 NameNode 或 DataNode ，那就是配置不成功，请仔细检查之前步骤，或通过查看启动日志排查原因。(6) 启动成功后可以通过 http://localhost:50070/查看nameNode和dataNode相关信息(7) 关闭hadoop    ./sbin/stop-dfs.sh    第二次之后启动 hadoop，无需进行 NameNode 的初始化，只需要运行 ./sbin/start-dfs.sh 即可

      7. 启动YARN

 (1) 启动yarn      ./sbin/start-yarn.sh       ./sbin/mr-jobhistory-daemon.sh start historyserver  #开启历史服务器，才能在Web中查看任务运行情况 (2) jps查看      开启后通过 jps 查看，可以看到多了 NodeManager 和 ResourceManager 两个后台进程 (3) 启动成功后可以通过页面http://localhost:8088/cluster查看集群任务的运行情况 (4) 关闭yarn       ./sbin/stop-yarn.sh        ./sbin/mr-jobhistory-daemon.sh stop historyserver

二. 安装Spark

1. 下载Spark   wget "http://d3kbcqa49mib13.cloudfront.net/spark-2.0.0-bin-hadoop2.7.tgz"2. 解压到/usr/local   sudo tar -xvzf spark-2.0.0-bin-hadoop2.7.tgz -C /usr/local   cd /usr/local   sudo mv spark-2.0.0-bin-hadoop2.7 spark   sudo chown -R hadoop spark3. 设置环境变化PATH   vim ~/.bashrc   export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/spark/bin   source ~/.bashrc

4). 配置Spark cd /usr/local/spark/conf cp spark-env.sh.template spark-env.sh vim spark-env.sh配置内容如下export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386export HADOOP_HOME=/usr/local/hadoop   export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopSPARK_MASTER_IP=masterSPARK_LOCAL_DIRS=/usr/local/sparkSPARK_DRIVER_MEMORY=1G

5). 启动sparksh /usr/local/sbin/start-all.sh启动成功后可以使用pyspark或spark-submit同时也可以访问以下链接查看spark任务http://master:8080

三.  使用Yarn运行Spark程序

     为了方便，以后我们都采用Yarn来运行Spark任务，不会再单独启动Spark

     1.  机器配置

sudo chown hadoop:root -R /usr/local/hadoopsudo chown hadoop:root -R /usr/local/sparksudo chmod 775 -R /usr/local/hadoopsudo chmod 775 -R /usr/local/spark

//bashrc配置export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386export HADOOP_HOME=/usr/local/hadoop   export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/libexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport PATH=$PATH:/usr/local/hadoop/bin:/usr/local/spark/binexport LD_LIBRARY_PATH=/usr/local/hadoop/lib/native:$LD_LIBRARY_PATH

     2. 启动yarn

cd /usr/local/hadoop./sbin/start-dfs.sh./sbin/start-yarn.sh./sbin/mr-jobhistory-daemon.sh start historyserver

     jps查看进程，应该有以下几个

16891 NodeManager16951 JobHistoryServer16502 SecondaryNameNode16028 NameNode17729 Jps16683 ResourceManager16228 DataNode

     3. 停止yarn

cd /usr/local/hadoop./sbin/stop-dfs.sh./sbin/stop-yarn.sh./sbin/mr-jobhistory-daemon.sh stop historyserver

      4. web界面查看

查看nameNode和dataNode: http://localhost:50070/查看yarn集群任务: http://localhost:8088/cluster

四.  问题汇总

     1. 问题        Hadoop 2.x.x - warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard.        解决方案        1. vi ~/.bashrc        2. 添加以下2行           export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native           export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib