docker on spark
来源:互联网 发布:一个人根本做不起淘宝 编辑:程序博客网 时间:2024/05/22 03:07
- 从docker 仓库 pull 镜像
docker pull sequenceiq/spark:1.4.0
构建 docker 镜像
docker build –rm -t sequenceiq/spark:1.4.0 .
-t 选项是你要构建的sequenceiq/spark image的tag,就好比ubuntu:13.10一样 –rm 选项是告诉Docker在构建完成后删除临时的Container,Dockerfile的每一行指令都会创建一个临时的Container,一般你是不需要这些临时生成的Container的运行镜像
- if using boot2docker make sure your VM has more than 2GB memory
- in your /etc/hosts file add $(boot2docker ip) as host ‘sandbox’ to make it easier to access your sandbox UI
- open yarn UI ports when running container
docker run -it -p 8088:8088 -p 8042:8042 -h sandbox sequenceiq/spark:1.4.0 bash
or
docker run -d -h sandbox sequenceiq/spark:1.4.0 -d
- 如果要进行交互式操作(例如Shell脚本),那我们必须使用-i -t参数同容器进行数据交互。但是当通过管道同容器进行交互时,就不需要使用-t参数
- -h来设定hostname
如果使用-p或者-P,那么容器会开放部分端口到主机,只要对方可以连接到主机,就可以连接到容器内部。当使用-P时,Docker会在主机中随机从49153 和65535之间查找一个未被占用的端口绑定到容器。你可以使用docker port来查找这个随机绑定端口。
如果在docker run后面追加-d=true或者-d,那么容器将会运行在后台模式。此时所有I/O数据只能通过网络资源或者共享卷组来进行交互。因为容器不再监听你执行docker run的这个终端命令行窗口。但你可以通过执行docker attach来重新附着到该容器的回话中。需要注意的是,容器运行在后台模式下,是不能使用–rm选项的。
-p 8088:8088 这个端口是resourcemanager 或者 集群 ,-p 8042:8042 这个端口是 nodemanager端口
版本
Hadoop 2.6.0 and Apache Spark v1.4.0 on Centos测试
There are two deploy modes that can be used to launch Spark applications on YARN.- YARN-client mode
In yarn-cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application.Estimating Pi (yarn-cluster mode):# execute the the following command which should write the "Pi is roughly 3.1418" into the logs# note you must specify --files argument in cluster mode to enable metricsspark-submit \--class org.apache.spark.examples.SparkPi \--files $SPARK_HOME/conf/metrics.properties \--master yarn-cluster \--driver-memory 1g \--executor-memory 1g \--executor-cores 1 \$SPARK_HOME/lib/spark-examples-1.4.0-hadoop2.6.0.jar
- YARN-cluster mode
# execute the the following command which should print the "Pi is roughly 3.1418" to the screenspark-submit \--class org.apache.spark.examples.SparkPi \--master yarn-client \--driver-memory 1g \--executor-memory 1g \--executor-cores 1 \$SPARK_HOME/lib/spark-examples-1.4.0-hadoop2.6.0.jar
- Spark runs on Docker
- docker on spark
- Spark视频第14期:Spark亚太研究院决胜大数据时代公益大讲坛:Spark on Docker深入揭秘
- docker部署spark集群
- 基于docker安装spark
- docker 构建spark集群
- spark-03-spark on yarn
- Running docker on Android
- Hadoop On Docker 实践
- Setup docker on OSX
- Hadoop on Docker
- Install Docker on Mac
- YARN on Docker
- hive on spark on yarn
- 【Docker】Docker运行单机版Spark
- KMeans on Spark
- run spark on local
- Spark on Yarn部署
- java DES加密,解密 (一)
- POJ2752Seek the Name, Seek the Fame (KMP)
- 介绍几款网页数据抓取软件
- @昵称符号截取(最简单的字符串截取)二
- qt error: C2001: 常量中有换行符
- docker on spark
- [iOS]Xcode no visible @interface for XXX declares…
- 繞過linux Driver Vermagic檢查
- http://blog.csdn.net/lpdx111/article/details/7638147
- libopencv_shape.so.3.0: cannot open shared object file: No such file or directory 解决笔记
- 2015 Multi-University Training Contest 6
- ConcurrentHashMap代码片段
- SSL VPN
- 事务特性