jupyter软件的安装

启动和配置

jupyter notebook --generate-config
jupyter notebook --no-browser &

配置

• 设置密码
  随便进入一个python客户端，执行下面命令，会生成密码

from notebook.auth import passwdpasswd()

然后保存到 ~/.jupyter/jupyter_notebook_config.py 配置中

其他常用配置

c.NotebookApp.ip = 'xx.xx.xx.xx'c.NotebookApp.port = 8080

[wls81@cnsz046690 bin]$ jupyter notebook –generate-config
Writing default config to: /wls/wls81/.jupyter/jupyter_notebook_config.py

[wls81@cnsz046690 bin]$ ipython
Python 3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:53:06)
Type “copyright”, “credits” or “license” for more information.

IPython 5.1.0 – An enhanced Interactive Python.
? -> Introduction and overview of IPython’s features.
%quickref -> Quick reference.
help -> Python’s own help system.
object? -> Details about ‘object’, use ‘object??’ for extra details.

In [1]: from notebook.auth import passwd

In [2]: passwd()
Enter password:
Verify password:
Out[2]: ‘sha1:484b80baac5b:b2e93a8ccc570b00e7269798bde7e8a5efebb4ae’

$vim ~/.jupyter/jupyter_notebook_config.py

主要修改如下几个参数：
c.NotebookApp.ip=’*’
c.NotebookApp.password = u’sha1:ce…粘贴上一步生成的密文’
c.NotebookApp.open_browser = False
c.NotebookApp.port =8888

在~/.bashrc 中配置，使用$SPARK_HOME/bin/pyspark 来启动jupyter notebook

export PATH=”/wls/wls81/anaconda3/bin:$PATH”
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=”notebook”

基于Scala spark的jupyter notebook

此处使用Apache toree给notebook安装Scala kernel

toree官网下载页,不需要解压，直接使用pip install安装

wget https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0/snapshots/dev1/toree-pip/toree-0.2.0.dev1.tar.gz
pip install toree-0.2.0.dev1.tar.gz

–master yarn-client \
–executor-memory 5G \
–num-executors 5 \
–executor-cores 1

val input=sc.parallelize(List(1,2,3,4))
val result=input.ma(x=>x*x)
println(result.collect().mkString(“,”))

textFile = sc.textFile(“/tmp/roitest.sql”)
textFile.count()

pyspark & > /dev/null 2>&1