Oozie-3.3.2安装配置运行实践
来源:互联网 发布:java集合框架 书籍 编辑:程序博客网 时间:2024/04/29 15:20
Oozie是一个开源的工作流调度系统,它能够管理逻辑复杂的多个Hadoop作业,按照指定的顺序将其协同运行起来。例如,我们可能有这样一个需求,某个业务系统每天产生20G原始数据,我们每天都要对其进行处理,处理步骤如下所示:
- 通过Hadoop先将原始数据同步到HDFS上;
- 借助MapReduce计算框架对原始数据进行转换,生成的数据以分区表的形式存储到多张Hive表中;
- 需要对Hive中多个表的数据进行JOIN处理,得到一个明细数据Hive大表;
- 将明细数据进行复杂的统计分析,得到排序后的报表信息;
- 需要将统计分析得到的结果数据同步到业务系统中,供业务调用使用。
上述过程可以通过工作流系统来编排任务,最终生成一个工作流实例,然后每天定时启动运行这个实例即可。在这种依赖于Hadoop存储和处理能力要求的应用场景下,Oozie可能能够简化任务调度和执行。
这里,我们在CentOS 6.2系统下安装Oozie-3.3.2,需要安装相关的依赖软件包,下面我们一步一步地进行安装,包括安装配置依赖软件包。这里,我们使用MySQL数据库存储Oozie数据,Hadoop使用的是1.2.1版本。
安装Oozie Server
Oozie Server可以为我们提供很多管理Job的便捷功能,比如,通过可视化界面去管理Job的运行状态,同时也支持我构建含有多个复杂Hadoop Job流程,各个Job之间的依赖关系完全可以通过一个工作流配置文件组装起来,然后由Oozie Server其管理执行。
- 安装Maven构建工具
下载安装,执行如下命令:
1
wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.2.1/binaries/apache-maven-3.2.1-bin.
tar
.gz
2
tar
xvzf apache-maven-3.2.1-bin.
tar
.gz
加入环境变量,使变量配置生效:
1
export
MAVEN_HOME=/home/shirdrn/cloud/programs/apache-maven-3.2.1
2
export
PATH=$PATH:$MAVEN_HOME/bin
- 安装MySQL数据库
安装MySQL数据库,执行如下命令:
1
sudo
rpm -e --nodeps mysql
2
yum list |
grep
mysql
3
sudo
yum
install
-y mysql-server mysql mysql-deve
为root用户设置密码:
1
mysqladmin -u root password
'8YOhyo988_Kjo0'
然后可以使用root账号登录MySQL数据库,进行管理:
1
mysql -u root -p
输入密码登录成功。
- 安装配置Tomcat
下载安装Tomcat Web服务器:
1
wget http://apache.dataguru.cn/tomcat/tomcat-7/v7.0.52/bin/apache-tomcat-7.0.52.
tar
.gz
2
tar
xvzf apache-tomcat-7.0.52.
tar
.gz
设置环境变量:
1
export
CATALINA_HOME=/home/shirdrn/cloud/programs/apache-tomcat-7.0.52
2
export
PATH=$PATH:$CATALINA_HOME/bin
如果使用MySQL存储Oozie数据,需要将MySQL的驱动程序拷贝到Tomcat安装目录下,亦即$CATALINA_HOME/lib下面。
- 准备ExtJS工具包
下载ExtJS压缩包:
1
wget http://extjs.com/deploy/ext-2.2.zip
- 安装Oozie
下载安装,执行如下命令:
1
wget http://mirror.bit.edu.cn/apache/oozie/3.3.2/oozie-3.3.2.
tar
.gz
2
tar
xvzf oozie-3.3.2.
tar
.gz
3
cd
oozie-3.3.2
4
bin/mkdistro.sh -DskipTests
构建成后,可以在oozie-3.3.2/distro/target目录下看到构建后的文件,例如我的路径是/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2,内容如下所示:
1
[shirdrn@oozie-server oozie-3.3.2]$
pwd
2
/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2
3
[shirdrn@oozie-server oozie-3.3.2]$
ls
4
bin lib oozie-core oozie-sharelib-3.3.2.
tar
.gz
5
conf libtools oozie-examples.
tar
.gz oozie.war
6
docs.zip oozie-client-3.3.2.
tar
.gz oozie-server release-log.txt
将OOZIE_HOME变量指向该目录,修改~/bashrc文件:
1
export
OOZIE_HOME=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2
2
export
PATH=$PATH:$OOZIE_HOME/bin
将ExtJS工具包拷贝到目录$OOZIE_HOME中:
1
cp
~/cloud/programs/oozie-3.3.2/ext-2.2.zip $OOZIE_HOME/
在上面的目录下创建libext目录,并将hadoop相关的jar库文件拷贝到libext下面,我使用的是Hadoop 1.2.1版本:
1
[shirdrn@oozie-server oozie-3.3.2]$
mkdir
libext
2
[shirdrn@oozie-server oozie-3.3.2]$
cp
~/cloud/programs/hadoop-1.2.1/hadoop-*.jar libext/
3
[shirdrn@oozie-server oozie-3.3.2]$
cp
~/cloud/programs/hadoop-1.2.1/lib/*.jar ./libext/
同时,我们使用了MySQL来存储Oozie的元数据,现在需要将MySQL的驱动程序添加到libext目录下:
1
cp
~/packages/mysql-connector-java-5.1.29/mysql-connector-java-5.1.29/mysql-connector-java-5.1.29-bin.jar libext/
执行下面的命令开始安装:
1
bin/oozie-setup.sh prepare-war
运行结果,示例如下:
01
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
02
03
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/asm-3.2.jar
04
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/aspectjrt-1.6.11.jar
05
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/aspectjtools-1.6.11.jar
06
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-beanutils-1.7.0.jar
07
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-beanutils-core-1.8.0.jar
08
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-cli-1.2.jar
09
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-codec-1.4.jar
10
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-collections-3.2.1.jar
11
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-configuration-1.6.jar
12
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-daemon-1.0.1.jar
13
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-digester-1.8.jar
14
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-el-1.0.jar
15
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-httpclient-3.0.1.jar
16
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-io-2.1.jar
17
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-lang-2.4.jar
18
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-logging-1.1.1.jar
19
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-logging-api-1.0.4.jar
20
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-math-2.1.jar
21
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-net-3.1.jar
22
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/core-3.1.1.jar
23
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-ant-1.2.1.jar
24
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-capacity-scheduler-1.2.1.jar
25
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-client-1.2.1.jar
26
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-core-1.2.1.jar
27
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-examples-1.2.1.jar
28
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-fairscheduler-1.2.1.jar
29
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-minicluster-1.2.1.jar
30
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-test-1.2.1.jar
31
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-thriftfs-1.2.1.jar
32
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-tools-1.2.1.jar
33
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hsqldb-1.8.0.10.jar
34
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jackson-core-asl-1.8.8.jar
35
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jackson-mapper-asl-1.8.8.jar
36
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jasper-compiler-5.5.12.jar
37
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jasper-runtime-5.5.12.jar
38
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jdeb-0.8.jar
39
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jersey-core-1.8.jar
40
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jersey-json-1.8.jar
41
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jersey-server-1.8.jar
42
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jets3t-0.6.1.jar
43
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jetty-6.1.26.jar
44
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jetty-util-6.1.26.jar
45
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jsch-0.1.42.jar
46
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/junit-4.5.jar
47
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/kfs-0.2.2.jar
48
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/log4j-1.2.15.jar
49
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/mockito-all-1.8.5.jar
50
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/mysql-connector-java-5.1.29-bin.jar
51
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/oro-2.0.8.jar
52
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/servlet-api-2.5-20081211.jar
53
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/slf4j-api-1.4.3.jar
54
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/slf4j-log4j12-1.4.3.jar
55
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/xmlenc-0.52.jar
56
57
New Oozie WAR file with added 'ExtJS library, JARs' at /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie.war
58
59
60
INFO: Oozie is ready to be started
这样,上述已经生成了/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie.war文件。
- 配置Oozie
修改conf/oozie-site.xml配置文件,内容如下所示:
01
<
property
>
02
<
name
>oozie.service.JPAService.jdbc.driver</
name
>
03
<
value
>com.mysql.jdbc.Driver</
value
>
04
<
description
>
05
JDBC driver class.
06
</
description
>
07
</
property
>
08
<
property
>
09
<
name
>oozie.service.JPAService.jdbc.url</
name
>
10
<
value
>jdbc:mysql://mysql-server:3306/oozie</
value
>
11
<
description
>
12
JDBC URL.
13
</
description
>
14
</
property
>
15
<
property
>
16
<
name
>oozie.service.JPAService.jdbc.username</
name
>
17
<
value
>shirdrn</
value
>
18
<
description
>
19
DB user name.
20
</
description
>
21
</
property
>
22
<
property
>
23
<
name
>oozie.service.JPAService.jdbc.password</
name
>
24
<
value
>0o21e</
value
>
25
<
description
>
26
DB user password.
27
IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value,
28
if empty Configuration assumes it is NULL.
29
</
description
>
30
</
property
>
默认情况下,Oozie的配置中有个配置项oozie.service.JPAService.create.db.schema,值为false,设置非自动创建数据库,我们保持默认设置,这样可以通过手动创建Oozie数据库,并对其进行权限控制。然后,我们在MySQL数据库中创建数据库,名称为oozie,并进行访问授权:
1
CREATE DATABASE oozie;
2
GRANT ALL ON oozie.* TO
'shirdrn'
@
'oozie-server'
IDENTIFIED BY
'0o21e'
;
3
FLUSH PRIVILEGES;
然后可以执行如下命令,生成Oozie所需要的数据表:
1
bin/ooziedb.sh create -sqlfile oozie.sql -run
查看控制台输出日志,没有报错,并且在当前目录下可以看到,同时也生成了oozie.sql脚本文件。到MySQL数据库中可以看到生成的表,说明上述操作执行成功。
下面可以启动Oozie,使用如下命令:
1
bin/oozied.sh start
启动信息,示例如下所示:
01
Setting OOZIE_HOME: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2
02
Setting OOZIE_CONFIG: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf
03
Sourcing: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-env.sh
04
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
05
Setting OOZIE_CONFIG_FILE: oozie-site.xml
06
Setting OOZIE_DATA: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data
07
Setting OOZIE_LOG: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs
08
Setting OOZIE_LOG4J_FILE: oozie-log4j.properties
09
Setting OOZIE_LOG4J_RELOAD: 10
10
Setting OOZIE_HTTP_HOSTNAME: oozie-server
11
Setting OOZIE_HTTP_PORT: 11000
12
Setting OOZIE_ADMIN_PORT: 11001
13
Setting OOZIE_HTTPS_PORT: 11443
14
Setting OOZIE_BASE_URL: http://oozie-server:11000/oozie
15
Setting CATALINA_BASE: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
16
Setting OOZIE_HTTPS_KEYSTORE_FILE: /home/shirdrn/.keystore
17
Setting OOZIE_HTTPS_KEYSTORE_PASS: password
18
Setting CATALINA_OUT: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/catalina.out
19
Setting CATALINA_PID: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid
20
21
Using CATALINA_OPTS: -Xmx1024m -Dderby.stream.error.file=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/derby.log
22
Adding to CATALINA_OPTS: -Doozie.home.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2 -Doozie.config.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf -Doozie.log.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs -Doozie.data.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=m1 -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://m1:11000/oozie -Doozie.https.keystore.file=/home/shirdrn/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=
23
24
Using CATALINA_BASE: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
25
Using CATALINA_HOME: /home/shirdrn/cloud/programs/apache-tomcat-7.0.52
26
Using CATALINA_TMPDIR: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp
27
Using JRE_HOME: /usr/java/jdk1.7.0_25/
28
Using CLASSPATH: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/bin/tomcat-juli.jar:/home/shirdrn/cloud/programs/apache-tomcat-7.0.52/bin/bootstrap.jar
29
Using CATALINA_PID: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid
从上面日志可以看到,Oozie管理控制台连接为http://oozie-server:11000/oozie,可以看到图形化界面。
- 整合Oozie和Hadoop
我们的Hadoop平台使用的是用户shirdrn,用户组为shirdrn,这里配置Hadoop代理用户也使用该用户,部署Oozie的主机名为oozie-server。修改Hadoop的配置文件core-site.xml,增加如下配置内容:
1
<!-- OOZIE -->
2
<
property
>
3
<
name
>hadoop.proxyuser.shirdrn.hosts</
name
>
4
<
value
>oozie-server</
value
>
5
</
property
>
6
<
property
>
7
<
name
>hadoop.proxyuser.shirdrn.groups</
name
>
8
<
value
>shirdrn</
value
>
9
</
property
>
修改完上述配置后,需要重新启动Hadoop集群才能生效。
安装Oozie Client
我们可以通过在外部的一个Oozie客户端去提交工作流任务,实际上就是一个客户端程序,通过与Oozie Server进行交互,提交任务,并由Oozie Server去调用执行。
我们可以回到前面解压缩Oozie发行包oozie-3.3.2.tar.gz的目录下,通过前面的构建,现在已经可以看到有一个client目录,该目录下就是Oozie的客户端相关文件。含有Oozie客户端脚本的路径,我这里为/home/shirdrn/cloud/programs/oozie-3.3.2/client/target/oozie-client-3.3.2-client/oozie-client-3.3.2。
查看Oozie客户端运行job的命令帮助信息,可以执行如下命令:
1
cd
/home/shirdrn/cloud/programs/oozie-3.3.2/client/target/oozie-client-3.3.2-client/oozie-client-3.3.2
2
bin/oozie help
3
bin/oozie help job
我们可以找到,Oozie发行包中自带的examples,我这里对应的目录是/home/shirdrn/cloud/programs/oozie-3.3.2/examples/target/oozie-examples-3.3.2-examples/examples/apps,我们可以通过运行这些例子来验证安装是否成功。
首先,将Oozie自带的examples上传到HDFS上:
1
bin/hadoop fs -
mkdir
/oozie
2
bin/hadoop fs -copyFromLocal /home/shirdrn/cloud/programs/oozie-3.3.2/examples/target/oozie-examples-3.3.2-examples/examples /user/shirdrn/examples
我们拿examples中的map-reduce来进行验证,修改job.properties文件,配置内容如下所示:
1
nameNode=hdfs://m1:9000
2
jobTracker=m1:19830
3
queueName=default
4
examplesRoot=examples
5
6
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
7
outputDir=map-reduce
我的环境下,Namenode服务端口为hdfs://m1:9000,JobTracker为m1:19830,运行任务,执行如下命令:
1
cd
/home/shirdrn/cloud/programs/oozie-3.3.2/client/target/oozie-client-3.3.2-client/oozie-client-3.3.2
2
bin/oozie job -oozie http://oozie-server:11000/oozie -config /home/shirdrn/cloud/programs/oozie-3.3.2/examples/target/oozie-examples-3.3.2-examples/examples/apps/map-reduce/job.properties -run
可以通过OozieWeb管理控制台查看提交运行的任务,如图所示:
以及,job配置,运行状态等信息,如图所示:
上面命令选项-run
表示直接运行一个job,当然你可以使用其他选项,如-submit
是提交job,-rerun
是重新运行job,-suspend
是挂起job等等,可以查看命令帮助,或参考相关文档。
- Oozie-3.3.2安装配置运行实践
- Oozie-3.3.2安装配置运行实践
- Oozie-3.3.2安装配置运行实践
- Oozie-2-安装、配置、使用
- oozie 安装和运行
- oozie-3.3.2版本安装
- Oozie安装配置手册
- Oozie编译-安装-配置
- Oozie安装配置简介
- 安装和配置 Oozie
- Hue-安装-配置-添加Oozie
- 安装Oozie的配置问题
- Oozie的web界面安装、配置和运行(CDH环境)
- oozie 安装
- oozie 安装
- oozie 安装
- oozie安装
- Oozie安装
- cisco路由器各接口模块代表的含义是什么
- 精通CSS+DIV网页样式与布局--设置表单和表格
- android中如何在等分LinearLayout
- Windows下Node.js环境的搭建并结合WebStorm IDE开发环境
- java 自定义事件
- Oozie-3.3.2安装配置运行实践
- 添加一个快捷图标到网页
- spring-mvc提供httpInvoker服务
- BitBlt
- 电脑台式机清洁小知识
- oracle 中select distinct 和order by 的问题
- win7系统下Mozilla Firefox无法启动解决办法
- java访问权限控制
- Qt Model/View 学习笔记 (七)