Azkaban 2.5.0 job type 插件安装
来源:互联网 发布:知乎 国开博裕和博裕 编辑:程序博客网 时间:2024/05/18 06:20
一、环境及软件
安装环境:
- 安装目录: /usr/local/ae/ankaban
- Hadoop 安装目录 export HADOOP_HOME=/usr/local/ae/hadoop-1.2.1
- azkaban-executor-2.5.0安装目录:/usr/local/ae/azkaban/azkaban-executor-2.5.0
安装软件:
- azkaban-jobtype-2.5.0.tar.gz
Azkaban jobtype plugin source:github.com/azkaban/azkaban-plugins/tree/master/plugins/jobtype
二、配置jobtype
- 解压 azkaban-jobtype-2.5.0.tar.gz 到 $AZKABAN-EXECUTOR_HOME/plugins/jobtypes 目录下
user@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.0/plugins/jobtypes$ tar -zxvx azkaban--jobtype-2.5.0.tar.gz
- jobtype properties 配置
common.properties## everything that the user job can knowhadoop.home=/usr/local/ae/hadoop-1.2.1#hive.home=#pig.home=azkaban.should.proxy=falsejobtype.global.classpath=${hadoop.home}/hadoop-core-1.2.1.jar,${hadoop.home}/*,${hadoop.home}/conf/,${hadoop.home}/lib/*
commonprivate.properties
## hadoop security manager setting common to all hadoop jobshadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_1_0## hadoop security related settings# proxy.keytab.location=YOUR_AZKABAN_KERBEROS_PRINCIPAL# proxy.user=KEYTAB_LOCATIONazkaban.should.proxy=falseobtain.binary.token=false# obtain.namenode.token=true# obtain.jobtracker.token=true# global classpath items for all jobs. e.g. hadoop-core jar, hadoop conf#jobtype.global.classpath=${hadoop.home}/*,${hadoop.home}/conf# global jvm args for all jobs. e.g. java.io.temp.dir, java.library.path#jobtype.global.jvm.args=# hadoophadoop.home=/usr/local/ae/hadoop-1.2.1#pig.home=#hive.home=
三、配置 jobtype 插件安装目录到 Azkaban-executor
- 配置 azkaban.properties
在 azkaban.properties 文件中添加以下配置# Azkaban JobTypes Pluginsazkaban.jobtype.plugin.dir=plugins/jobtypes
四、重启 Azkaban-executor
user@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.0$ sh bin/azkaban-executor-shutdown.shuser@ae01:/usr/local/ae/azkaban/azkaban-executor-2.5.0$ sh bin/azkaban-executor-start.sh
五、HadoopJava jobtype profile
type=hadoopJava#jvm.argsjob.class=×××××××××××××××××××××****main.args=×××××××××method.run=run#method.cancelclasspath=./lib/*,/usr/local/ae/hadoop-1.2.1/*,/usr/local/ae/hadoop-1.2.1/lib/*,/usr/local/ae/hbase-0.94.8/*,/usr/local/ae/hbase-0.94.8/lib/*#dependencies#getJobGeneratedProperties
Note:1. 若 ${job.class} 继承自 AbstractHadoopJob 则需要设置 FileIutPutFormat 与 FileOutPutFormat
2. Azkaban 在启动是会生成两个日志文件azkaban-access.log/azkaban-webserver.log,他们的生成位置是在你执行脚本的目录,所以建议你最好还是在AZKABAN_HOME 目录下执行启动脚本,如果你喜欢在 ./bin 目录下启动,你需要将上文第3.1步骤的红色标记处修改目录位置为 ../${dir}
转自:http://www.cnblogs.com/tannerBG/p/3844833.html
0 0
- Azkaban 2.5.0 job type 插件安装
- Azkaban-2.5.0的安装和配置
- azkaban 2.5 安装详解
- hadoop学习之azkaban(5):azkaban-2.5.0的详细安装过程
- Azkaban进行job调度
- Azkaban 2.5.0 搭建
- centos7(vm)下安装Azkaban 2.5.0(x86)
- Azkaban安装
- Azkaban安装
- azkaban安装
- Hadoop Family Job Schedule----Azkaban
- Hadoop Family Job Schedule----Azkaban
- Azkaban-2.5及Plugins的安装配置
- [解决]azkaban执行错误 Job failed, Cannot request memory (Xms 0 kb, Xmx 0 kb) from system for job
- azkaban 3.2 Job failed, Cannot request memory (Xms 0 kb, Xmx 0 kb) from system for job
- Azkaban安装与应用
- azkaban 3.0 编译安装
- azkaban 的安装
- 把linux文件夹压缩成tar.gz的命令
- 关于Android中接收不到组播信息
- Mac上使用mamp安装配置joomla3
- 解决sublime package control 出现There are no packages available for installation
- 如何使用fdisk命令进行分区
- Azkaban 2.5.0 job type 插件安装
- The requested resource (/) is not available
- Swift3 String.Index 字符串截取
- 编译作业——LR文法的语法分析器
- CSAPP第十二章读书笔记
- spring jdbc
- log4j 写入DB 是不存在单引号的问题的!大家不要被不知所以然的家伙骗!
- Android开发 四大组件是否运行在主线程中?
- memcpy和memmove的区别