azkaban3.38.0 mac安装

1.下载 azkaban3.38.0压缩包：https://github.com/azkaban/azkaban/releases/tag/3.38.0

2.解压后文件夹重命名为 azkaban

3.在 azkaban 目录下build

./gradlew clean

./gradlew distTar

4.在 azkaban 文件夹同级目录下新建 azkaban3.38.0文件夹，并分别将 azkaban-db、azkaban-exec-server、azkaban-web-server中，/build/distributions下的 tar.gz 拷贝到 azkaban3.38.0文件夹中，并用 tar -zxvf 命令解压

5.将azkaban-solo-server/build/distributions下的 tar.gz 文件解压，并将temp 和 plugins 文件夹用 cp -r 命令拷贝到 azkaban3.38.0/azkaban-exec-server和azkaban3.38.0/azkaban-web-server文件夹下

*******之后全对 azkaban3.38.0文件夹下文件操作*******

6.1修改 exec 和 web 文件夹下 conf 文件夹中的 azkaban.properties 文件

# database.type=h2
# h2.path=./h2
# h2.create.tables=true
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

6.2两个文件夹都增加 log4j.properties

log4j.rootLogger=INFO,C
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.Target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

7.在 web/conf 下添加keystore

keytool -keystore keystore -alias jetty -genkey -keyalg RSA

然后随意

8.在 exec和 web 路径下新建 extlib 文件夹，把 lib 文件夹下的 mysql jar 包拷贝到 extlib 中

9.mysql 新建用户，配置信息和 azkaban.properties中的配置统一，并赋权限

10.在 azkaban 数据库中，执行 source /path/create-all-sql-0.1.0-SNAPSHOT.sql

11.先执行 exec-start.sh，后开启 web-start.sh，注意都使用 bin/*-start.sh 命令，*是什么都懂的

12.页面打开 localhost:8081

遇到问题：

1.没找到 hbase 的 jar 包？？？

解决：不要 source /etc/profile，或者别写 HBASE_HOME

2.在页面新建任务后，上传 zip 包，失败

原因：azkaban 把上传的 zip 包扔到 mysql 库里了，很蠢

解决：用 /etc/my.cnf 修改 mysql 最大上传包大小，mac 下一般没有这个文件，自己 vim 一个

# Example MySQL config file for medium systems.
  #
  # This is for a system with little memory (32M - 64M) where MySQL plays
  # an important part, or systems up to 128M where MySQL is used together with
  # other programs (such as a web server)
  #
  # MySQL programs look for option files in a set of
  # locations which depend on the deployment platform.
  # You can copy this option file to one of those
  # locations. For information about these locations, see:
  # http://dev.mysql.com/doc/mysql/en/option-files.html
  #
  # In this file, you can use all long options that a program supports.
  # If you want to know which options a program supports, run the program
  # with the "--help" option.
  # The following options will be passed to all MySQL clients
  [client]
  default-character-set=utf8
  #password   = your_password
  port        = 3306
  socket      = /tmp/mysql.sock
  # Here follows entries for some specific programs
  # The MySQL server
  [mysqld]
  character-set-server=utf8
  init_connect='SET NAMES utf8
  port        = 3306
  socket      = /tmp/mysql.sock
  skip-external-locking
  key_buffer_size = 16M
  max_allowed_packet = 999M
  table_open_cache = 64
  sort_buffer_size = 512K
  net_buffer_length = 8K
  read_buffer_size = 256K
  read_rnd_buffer_size = 512K
  myisam_sort_buffer_size = 8M
  character-set-server=utf8
  init_connect='SET NAMES utf8'
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
#skip-networking


  # Replication Master Server (default)
  # binary logging is required for replication
  log-bin=mysql-bin


    # binary logging format - mixed recommended
    binlog_format=mixed


      # required unique id between 1 and 2^32 - 1
      # defaults to 1 if master-host is not set
      # but will not function as a master if omitted
      server-id   = 1


    # Replication Slave (comment out master section to use this)
    #
    # To configure this host as a replication slave, you can choose between
    # two methods :
    #
    # 1) Use the CHANGE MASTER TO command (fully described in our manual) -
    #    the syntax is:
    #
    #    CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
    #    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
    #
    #    where you replace <host>, <user>, <password> by quoted strings and
    #    <port> by the master's port number (3306 by default).
    #
    #    Example:
    #
    #    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
    #    MASTER_USER='joe', MASTER_PASSWORD='secret';
    #
    # OR
    #
    # 2) Set the variables below. However, in case you choose this method, then
    #    start replication for the first time (even unsuccessfully, for example
    #    if you mistyped the password in master-password and the slave fails to
    #    connect), the slave will create a master.info file, and any later
    #    change in this file to the variables' values below will be ignored and
    #    overridden by the content of the master.info file, unless you shutdown
    #    the slave server, delete master.info and restart the slaver server.
    #    For that reason, you may want to leave the lines below untouched
    #    (commented) and instead use CHANGE MASTER TO (see above)
    #
    # required unique id between 2 and 2^32 - 1
    # (and different from the master)
    # defaults to 2 if master-host is set
    # but will not function as a slave if omitted
    #server-id       = 2
    #
    # The replication master for this slave - required
    #master-host     =   <hostname>
    #
    # The username the slave will use for authentication when connecting
    # to the master - required
    #master-user     =   <username>
    #
    # The password the slave will authenticate with when connecting to
    # the master - required
    #master-password =   <password>
    #
    # The port the master is listening on.
    # optional - defaults to 3306
    #master-port     =  <port>
    #
    # binary logging - not required for slaves, but recommended
    #log-bin=mysql-bin


      # Uncomment the following if you are using InnoDB tables
      #innodb_data_home_dir = /usr/local/mysql/data
      #innodb_data_file_path = ibdata1:10M:autoextend
      #innodb_log_group_home_dir = /usr/local/mysql/data
      # You can set .._buffer_pool_size up to 50 - 80 %
      # of RAM but beware of setting memory usage too high
      #innodb_buffer_pool_size = 16M
      #innodb_additional_mem_pool_size = 2M
      # Set .._log_file_size to 25 % of buffer pool size
      #innodb_log_file_size = 5M
      #innodb_log_buffer_size = 8M
      #innodb_flush_log_at_trx_commit = 1
      #innodb_lock_wait_timeout = 50


        [mysqldump]
        quick
        max_allowed_packet = 999M


          [mysql]
          no-auto-rehash
          # Remove the next comment character if you are not familiar with SQL
          #safe-updates
          default-character-set=utf8


        [myisamchk]
        key_buffer_size = 20M
        sort_buffer_size = 20M
        read_buffer = 2M
        write_buffer = 2M


          [mysqlhotcopy]
          interactive-timeout

max_allowed_packet的值自己根据情况改，然后重启 mysql

修改命令：set global max_allowed_packet = 2*1024*1024*100;

查看命令：show VARIABLES like '%max_allowed_packet%';

3.在页面如何用 flow 处理多个任务

解决：写多个 job 文件，然后用 dependencies 属性，指定这个 job 的前提任务，然后将几个.job 文件还有打好的spark jar 包都打到一个 zip 包里，提交到页面上一个 project 里