Apache Eagle安装配置指南

Apache Eagle开源没多久，网上关于安装它的文章几乎没有，在此我献上自己的安装步骤供大家参考。

官网安装文档地址：http://goeagle.io/docs/deployment-in-production.html

官网的安装大体步骤都没问题，只是在配置的时候需要注意几点。

1.环境依赖

HBase: 0.98 or later

Storm: 0.9.3 or later

Kafka: 0.8.x or later

Zookeeper: 3.4.5

Java: 1.7.x

Spark,用于用户模型训练

Logstash-1.5.5

2.Kafka配置

启动kafka前需要先配置kafka中的zookeeper地址

./kafka-server-start.sh ../config/server.properties &

创建eagle需要的topic

./kafka-topics.sh --create --zookeeper slave02.mesos:2181,slave03.mesos:2181,slave04.mesos:2181 --replication-factor 1 --partitions 1 --topic Demo_hdfs_audit_log

3.配置eagle

修改bin/eagle-env.sh

修改conf/eagle-service.conf

# eagle configurationeagle{        # eagle web service configuration        service{                # storage type: ["hbase","jdbc"]                # default is "hbase"                storage-type="hbase"                # hbase configuration: hbase.zookeeper.quorum                # default is "localhost"                hbase-zookeeper-quorum="slave02.mesos,slave03.mesos,slave04.mesos"                # hbase configuration: hbase.zookeeper.property.clientPort                # default is 2181                hbase-zookeeper-property-clientPort=2181                # hbase configuration: zookeeper.znode.parent                # default is "/hbase"                zookeeper-znode-parent="/hbase"        }}

新建一个conf/Demo-hdfsAuditLog-application.conf

hadoop@master conf]$ cat Demo-hdfsAuditLog-application.conf {  "envContextConfig" : {    "env" : "storm",    "mode" : "cluster",    "topologyName" : "Demo-hdfsAuditLog-topology",    "stormConfigFile" : "security-auditlog-storm.yaml",    "parallelismConfig" : {      "kafkaMsgConsumer" : 1,      "hdfsAuditLogAlertExecutor*" : 1    }  },  "dataSourceConfig": {    "topic" : "Demo_hdfs_audit_log",    "zkConnection" : "slave02.mesos:2181,slave03.mesos:2181,slave04.mesos:2181",    "zkConnectionTimeoutMS" : 15000,    "fetchSize" : 1048586,    "deserializerClass" : "eagle.security.auditlog.HdfsAuditLogKafkaDeserializer",    "transactionZKServers" : "slave02.mesos,slave03.mesos,slave04.mesos",    "transactionZKPort" : 2181,    "transactionZKRoot" : "/consumers",    "consumerGroupId" : "eagle.hdfsaudit.consumer",    "transactionStateUpdateMS" : 2000  },  "alertExecutorConfigs" : {     "hdfsAuditLogAlertExecutor" : {       "parallelism" : 1,       "partitioner" : "eagle.alert.policy.DefaultPolicyPartitioner",       "needValidation" : "true"     }  },  "eagleProps" : {    "site" : "Demo",    "dataSource": "hdfsAuditLog",  "dataJoinPollIntervalSec" : 30,    "mailHost" : "tendcloud.com",    "mailSmtpPort":"25",    "mailDebug" : "true",    "eagleService": {      "host": "master.mesos",      "port": 9099      "username": "admin",      "password": "secret"    }  },  "dynamicConfigSource" : {  "enabled" : true,  "initDelayMillis" : 0,  "delayMillis" : 30000  }}

注意配置这里面的eagleService的host

4.启动

创建hbase表

bin/eagle-service-init.sh

启动eagle web service

bin/eagle-service.sh start

引入metadata

bin/eagle-topology-init.sh

5.登陆eagle web

http://ip:9099/eagle-service

用户名admin，密码secret 

Storm中的topology

至于添加policy和site等过程请参见官网。

最后的web页面如下：