Druid中coordinator的介绍
来源：互联网 发布：c抓取网页数据 编辑：程序博客网 时间：2024/05/18 03:49
核心
1、主要介绍coordinate的作用
2、借助coordinate完成一些查询操作
coordinate的作用
druid协调节点主要负责管理segment和分发，更具体的说，druid协调节点和历史节点进行通信，根据配置进行加载segment或删除segment。druid的协调节点负责加载新的segment，删除过时的segment，管理segment副本和做segment的平衡。druid协调器定期运行，每次运行之间的时间是一个可配置的参数，每次druid协调器运行的时候，它将决定采取的适当行动之前评估集群的当前状态。和broker和historical节点相似，druid协调器和zookeeper连接获取当前集群的信息，协调器还维护与包含segment和rules信息的数据库的连接，有效的segment存储在segment表里，并列出集群中加载的所有segment，rules存储在rule表中并提升如何处理segment。在historical节点没有任何segment的时候，每层的可用historical首先按容量排序，最小容量服务器具有最高优先级，每个分配的segment总是分配给具有最小容量的节点，以保持节点之间的平衡，协调器在为其分配新的segment时不直接和historical进行通信，而是在historical节点的加载队列路径下创建新segment的一些临时信息，一旦看到这个请求，historical将加载该segment并开始服务。
运行协调器
io.druid.cli.Main server coordinator
rules
可以根据一组规则从集群自动加载和删除segment
清理segment
每次运行，协调器将数据库中有效的segment与当前集群中segment进行比较，不在数据库中但仍在集群中的segment将会被标记并添加到删除列表里面，被覆盖的segment
(版本太旧了，数据已被新的segment替代)也被删除，如果数据库中所有segment都被删除，则协调器将不会删除historical任何内容。
segment可用性
如果historical节点由于任何 原因重启或者变得不可用，则druid协调器将监听到一个节点已经丢失，并将该节点所服务的所有segment都视为丢弃，给定足够的时间，可以将segment重新分配给集群中
的其他historical节点，但是，每个segment都不会立即被删除，有一个过渡的数据结构存储具有关联寿命的所有丢弃的segment，在生命周期内协调器不会重新分配segment，因此，如果historical
在短时间内变得不可以用并再次可用，historical节点在启动并从高速缓存中提取segment，而不会在整个集群中重新分配这个segment
segment平衡
为了确保集群中的历史节点之间的分配均匀，协调器将在每次协调器运行时找到每个historical所有服务的segment的总大小，对于集群中的每个historical节点，协调器将确定利用率最高的historical和利用率最低的historical,计算两个节点之间利用率的差异百分比，如果超过某个阀值，则从最高使用节点移动到最低使用节点，每次协调器有运行，可以从一个节点移动到另一个节点的segment数量可配置
http请求
协调器公开了多个http请求进行交互
coordinate获取集群信息
GET
/status
返回druid版本，加载的扩展，使用的内存，总内存和其他有关节点的有用信息
[root@SZB-L0038788 druid-0.10.0]# curl -X GET http://10.20.23.38:8081/status{ "version": "0.10.0", "modules": [ { "name": "io.druid.storage.hdfs.HdfsStorageDruidModule", "artifact": "druid-hdfs-storage", "version": "0.10.0" }, { "name": "io.druid.server.lookup.namespace.NamespaceExtractionModule", "artifact": "druid-lookups-cached-global", "version": "0.10.0" }, { "name": "io.druid.firehose.kafka.KafkaEightDruidModule", "artifact": "druid-kafka-eight", "version": "0.10.0" }, { "name": "io.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule", "artifact": "druid-datasketches", "version": "0.10.0" }, { "name": "io.druid.metadata.storage.mysql.MySQLMetadataStorageModule", "artifact": "mysql-metadata-storage", "version": "0.10.0" }, { "name": "io.druid.query.aggregation.datasketches.theta.SketchModule", "artifact": "druid-datasketches", "version": "0.10.0" }, { "name": "io.druid.query.aggregation.histogram.ApproximateHistogramDruidModule", "artifact": "druid-histogram", "version": "0.10.0" } ], "memory": { "maxMemory": 266862592, "totalMemory": 266862592, "freeMemory": 157169296, "usedMemory": 109693296 }}coordinater 信息
/druid/coordinator/v1/leader
返回集群当前leader 协调器
[root@SZB-L0038788 druid-0.10.0]# curl http://10.20.23.38:8081/druid/coordinator/v1/leader 10.20.23.38:8081/druid/coordinator/v1/isLeader
返回带有 leader的json对象，为true或者false，表改服务器是否是该集群的leader协调器，此外，如果服务器是当前的leader，则返回HTTP 200否则返回HTTP 400，如果你只希望
active leader在负载均衡上被考虑在服务上，则适用于负载均衡状态检查
/druid/coordinator/v1/loadstatus
返回在集群中实际加载segment的百分比应该在集群中加载segment的百分比
{ "wikiticker123": 100, "app_auto_prem_qd_pp1": 100, "app_auto_prem_qd_pp3": 100}/druid/coordinator/v1/loadstatus?simple
返回待加载的segment，直到在集群中加载的segment可用于查询，不包括复制
{ "wikiticker123": 0, "app_auto_prem_qd_pp1": 0, "app_auto_prem_qd_pp3": 0}/druid/coordinator/v1/loadstatus?full
返回每个DataSource加载的segment，直到在集群中加载的segment都可以用，这个包括复制
{ "_default_tier": { "wikiticker123": 1, "app_auto_prem_qd_pp1": 2706, "app_auto_prem_qd_pp3": 1 }}/druid/coordinator/v1/loadqueue
返回每个历史节点加载和删除的segmentids
{ "10.20.23.38:8083": { "segmentsToLoad": [], "segmentsToDrop": [] }}/druid/coordinator/v1/loadqueue?simple
返回要加载和删除的segment数，以及每个historical节点的总segment加载和总大小
{ "10.20.23.38:8083": { "segmentsToLoad": 0, "segmentsToDrop": 0, "segmentsToLoadSize": 0, "segmentsToDropSize": 0 }}/druid/coordinator/v1/loadqueue?full
返回要为每个historical加载和删除的segment序列化json
{ "10.20.23.38:8083": { "segmentsToLoad": [], "segmentsToDrop": [] }}元数据信息
/druid/coordinator/v1/metadata/datasources
获取集群中DataSource的列表
[ "app_auto_prem_qd_pp1", "app_auto_prem_qd_pp3", "wikiticker123"]/druid/coordinator/v1/metadata/datasources?includeDisabled
返回集群中启用和禁用的DataSource
[ "app_auto_prem_qd_pp1", "app_auto_prem_qd_pp3", "wikiticker", "wikiticker11", "wikiticker123"]/druid/coordinator/v1/metadata/datasources?full
返回所有启用的数据源列表，其中包含存储的元数据存储中的相关这些数据源的所有元数据
{ "name": "wikiticker123", "properties": { "created": "2017-09-08T17:46:17.006+08:00" }, "segments": [ { "dataSource": "wikiticker123", "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00", "version": "2017-08-03T17:43:40.525+08:00", "loadSpec": { "type": "hdfs", "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip" }, "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user", "metrics": "count,added,deleted,delta,user_unique", "shardSpec": { "type": "none" }, "binaryVersion": 9, "size": 5537818, "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00" } ]}/druid/coordinator/v1/metadata/datasources/{dataSourceName}
返回指定DataSource的完整元数据
{ "name": "wikiticker123", "properties": { "created": "2017-09-08T17:49:17.231+08:00" }, "segments": [ { "dataSource": "wikiticker123", "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00", "version": "2017-08-03T17:43:40.525+08:00", "loadSpec": { "type": "hdfs", "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip" }, "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user", "metrics": "count,added,deleted,delta,user_unique", "shardSpec": { "type": "none" }, "binaryVersion": 9, "size": 5537818, "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00" } ]}/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments
返回元数据中所有segment的列表
[ "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"]/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments?full
返回元数据所有segment的信息
[ { "dataSource": "wikiticker123", "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00", "version": "2017-08-03T17:43:40.525+08:00", "loadSpec": { "type": "hdfs", "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip" }, "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user", "metrics": "count,added,deleted,delta,user_unique", "shardSpec": { "type": "none" }, "binaryVersion": 9, "size": 5537818, "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00" }]POST /druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments
返回DataSource的所有segment列表，所有的segment重叠
例如 ["2012-01-01T00:00:00.000/2012-01-03T00:00:00.000", "2012-01-05T00:00:00.000/2012-01-07T00:00:00.000"][root@SZB-L0038788 druid-0.10.0]# curl -POST http://10.20.23.38:8081/druid/coordinator/v1/metadata/datasources/wikiticker123/segments["wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"]POST
/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments?full
返回DataSource的所有segment列表的完整信息，所有的segment重叠
[root@SZB-L0038788 druid-0.10.0]# curl -POST http://10.20.23.38:8081/druid/coordinator/v1/metadata/datasources/wikiticker123/segments?full[ { "dataSource": "wikiticker123", "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00", "version": "2017-08-03T17:43:40.525+08:00", "loadSpec": { "type": "hdfs", "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip" }, "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user", "metrics": "count,added,deleted,delta,user_unique", "shardSpec": { "type": "none" }, "binaryVersion": 9, "size": 5537818, "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00" }]/druid/coordinator/v1/metadata/datasources/{dataSourceName}/segments/{segmentId}
返回存储在元数据存储中指定的完成元数据
curl -POST http://10.20.23.38:8081/druid/coordinator/v1/metadata/datasources/wikiticker123/segments/wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00{ "dataSource": "wikiticker123", "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00", "version": "2017-08-03T17:43:40.525+08:00", "loadSpec": { "type": "hdfs", "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip" }, "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user", "metrics": "count,added,deleted,delta,user_unique", "shardSpec": { "type": "none" }, "binaryVersion": 9, "size": 5537818, "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00"}数据源信息
/druid/coordinator/v1/datasources
返回集群中DataSource列表
[ "app_auto_prem_qd_pp1", "app_auto_prem_qd_pp3", "wikiticker123"]/druid/coordinator/v1/datasources?simple
返回集群中DataSource列表包含segment计算，总segment字节大小，minTime和maxTime
[ { "name": "app_auto_prem_qd_pp1", "properties": { "tiers": { "_default_tier": { "size": 40878466, "segmentCount": 2706 } }, "segments": { "maxTime": "2017-07-28T08:00:00.000+08:00", "size": 40878466, "minTime": "2003-02-25T08:00:00.000+08:00", "count": 2706 } } }, { "name": "app_auto_prem_qd_pp3", "properties": { "tiers": { "_default_tier": { "size": 82445053, "segmentCount": 1 } }, "segments": { "maxTime": "2017-08-23T08:00:00.000+08:00", "size": 82445053, "minTime": "2017-08-22T08:00:00.000+08:00", "count": 1 } } }, { "name": "wikiticker123", "properties": { "tiers": { "_default_tier": { "size": 5537818, "segmentCount": 1 } }, "segments": { "maxTime": "2015-09-13T08:00:00.000+08:00", "size": 5537818, "minTime": "2015-09-12T08:00:00.000+08:00", "count": 1 } } }]/druid/coordinator/v1/datasources?full
返回集群中DataSource列表，其中包含这些DataSource所有元数据
{ "name": "wikiticker123", "properties": { "created": "2017-09-08T17:46:17.006+08:00" }, "segments": [ { "dataSource": "wikiticker123", "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00", "version": "2017-08-03T17:43:40.525+08:00", "loadSpec": { "type": "hdfs", "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip" }, "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user", "metrics": "count,added,deleted,delta,user_unique", "shardSpec": { "type": "none" }, "binaryVersion": 9, "size": 5537818, "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00" } ]}/druid/coordinator/v1/datasources/{dataSourceName}
返回指定DataSource的json对象，包括segment计数，总segment大小，minTime和maxTime
/druid/coordinator/v1/datasources/{dataSourceName}?full
返回DataSource完整元数据
/druid/coordinator/v1/datasources/{dataSourceName}/intervals
返回一组segment间隔
/druid/coordinator/v1/datasources/{dataSourceName}/intervals?simple
返回指定DataSource的segment全部信息
/druid/coordinator/v1/datasources/{dataSourceName}/intervals?full
返回DataSource映射到该间隔的服务器名称
/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}
返回间隔一组segmentID，需要注意 间隔参数有/修改成_例如2017-07-27T08:00:00.000+08:00/2017-07-28T08:00:00.000+08:00需要写成 2017-07-27T08:00:00.000+08:00_2017-07-28T08:00:00.000+08:00
/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}?simple
将segment间隔的大小和数量返回为json
{ "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00": { "size": 5537818, "count": 1 }}/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}?full
将segment的大小完整元数据返回
{ "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00": { "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00": { "metadata": { "dataSource": "wikiticker123", "interval": "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00", "version": "2017-08-03T17:43:40.525+08:00", "loadSpec": { "type": "hdfs", "path": "hdfs://10.20.23.29:9000/druid/index/output/wikiticker123/20150912T080000.000+0800_20150913T080000.000+0800/2017-08-03T17_43_40.525+08_00/0_index.zip" }, "dimensions": "channel,cityName,comment,countryIsoCode,countryName,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,metroCode,namespace,page,regionIsoCode,regionName,user", "metrics": "count,added,deleted,delta,user_unique", "shardSpec": { "type": "none" }, "binaryVersion": 9, "size": 5537818, "identifier": "wikiticker123_2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00_2017-08-03T17:43:40.525+08:00" }, "servers": [ "10.20.23.38:8083" ] } }}/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}/serverview
返回segment的大小完整信息包含服务器信息
/druid/coordinator/v1/datasources/{dataSourceName}/segments
返回指定dataSource的所有segment列表
/druid/coordinator/v1/datasources/{dataSourceName}/segments?full
返回指定dataSource的所有segment列表的完整信息
/druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}
返回指定dataSource的指定segment的完整信息
/druid/coordinator/v1/datasources/{dataSourceName}/tiers
返回DataSource的规则
rules 规则
/druid/coordinator/v1/rules
返回集群中所有规则
{ "_default": [ { "tieredReplicants": { "_default_tier": 2 }, "type": "loadForever" } ]}/druid/coordinator/v1/rules/{dataSourceName}
返回指定数据源的规则
/druid/coordinator/v1/rules/{dataSourceName}?full
返回指定数据源的所有规则
/druid/coordinator/v1/rules/history?interval=< interval>
返回所有DataSource的规则的审计历史记录，可以配置druid.audit.manager.auditHistoryMillis这个执行间隔默认是1周
{ "error": "Invalid format: \"2015-09-12T08:00:00.000 08:00\" is malformed at \" 08:00\""}/druid/coordinator/v1/rules/history?count=< n>
返回所有数据源的规则审计数目
/druid/coordinator/v1/rules/{dataSourceName}/history?interval=<
interval>
返回指定DataSource的规则的审计历史记录，可以配置druid.audit.manager.auditHistoryMillis这个执行间隔默认是1周
/druid/coordinator/v1/rules/{dataSourceName}/history?count=< n>
返回指定数据源的规则审计数目
Intervals 间隔
需要注意分隔符有_替换 /
/druid/coordinator/v1/intervals
返回所有数据源总大小和数量的间隔
/druid/coordinator/v1/intervals/{interval}
返回指定间隔的总大小和数量
curl http://10.20.23.38:8081/druid/coordinator/v1/intervals/2015-09-12T08:00:00.000+08:00_2015-09-13T08:00:00.000+08:00{ "size": 5550558, "count": 2}/druid/coordinator/v1/intervals/{interval}?simple
返回指定间隔的大小和数量
{ "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00": { "size": 5550558, "count": 2 }}/druid/coordinator/v1/intervals/{interval}?full
返回指定间隔每个数据源的大小和数量
{ "2015-09-12T08:00:00.000+08:00/2015-09-13T08:00:00.000+08:00": { "wikiticker123": { "size": 5537818, "count": 1 }, "app_auto_prem_qd_pp1": { "size": 12740, "count": 1 } }}POST
/druid/coordinator/v1/datasources/{dataSourceName}
返回指定数据源规则和segment信息
{ "tiers": { "_default_tier": { "size": 5537818, "segmentCount": 1 } }, "segments": { "maxTime": "2015-09-13T08:00:00.000+08:00", "size": 5537818, "minTime": "2015-09-12T08:00:00.000+08:00", "count": 1 }}/druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}
返回指定DataSource和指定segment的信息
Rules
/druid/coordinator/v1/rules/{dataSourceName}
返回指定DataSource的规则
DELETE
/druid/coordinator/v1/datasources/{dataSourceName}
禁用datasource
/druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}
@Deprecated.
/druid/coordinator/v1/datasources/{dataSourceName}?kill=true&interval={myISO8601Interval}
指定的间隔和DataSource执行kill
/druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId}
禁用segment
- Druid中coordinator的介绍
- Druid的Coordinator负载均衡原理
- Druid大数据实时处理的开源分布式系统——Coordinator
- Druid是什么和Druid的介绍
- com.alibaba.druid的介绍
- Coordinator
- druid介绍
- Druid介绍
- DRUID介绍
- Druid介绍
- oozie的coordinator配置文件讲解
- Druid 简单介绍
- Druid原理介绍(1)
- Druid原理介绍(2)
- 阿里巴巴druid介绍
- Druid 介绍及配置
- Druid介绍与实践
- Druid 介绍及配置
- cdq分治bzoj3295
- webServer
- 将字符串json写入到服务器文件中
- Google官方刷题资源-基础题-第四题~第八题
- Android6.0 Sensor架构和问题分析
- Druid中coordinator的介绍
- zigbee（题型整理4）
- 494. Target Sum
- 鸡尾酒排序
- Nginx+tomcat 负载均衡需要注意的问题
- Log4j初学
- 典型的网络安全威胁
- mxnet安装记录
- 从请求对象中获取客户端ip
