Presto-[11]-Administration-Resource Groups

原文

https://prestodb.io/docs/current/admin/resource-groups.html

Resource groups默认不生效，需要设置experimental.resource-groups-enabled=true 

以组（可嵌套）的形式管理查询，组织排队情况

manager来配置resource groups和相关的角色选择，并且这是pluggable的，

添加etc/resource-groups.properties 添加如下内容，可使能一个内嵌的manager，读取JSON配置文件：

resource-groups.configuration-manager=fileresource-groups.config-file=etc/resource_groups.json

修改resource-groups.config-file的值指向一个JSON config file，可以是绝对路径，或相对Presto的相对目录

Resource Group Properties

• name (required): 组名

• maxQueued (required): 最大排队查询数，超过该值，新的查询将被拒绝  

• hardConcurrencyLimit (required):处于运行中状态的最大查询数  

• softMemoryLimit (required): 在接收一个新查询排队前该组可能使用的最大分布内存 ，

  可以是绝对值如1GB，或集群内存比例 

• softCpuLimit (optional): 该组周期性(see cpuQuotaPeriod)使用CPU time 的最大值  

• hardCpuLimit (optional): 该组周期性(see cpuQuotaPeriod)使用CPU time 的最大值  

• schedulingPolicy (optional): 标识如何调度查询，子组合适符合条件来启动他们的查询，可以是下面的3值之一:

  • fair (default): 排队中的查询以first-in-first-out形式调度，如果子组中有排队，轮流执行 .
  • weighted: 依据优先级权重执行 (通过 query_priority指定 session property). 子组依据 schedulingWeight.参数被调度
  • query_priority: 所有的子组必须设置优先级  query_priority. Queued queries 严格按照优先级被选中执行.

• schedulingWeight (optional): weight of this sub group. See above. Defaults to 1.

• jmxExport (optional): If true, group statistics are exported to JMX for monitoring. Defaults to false.

• queuedTimeLimit (optional): maximum amount of time a query may be in the queue for this group before it is marked as failed.

• runningTimeLimit (optional): maximum amount of time a query in this group can execute (not including queued time) before it is marked as failed.

• subGroups (optional): list of sub groups.

Selector Properties

• user (optional): regex to match against user name. Defaults to .*

• source (optional): regex to match against source string. Defaults to .*

• queryType (optional): string to match against the type of the query submitted. 

  query type :

  DATA_DEFINITION: Queries that alter/create/drop the metadata of schemas/tables/views, and that manage prepared statements, privileges, sessions, and transactions.

  DELETE: DELETE queries.

  DESCRIBE: DESCRIBE, DESCRIBE INPUT, DESCRIBE OUTPUT, and SHOW queries.

  EXPLAIN: EXPLAIN queries.INSERT: INSERT and CREATE TABLE AS SELECT queries.

  SELECT: SELECT queries.

• group (required): 这些查询运行的组.

Global Properties

• cpuQuotaPeriod (optional): cpu quotas 强制period

Selectors被有序执行，首个匹配的将被应用。下面的配置有5个resource group templates，adhoc_${USER} group, ${USER}

会把submitted中的用户名替代进去，${SOURCE}也类似，source name can be set as follows

• CLI: use the --source option.
• JDBC: set the ApplicationName client info property on the Connection instance.

有3个selectors定义那些queries在那些resource group中运行

• The first selector places queries from bob into the admin group.
• The second selector states that all data definition queries that come from a source that includes “pipeline” should run in the user’s personal data definition group, which belongs to the globa.data_definition parent group.
• The third selector states that all queries that come from a source that includes “pipeline” should run in the user’s personal pipeline group, which belongs to the global.pipeline parent group.
• The last selector is a catch all, which puts all queries into the user’s adhoc group.

All together these selectors implement the policy that bob is an admin and all other users are subject to the following limits:

• Users are allowed to have up to 2 adhoc queries running. Additionally, they may run one pipeline.
• No more than 5 “pipeline” queries may run at once.
• No more than 100 total queries may run at once, unless they’re from the admin.

{  "rootGroups": [    {      "name": "global",      "softMemoryLimit": "80%",      "hardConcurrencyLimit": 100,      "maxQueued": 1000,      "schedulingPolicy": "weighted",      "jmxExport": true,      "subGroups": [        {          "name": "data_definition_${USER}",          "softMemoryLimit": "10%",          "hardConcurrencyLimit": 3,          "maxQueued": 10,          "schedulingWeight": 1        },        {          "name": "adhoc_${USER}",          "softMemoryLimit": "10%",          "hardConcurrencyLimit": 2,          "maxQueued": 1,          "schedulingWeight": 9,          "schedulingPolicy": "query_priority"        },        {          "name": "pipeline",          "softMemoryLimit": "20%",          "hardConcurrencyLimit": 5,          "maxQueued": 100,          "schedulingWeight": 1,          "jmxExport": true,          "subGroups": [            {              "name": "pipeline_${USER}",              "softMemoryLimit": "10%",              "hardConcurrencyLimit": 1,              "maxQueued": 100,              "schedulingPolicy": "query_priority"            }          ]        }      ]    },    {      "name": "admin",      "softMemoryLimit": "100%",      "hardConcurrencyLimit": 200,      "maxQueued": 100,      "schedulingPolicy": "query_priority",      "jmxExport": true    }  ],  "selectors": [    {      "user": "bob",      "group": "admin"    },    {      "source": ".*pipeline.*",      "queryType": "DATA_DEFINITION",      "group": "global.data_definition_${USER}"    },    {      "source": ".*pipeline.*",      "group": "global.pipeline.pipeline_${USER}"    },    {      "group": "global.adhoc_${USER}"    }  ],  "cpuQuotaPeriod": "1h"}