【解决】hive动态增加partitions不能超过100的问题

来源：互联网发布：淘宝无法投诉卖家编辑：程序博客网时间：2024/05/21 10:33

【解决】hive动态增加partitions不能超过100的问题，全量动态生成partitions超过100会出现如下异常：

The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100

解决100限制，可设置如下参数：

set hive.support.concurrency=false;set hive.exec.dynamic.partition.mode=nonstrict;  SET hive.exec.max.dynamic.partitions=100000;   SET hive.exec.max.dynamic.partitions.pernode=100000;   insert overwrite table ods.cms_entity PARTITION (DAY)  select   ENTITY_ID  ,  ENTITY_NAME  ,  ENTITY_DESC  ,  ENTITY_TYPE  ,  ENTITY_PID  ,  ENTITY_TIME  ,  ENTITY_PRIORITY  ,  ENTITY_STATUS  ,  ENTITY_CHANNEL  ,  ENTITY_EDITOR  ,  ENTITY_TEMPLATE  ,  ENTITY_URL    ,  ENTITY_CATEGORY  ,  ENTITY_PARAM  ,  ENTITY_SHORTNAME  ,  ENTITY_SUBTYPE  ,  ENTITY_COMPDELAY  ,  day  from stage.cms_entity_by_day;

设置动态partitions为100000，该语句插入几年按天的partitions超过1000个，成功执行语句。

说明，防止出现hive分区并发访问被锁的问题需加上配置

set hive.support.concurrency=false;

1 0