Hive的两个问题
来源:互联网 发布:武装突袭3低配优化 编辑:程序博客网 时间:2024/05/21 07:59
Hive的两个问题:
问题一:Too Many Small Partitions
It can be tempting to partition your data into many small partitions to try to increase speed and concurrency.
However, Hive functions best when data is partitioned into larger partitions.
For example, consider partitioning a 100 TB table into 10,000 partitions, each 10 GB in size. In addition,
do not use more than 10,000 partitions per table. Having too many small partitions puts significant strain on the Hive MetaStore and does not
improve performance.
问题二:Hive Queries Fail with "Too many counters" Error
Hive operations use various counters while executing MapReduce jobs.
These per-operator counters are enabled by the configuration setting hive.task.progress.
This is disabled by default; if it is enabled, Hive may create a large number of counters (4 counters per operator, plus another 20).
Note:
If dynamic partitioning is enabled, Hive implicitly enables the counters during data load.
By default, CDH restricts the number of MapReduce counters to 120.
Hive queries that require more counters will fail with the "Too many counters" error.
What To Do
If you run into this error, set mapreduce.job.counters.max in mapred-site.xml to a higher value.
问题一:Too Many Small Partitions
It can be tempting to partition your data into many small partitions to try to increase speed and concurrency.
However, Hive functions best when data is partitioned into larger partitions.
For example, consider partitioning a 100 TB table into 10,000 partitions, each 10 GB in size. In addition,
do not use more than 10,000 partitions per table. Having too many small partitions puts significant strain on the Hive MetaStore and does not
improve performance.
问题二:Hive Queries Fail with "Too many counters" Error
Hive operations use various counters while executing MapReduce jobs.
These per-operator counters are enabled by the configuration setting hive.task.progress.
This is disabled by default; if it is enabled, Hive may create a large number of counters (4 counters per operator, plus another 20).
Note:
If dynamic partitioning is enabled, Hive implicitly enables the counters during data load.
By default, CDH restricts the number of MapReduce counters to 120.
Hive queries that require more counters will fail with the "Too many counters" error.
What To Do
If you run into this error, set mapreduce.job.counters.max in mapred-site.xml to a higher value.
0 0
- Hive的两个问题
- 两个Hive无法启动问题的解决
- mysql jdbc在hive中没有安装导致的两个问题
- hive union 的问题
- hive的union问题
- hive union 的问题
- hive的问题
- [Hive]求两个集合的减集
- hive改表结构的两个坑
- hive改表结构的两个坑
- hive改表结构的两个坑
- [Hive]关于Hive的启动问题
- hive的create table 问题
- Hive安装出现的问题
- hive 终端产生的问题
- Hive 安装出现的问题
- hive登录账号的问题
- hive安装出现的问题
- HTML练习注册用户
- 同位语-名词性从句(基础)
- iOS 微信支付宝支付完成后跳转回自己的app
- java获取项目路径
- “Rescue(营救),ZOJ1649”的一种解法和疑惑
- Hive的两个问题
- 解决Axure在Chorme无法直接打开的问题
- 经典例题:交换两个数
- 启动ip wizard时报the ip wizard does not support dhcp
- Perl基础总结
- iOS开发之各种动画各种页面切面效果
- Caffe在Cifar10上复现ResNet
- angularJS中自定义服务的理解
- 浅谈快速排序