Hive 关于修改表结构其他客户端不同步的问题
来源:互联网 发布:双字符域名注册 编辑:程序博客网 时间:2024/05/22 13:20
项目混用了多个HiveServer2 和 HiveCli。一段时间以后,出现了一个奇怪的问题。一个客户端修改了表结构以后,其他客户的访问的时候,却发现表结构并没有被同步修改。
查了不少资料,最后发现是因为HiveServer2 服务器开启了缓存的问题,记录如下:
问题出现原因是由于 L2 cache 这种缓存机制不支持分布式缓存。
关闭 L2 cache 后问题解决
<property>
<name>datanucleus.cache.level2.type</name>
<value>none</value>
</property>
因没有 HiveServer2 的配置权限,发现一个临时的解决办法。登陆到执行作业的 HiveServer2 上,使用 set 语句关闭这个参数,然后访问修改的表(desc 一下就可以),退出。下次再访问的时候就发现缓存已经更新了。
set datanucleus.cache.level2.type=none;
Hive 默认是关闭了这个参数的:
datanucleus.cache.level2.type
Default Value: none in Hive 0.9 and later; SOFT in Hive 0.7 to 0.8.1
Added In: Hive 0.7.0
NONE = disable the datanucleus level 2 cache, SOFT = soft reference based cache, WEAK = weak reference based cache.
Warning note: For most Hive installations, enabling the datanucleus cache can lead to correctness issues, and is dangerous. This should be left as "none".
JDO Cache(高速缓存):
Cachingis an essential mechanism in providing efficient usage of resources in manysystems. Data management using JDO is no different and provides a definition ofcaching at 2 levels. Caching allows objects to be retained and returned rapidlywithout having to make an extra call to the datastore. The 2 levels of cachingavailable with DataNucleus are
1.Level 1 Cache - mandated by the JDO specification, and represents the cachingof instances within a PersistenceManager
2.Level 2 Cache - represents the caching of instances within aPersistenceManagerFactory (across multiple PersistenceManager's)
Youcan think of a cache as a Map, with values referred to by keys. In the case ofJDO, the key is the object identity (identity is unique in JDO).
Bydefault the Level 2 Cache is enabled. The user can configure the Level 2 Cacheif they so wish. This is controlled by use of the persistence propertydatanucleus.cache.level2.type. You set this to "type" of cacherequired. With the Level 2 Cache you currently have the following options.
1.none - turn OFF Level 2 caching.
2.weak - use the internal (weak reference based) L2 cache. Provides support forthe JDO 2 interface of being able to pin objects into the cache, and unpin themwhen required.This option does notsupport distributed caching, solely running within the JVM of the clientapplication. Weak references are held to non pinned objects.
3.soft - use the internal (soft reference based) L2 cache. Provides support forthe JDO 2 interface of being able to pin objects into the cache, and unpin themwhen required.This option does notsupport distributed caching, solely running within the JVM of the clientapplication. Soft references are held to non pinned objects.
参考:
http://blog.csdn.net/lalaguozhe/article/details/9184593
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties
- Hive 关于修改表结构其他客户端不同步的问题
- Hive分区表修改表结构的问题
- Hive分区表修改表结构的问题
- 修改Hive表结构
- 关于service和activity交互修改sharedpreferences数据不同步的问题
- 关于服务器时钟不同步的问题
- 如何让gg只同步数据插入操作,其他的像删除和修改操作不同步?
- svn客户端与svn_eclipse插件不同步问题
- 关于服务器端通知客户端修改信息的问题
- 关于服务器端通知客户端修改信息的问题
- [Hive]关于Hive的启动问题
- 关于修改表结构
- hive中表结构修改,表删除
- vmware时间不同步的问题
- 关于ubuntu和windows时间不同步问题
- WCF客户端无法访问服务原因之一时间不同步问题
- 关于解决ubuntu,windows双系统的引导,及时间不同步问题
- 关于gluster数据不同步的处理
- 实战Solr分布式实时搜索集群搭建
- JNI编程-让C++和Java相互调用
- Android程序效率的几点建议
- Android中的“再按一次退出程序”代码实现
- UML用例图总结
- Hive 关于修改表结构其他客户端不同步的问题
- 通过Sqoop工具将Oracle数据导入到HBase
- 好习惯要坚持
- win7 Ubuntu 双系统重装 系统引导修复
- 设计——分享一些图标库,知乎的一场live里推荐的
- AnjularJS杂记1----关于AngularJS你该知道的
- sails.js使用mysql-queries执行批量SQL
- upupw 知识点
- js获取当前根目录