Hive Metastore Thrift Server Partition压力测试

测试环境:

虚拟机10.1.77.84

四核Intel(R) Xeon(R) CPU E5649  @ 2.53GHz，8G内存

起单台metastore server

JVM args : -Xmx7000m -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC

测试表Schema:

table name: metastore_stress_testing

# col_name data_type comment             

a string None

# Partition Information

# col_name data_type comment

pt string None

# Detailed Table Information

Database:           default             

Owner:              hadoop              

CreateTime:         Wed May 29 11:29:30 CST 2013

LastAccessTime:     UNKNOWN             

Protect Mode:       None                

Retention:          0                   

Location:           hdfs://10.1.77.86/user/hive9/warehouse/metastore_stress_testing

Table Type:         MANAGED_TABLE       

Table Parameters:

transient_lastDdlTime1369798170          

# Storage Information

SerDe Library:      org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

InputFormat:        org.apache.hadoop.mapred.TextInputFormat

OutputFormat:       org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

Compressed:         No                  

Num Buckets:        -1                  

Bucket Columns:     []                  

Sort Columns:       []                  

Storage Desc Params:

serialization.format1

测试方法：

Hive在解析一条query的时候，如果不加table partition predicate(比如直接select * from dpdw_traffic_base)，会在生成物理执行计划，物理计划优化环节PartitionPruner 中调用Hive Metastore Client的getPartition(Table)方法，获取整张表的的List<Partition>，由于每个Partition 可以看成一张大表下的子表，里面包含了各自独立的InputFormatClass, OutputFormatClass, Deserializer, Column Schema , StorageDescriptor，Parameters等等，如果表结构比较复杂，column比较多，Partitions又比较多，会造成返回的对象非常大，而且会对metastore server 造成很大压力。

我们通过模拟HiveMetastoreClient的listPartitions方法来测试性能表现，HiveConf中默认设置metastore client socket timeout超时为20秒(hive.metastore.client.socket.timeout)。为了防止测试过程中read timeout ,我们先将这个值设得很大

生成物理执行计划的call stack:

测试代码：

public class MetastoreTestConcurrent { public long timeTasks(int nThread) throws Exception { final CountDownLatch startGate = new CountDownLatch(1); final CountDownLatch endGate = new CountDownLatch(nThread); for (int i = 0; i < nThread; i++) {   Thread t = new Thread(new Runnable() {     @Override     public void run() {         try {             startGate.await();             try {               HiveConf hConf = new HiveConf();               hConf.setIntVar(ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT, 100000);               HiveMetaStoreClient hmsc = new HiveMetaStoreClient(hConf);               long start = System.currentTimeMillis();               List<Partition> parts = hmsc.listPartitions("default", "metastore_stress_testing", (short) -1);               long end = System.currentTimeMillis();               System.out.println(Thread.currentThread().getName() + " partition_number:" +                 parts.size() +" duration:" + (end - start));             } catch (Exception e) {}             finally {             endGate.countDown();             }         } catch (InterruptedException e) {}        }     });     t.start();   }     long start = System.currentTimeMillis();     startGate.countDown();     endGate.await();     long end = System.currentTimeMillis();     return end - start;}     public static void main(String[] args) throws Exception {      MetastoreTestConcurrent mtc = new MetastoreTestConcurrent();      System.out.println("total duration :" + mtc.timeTasks(20));     }}

测试结果：

hive metastore server 默认启动的时候会启最小minWorkerThreads = 200， 最大maxWorkerThreads = 100000 的ThreadPoolExecutor，并开启TCP keepalive = true

单个Thread:

duration=3500 millis

partition number=1000

duration=16874 millis

partition number=5000

duration=53697 millis

partition number=20000

Partition List RamUsage: 46264576 bytes

duration:75809 millis

partition number:27092

duration :245805 millis

partition number:50000

三个thread

Thread Thread-2  partition_number:26596 duration:96902 millis

Thread Thread-0  partition_number:26596 duration:97135 millis

Thread Thread-1  partition_number:26596 duration:97193 millis

total duration :100235 millis

十个thread

Thread Thread-4 partition_number:27952 duration:255172 millis

Thread Thread-2 partition_number:27952 duration:257144 millis

Thread Thread-0 partition_number:27952 duration:257424 millis

Thread Thread-7 partition_number:27952 duration:258388 millis

Thread Thread-5 partition_number:27952 duration:260496 millis

Thread Thread-1 partition_number:27952 duration:261106 millis

Thread Thread-3 partition_number:27952 duration:261173 millis

Thread Thread-6 partition_number:28386 duration:324847 millis

Thread Thread-8 partition_number:28506 duration:356391 millis

Thread Thread-9 partition_number:28507 duration:356861 millis

total duration: 361216 millis

五个Thread:

Thread-1 partition_number:50000 duration: 555805 millis

Thread-3 partition_number:50000 duration:557810 millis

Thread-2 partition_number:50000 duration:558112 millis

Thread-0 partition_number:50000 duration:558480 millis

Thread-4 partition_number:50000 duration:558897 millis

total duration :560753 millis

结论

Hive在单线程环境下调用listPartitions时间基本成线性增长，在Partition数为20000的时候，返回的结果已经将近46MB了，这还是只有一个实体column和一个virutal column的情况下，由于前面提到partition之间是相互独立的，所以如果partition非常多，会crash metastore server，也会crach本地hive client。

在多线程环境下，调用时间明显增加，因为metastore server没有query cache，每一次调用都会变成对于底层数据库的一次查询。

综合来看，partition比较多的话，一个是在compile环节上会一直block在获取partition schema环节上，从而无法提交job，另一个是即使获得了全表下所有partition信息，会生成job splits file，这个文件也会非常大，会撑满jvm heap，也有可能撑满本地磁盘（会先serialize到本地再提交到hdfs上)

ps:

大前天晚上部署的hive metastore server宕机了
从日志上看应该是用户获取partition信息(get_partitions_with_auth_result方法)的时候size太大导致的OOM，看来还是得限制下partition个数

Exception in thread "pool-1-thread-196" java.lang.OutOfMemoryError: Java heap space        at java.util.Arrays.copyOf(Arrays.java:2786)        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)        at org.apache.thrift.transport.TSaslTransport.write(TSaslTransport.java:445)        at org.apache.thrift.transport.TSaslServerTransport.write(TSaslServerTransport.java:40)        at org.apache.thrift.protocol.TBinaryProtocol.writeI16(TBinaryProtocol.java:154)        at org.apache.thrift.protocol.TBinaryProtocol.writeFieldBegin(TBinaryProtocol.java:109)        at org.apache.hadoop.hive.metastore.api.FieldSchema.write(FieldSchema.java:421)        at org.apache.hadoop.hive.metastore.api.StorageDescriptor.write(StorageDescriptor.java:1086)        at org.apache.hadoop.hive.metastore.api.Partition.write(Partition.java:897)        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_with_auth_result.write(ThriftHiveMetastore.java:33814)        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:34)        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)

本文链接http://blog.csdn.net/lalaguozhe/article/details/9070203，转载请注明