spark sql 表分区出现损坏

表分区出现损坏

 

1) Create a table"test". "create table test (n string) partitioned by (pstring)"
2) Load some data into partition(p='1')
3)Remove the path related to partition(p='1') of table test manually."hadoop fs -rmr ..../warehouse/..../test/p=1"
4)Run spark sql, spark-sql -e "select n from test where p='1';"

Then it throws exception:

Error:org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:hdfs://hacluster/user/hive/warehouse/dp_tmp.db/spark_15044/p=2016-12-26(state=,code=0)

解决办法：

1、损坏分区数据不参与计算(hive采用这种方式解决)

setspark.sql.hive.verifyPartitionPath=true;

表示忽略这个损坏的分区数据

2、重新建分区overwrite 或者 drop tablepartition()