load数据到hive分区表报错: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTas

来源:互联网 发布:java中main函数的参数 编辑:程序博客网 时间:2024/05/16 05:21

在启动hive创建表的时候,兴许你们回碰到这个错误:http://blog.csdn.net/qq_35732963/article/details/54139581;百度一下,说是字符编码的问题,那个好解决,但是下面这个错,却不一样,归根到底是数据库出了问题!

1、上传数据到hive表时报错:

hive> load data local inpath '/home/hadoop/data1.txt' into table partition_tablespartition(gender='M');
Loading data to table hive1.partition_tables partition (gender=M)

Failed with exception MetaException(message:javax.jdo.JDODataStoreException: Error(s) were found while auto-creating/validating the datastore for classes. The errors are printed in the log, and are attached to this exception.
    at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
    at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
    at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
    at org.apache.hadoop.hive.metastore.ObjectStore.addPartition(ObjectStore.java:1431)
。。。。

    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

2、说到底还是改下数据库方面的:删除原先自动创建的hive数据库,新建metastore数据库

mysql> create database metastore;
Query OK, 1 row affected (0.01 sec)

mysql> show create database metastore;
+-----------+--------------------------------------------------------------------+
| Database  | Create Database                                                    |
+-----------+--------------------------------------------------------------------+
| metastore | CREATE DATABASE `metastore` /*!40100 DEFAULT CHARACTER SET utf8 */ |
+-----------+--------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql> alter database metastore  character set latin1;
Query OK, 1 row affected (0.00 sec)

3、去hadoop用户下修改hive/conf里面的hive-site.xml

这里原本是hive的数据库,但由于报错,把原来的hive'删了,新建一个metastore

[hadoop@master conf]$ vi hive-site.xml

<!-- metadata database connection configuration -->
<property>
  <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true</value>   ----这个参数使用来设置元数据连接字串
    <description>JDBC connect string for a JDBC metastore</description>
</property>


4、 ---给创建的2个文件 权限

[root@master hadoop]# ll  
total 76
drwxrwxr-x.  4 hadoop hadoop 4096 Dec 26 19:30 app
-rw-r--r--   1 hadoop hadoop   55 Jan  9 19:12 data1.txt
-rw-r--r--   1 root   root     44 Jan  9 19:13 data2.txt

[root@master hadoop]# chgrp hadoop data2.txt
[root@master hadoop]# chown hadoop data2.txt

5、启动hive

[hadoop@master conf]$ hive
Logging initialized using configuration in file:/home/hadoop/hive-0.14.0/conf/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.6.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hive-0.14.0/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

hive> create database hive1;
OK
Time taken: 0.415 seconds
hive> create table partition_tables
    > (tid int,tname string, age int)
    > partitioned by (gender string)
    > row format delimited fields terminated by ',';
OK
Time taken: 0.328 seconds

6、再次上传文件到分区表,显示上传成功

hive> load data local inpath '/home/hadoop/data1.txt' into table partition_tables partition(gender='M');
Loading data to table default.partition_tables partition (gender=M)
Partition default.partition_tables{gender=M} stats: [numFiles=1, numRows=0, totalSize=55, rawDataSize=0]
OK
Time taken: 1.138 seconds

hive>  load data local inpath '/home/hadoop/data2.txt' into table partition_tables partition(gender='F');
Loading data to table default.partition_tables partition (gender=F)
Partition default.partition_tables{gender=F} stats: [numFiles=1, numRows=0, totalSize=44, rawDataSize=0]
OK
Time taken: 0.661 seconds


7、进入http://master:50070/查看partition_table分区是否成功



1 0