hive错误记录

来源：互联网发布：java数据库代码生成器编辑：程序博客网时间：2024/05/22 18:22

此文来记录hive使用时的错误，不定时更新

Failed with exception java.io.IOException:org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1167307331-192.168.6.100-1477735992829:blk_1073741922_1112 file=/project/sogou/raw/sogou.50w.utf8
用dfs上建外部表时，执行查询时报错。

解决方法:

1.1 在浏览器上http://master:50070/corrupt_files.jsp 查看webUI

NameNode ‘master:9000’

Started:Fri May 05 05:02:21 PDT 2017

Version:2.6.4, 5082c73637530b0b7e115f9625ed7fac69f937e6

Compiled:2016-02-12T09:45Z by jenkins from (detached from 5082c73)
Cluster ID:CID-00d33199-9e38-442e-852f-51d35856be87
Block Pool ID:BP-1167307331-192.168.6.100-1477735992829

Browse the filesystem

NameNode Logs

Go back to DFS home

Reported Corrupt Files

blk_1073741919 /project/sogou/ext/sogou.50w.utf8.flt
blk_1073741922 /project/sogou/raw/sogou.50w.utf8

Total: At least 2 corrupt file(s)

发现有2个 Corrupt Files

1 . 2 使用fsck 命令。HDFS支持fsck命令用以检查各种不一致。fsck用以报告各种文件问题，如block丢失或缺少block等

    [hadoop@master hive]$  hadoop fsck /

DEPRECATED: Use of this script to execute hdfs command is deprecated.    Instead use the hdfs command for it.17/05/05 05:48:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableConnecting to namenode via http://master:50070FSCK started by hadoop (auth:SIMPLE) from /192.168.6.100 for path / at Fri May 05 05:48:44 PDT 2017...................................../project/sogou/ext/sogou.50w.utf8.flt: CORRUPT blockpool BP-1167307331-192.168.6.100-1477735992829 block blk_1073741919/project/sogou/ext/sogou.50w.utf8.flt: MISSING 1 blocks of total size 64403097 B../project/sogou/raw/sogou.50w.utf8: CORRUPT blockpool BP-1167307331-192.168.6.100-1477735992829 block blk_1073741922/project/sogou/raw/sogou.50w.utf8: MISSING 1 blocks of total size 57403097 B........................................./user/hive/warehouse/userinfo/o.txt:  Under replicated BP-1167307331-192.168.6.100-1477735992829:blk_1073742062_1252. Target Replicas is 3 but found 2 replica(s).Status: CORRUPT Total size:    124244773 B Total dirs:    116 Total files:   78 Total symlinks:        0 Total blocks (validated):  73 (avg. block size 1701983 B)  ********************************  CORRUPT FILES:    2  MISSING BLOCKS:   2  MISSING SIZE:     121806194 B  CORRUPT BLOCKS:   2  ******************************** Minimally replicated blocks:   71 (97.26028 %) Over-replicated blocks:    0 (0.0 %) Under-replicated blocks:   1 (1.369863 %) Mis-replicated blocks:     0 (0.0 %) Default replication factor:    2 Average block replication: 1.9452055 Corrupt blocks:        2 Missing replicas:      1 (0.6802721 %) Number of data-nodes:      2 Number of racks:       1FSCK ended at Fri May 05 05:48:45 PDT 2017 in 54 millisecondsThe filesystem under path '/' is CORRUPT

1 . 3

发现正好有两个MISSING，说了有2个block坏了。
删除坏块 hadoop fsck -delete Corrupt Files.

删除后DFS上文件消失。需要重新上传文件。在重新建外部表，问题解决。

0 0