spark 中写hdfs的异常解决
来源:互联网 发布:淘宝怎么添加好友数字 编辑:程序博客网 时间:2024/06/07 09:47
报错信息:
ERROR DFSClient: Failed to close inode 16604
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.15.201:50010,DS-00e134aa-a7c3-44be-b1e2-0c75ccb7c4bc,DISK]], original=[DatanodeInfoWithStorage[192.168.15.201:50010,DS-00e134aa-a7c3-44be-b1e2-0c75ccb7c4bc,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:925)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:988)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1156)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:454)
按照网上说明该了hdfs-site.xml 设置了
- dfs.client.block.write.replace-datanode-on-failure.policy : NEVER
- dfs.client.block.write.replace-datanode-on-failure.enable : true
我是spark-shell 里面执行的 ,最后在scala环境中执行下面两句搞定
scala> System.setProperty("dfs.client.block.write.replace-datanode-on-failure.enable", "true")
res1: String = true
scala> System.setProperty("dfs.client.block.write.replace-datanode-on-failure.policy","NEVER")
res3: String = never
ps:我在so。com里面搜索出来的,baidu里面冗余无效信息太多
- spark 中写hdfs的异常解决
- [解决]Spark运行中java.net.UnknownHostException: nameservice1的异常
- [解决]Spark运行中java.net.UnknownHostException: nameservice1的异常
- spark读取hdfs异常记录
- spark streaming 写hdfs问题
- spark读写hdfs后出现的异常错误
- python spark中parquet文件写到hdfs,同时避免太多的小文件(block小文件合并)
- HDFS中zookeeper的异常分析
- spark读取hdfs数据本地性异常
- spark读取hdfs数据本地性异常
- hdfs块丢失导致的异常问题排查解决
- spark 中删除HDFS文件
- 如何在spark中删除hdfs的某个文件夹
- Spark在Hadoop的HDFS中读取数据
- 写hdfs的API
- 异常 spark 运行 未解决
- 向HDFS 中写数据
- 如何解决spark写hive慢的问题
- Static在C和C++中的用法和区别
- Java 多线程下载
- Js 小知识
- Windows socket之WSAEventSelect模型
- 图论:HDU2544-最短路(最全、最经典的最短路入门及小结)
- spark 中写hdfs的异常解决
- 剑指offer题19
- Linux常用开发命令
- 基于Spark streaming的SQL服务实时自动化运维
- 一个电脑安装两个版本的jdk
- AutoLoadCache 4.11 发布,增加刷新缓存及续租缓存功能
- python opencv入门 滚动条控制调色板(5)
- 51nod 1287 加农炮 思维
- 微信小程序弹出对话框