使用spark streaming报错ERROR DFSClient: Failed to close inode xxxx
来源:互联网 发布:ubuntu快捷键 编辑:程序博客网 时间:2024/06/05 20:53
转载自:http://blog.csdn.net/xiaolixiaoyi/article/details/45875101
好几个Spark streaming的程序同时运行,发现spark报出了如下错误:
ERROR DFSClient: Failed to close inode 23828
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /spark/checkpoint/temp (inode 23828): File does not exist. Holder DFSClient_NONMAPREDUCE_-353217694_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3516)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3313)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3169)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy13.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349)
at org.apache.Hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.Java:588)
查了好久,发现貌似是因为几个程序使用了相同的路径存放checkpoint,让他们使用不同的路径即可。
- 使用spark streaming报错ERROR DFSClient: Failed to close inode xxxx
- 使用spark streaming报错ERROR DFSClient: Failed to close inode xxxx
- hdfs.DFSClient: Failed to close file 解决办法
- spark报错:ERROR SparkUI: Failed to bind SparkUI,解决办法
- HDFS append操作: DataStreamer Exception <ERROR>Failed to close inode 347753
- mysql-创建用户报错ERROR 1396 (HY000): Operation CREATE USER failed for 'XXXX'@'XXXX'
- android studio 报Error:failed to find Build Tools revision XXXX
- error: failed to push some refs to 'git@x'x'x:xxxx/xxx.git' git提交本地分支到远程报错
- iOS 报错:process launch failed: failed to get the task for process XXXX
- SSH报错之 hibernate.LazyInitializationException: failed to lazily initialize a collection of role: XXXX
- Shiro servlet启动报错 Failed to initialize component [org.apache.catalina.webresources.JarResource@xxxx
- Failed to sync Gradle project'xxxx' error:Re-downlod....
- composer报错:Failed to decode response: zlib_decode(): data error
- 报错信息: Error:Failed to crunch file
- 数据库语法报错:ERROR 1064 (42000):for the right syntax to use near 'XXXX'
- Spark Streaming 报错:kafka.cluster.BrokerEndPoint cannot be cast to kafka.cluster.Broker
- genymotion报错:Failed to open a session for the virtual machine,Unable to load R3 module xxxx/VBoxDD.D
- gitlab push时报错error:failed to push some refs to 'git@xxx.xx.xx.xxx:xxxx/xxx.git'
- 进程管理
- 匿名函数
- 视频编码(H264概述)
- Nginx负载均衡(架构之路)
- hdu-2072-单词数(字典树)
- 使用spark streaming报错ERROR DFSClient: Failed to close inode xxxx
- JavaScript用穷举写百钱买百鸡问题
- 抽象数据类型(ADT)
- 4. Median of Two Sorted Arrays
- [python3.6 flask web学习]web服务器基本结构
- recyclere 点击事件
- Bootstrap 写的webview 在安卓里图片失真,变形
- java数据库初探
- 队列ADT