Nodemanager in unhealthy state
来源:互联网 发布:黄金坑选股公式源码 编辑:程序博客网 时间:2024/06/08 18:45
Unhealthy Node local-dirs and log-dirs are bad”
ambari的系统里出现这个告警,但是nodemanager也没有挂掉,所有的mapreduce任务都不能进行,处于卡主状态,有的日志还出现这个
2017-02-20 15:45:07,920 ERROR [WALProcedureStoreSyncThread] wal.WALProcedureStore: Unable to close the stream
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/hbase/data/MasterProcWALs/state-00000000000000000058.log (inode 45740): File does not exist. Holder DFSClient_NONMAPREDUCE_-96565186_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3597)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3400)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3256)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:677)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:213)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:485)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
yarn all application 中 显示任务已经accept,但是没有执行,没有分配job url,原因就是磁盘的空间太满了,跟yarn-site.xml中的以下配置有关
<property> <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name> <value>98.5</value></property>
try adding the property yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage to yarn-site.xml. This property specifies the maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0.你可以清理磁盘空间,或者你可以暂时的吧这个使用率调大,但是这不是解决问题的根本,最好是清理磁盘空间或扩展
- Nodemanager in unhealthy state
- Error starting NodeManager-Service NodeManager failed in state INITED; cause: java.lang.Null
- Patterns in SOME –State
- Proxy and State in Python
- prepareAsync called in state 1
- Mobility Management in Connected State
- 状态(state) in Java
- State Machine Design in C++
- hadoop Unhealthy Nodes
- hadoop Unhealthy Nodes问题解决
- hadoop Unhealthy Nodes问题解决
- Hadoop集群nodes unhealthy解决方法
- Session State in ASP.NET Web Services
- State Machine Workflows in Windows Workflow Foundation
- Missed Signals : state-dependent in concurrent
- [Study] Optimizing DP in state and decision
- Error: MediaPlayer start called in state 64
- The State Of NoSql in 2012
- SVN客户端的使用
- 《阿里巴巴Java开发手册(正式版)》--安全规约
- Hive扩展功能(八)--表的索引
- Android官方的下拉刷新SwipeRefreshLayout
- Go如何保证gorountine执行完毕后继续执行
- Nodemanager in unhealthy state
- SVN客户端的安装
- 欢迎使用CSDN-markdown编辑器
- 4445: [Scoi2015]小凸想跑步
- GDKOI划水记
- 通过ajax请求springmvc,将值显示在select下拉框中
- SVN(服务器)安装
- js中的作用域和执行上下文的区别
- ReentrantLock实现原理