[Java代码] [YARN] MRAppMaster心跳原理
来源:互联网 发布:最优化算法 pdf 编辑:程序博客网 时间:2024/05/22 15:10
最近集群遇到一个问题,就是集群在跑任务的时候,AM会超时10min而被KILL,但任务重跑则成功,问题是随机的出现的,所以初步怀疑是因为AM心跳汇报出现问题或则RM因为繁忙hang住,AM因为某些机制导致等待10min不汇报心跳,所以我们还是先了解,AM是如何向RM汇报心跳的。
在MRAppMaster中,ContainerAllocatorRouter负责向RM申请资源(发送心跳)
RMContainerAllocator其最终父类是RMCommunicator,它实现了RMHeartbeatHandler接口
- public interface RMHeartbeatHandler {http://www.kmrlyy.com/fujianyan/33454.html
- long getLastHeartbeatTime(); // 获取上一次心跳的时间
- void runOnNextHeartbeat(Runnable callback); // 回调注册到callback队列的callback函数
- }http://www.nvzi91.cn/niaodaoyan/29938.html
复制代码每一次心跳回来,都会执行一次注册在heartbeatCallbacks中的回调函数:
- allocatorThread = new Thread(new Runnable() {
- @Override
- public void run() {
- while (!stopped.get() && !Thread.currentThread().isInterrupted()) {
- ......http://www.kmrlyy.com/fujianyan/33455.html
- heartbeat();
- lastHeartbeatTime = context.getClock().getTime();// 记录上一次心跳时间
- executeHeartbeatCallbacks(); // 执行回调函数
- ....http://www.nvzi91.cn/yindaoyan/29939.html
- });http://www.nvzi91.cn/zigongjiliu/29942.html
- http://www.kmrlyy.com/gongjingmilan/33456.html
复制代码RMCommunicator类中:
- private void executeHeartbeatCallbacks() {
- Runnable callback = null;
- while ((callback = heartbeatCallbacks.poll()) != null) {
- callback.run();
- }
- }http://www.nvzi91.cn/yindaoyan/29940.html
- http://www.kmrlyy.com/penqiangyan/33457.html
复制代码在RMCommunicator启动时,首先会向RM注册,把自己的host和port告诉RM,然后在启动一条线程(startAllocatorThread)定期的调用RMContainerAllocator中实现的heartbeat方法(向RM申请资源,定期汇报信息,告诉RM自己还活着)。
AM初始化同时也会初始化RMCommunicator:
- protected void serviceStart() throws Exception {
- scheduler= createSchedulerProxy(); // 获取RM的代理
- register(); // 注册
- startAllocatorThread(); // 心跳线程
- ....
- }http://www.nvzi91.cn/luanchaonanzhong/29941.html
- http://www.kmrlyy.com/niaodaoyan/33458.html
复制代码AM的ContainerAllocatorRouter事件处理流程如下图:
注册流程:
调用RMCommunicator远程调用ApplicationMasterService的registerApplicationMaster方法,设置维护responseId,然后把它加入AMLivelinessMonitor中,并使用map记录时间,用来监控AM是否因为长时间没有心跳而超时,如果AM长时间没有心跳信息更新,RM就会通知NodeManager把AM移除。
心跳线程:
在发送心跳的过程中,即也是获取资源的过程
- @Overridehttp://m.nvzi91.cn/penqiangyan/29351.html
- protected synchronized void heartbeat() throws Exception {
- scheduleStats.updateAndLogIfChanged("Before Scheduling: ");
- List<Container> allocatedContainers = getResources();// 重要的方法
- if (allocatedContainers.size() > 0) {
- scheduledRequests.assign(allocatedContainers);
- }
- ......
- }
复制代码获取资源的过程:
- private List<Container> getResources() throws Exception {
- ...
- response = makeRemoteRequest(); // 和RM进行交互
- ...
- // 优先处理RM发送过来的命令
- if (response.getAMCommand() != null) {
- switch(response.getAMCommand()) {
- case AM_RESYNC:
- case AM_SHUTDOWN:
- eventHandler.handle(new JobEvent(this.getJob().getID(),
- JobEventType.JOB_AM_REBOOT));
- throw new YarnRuntimeException("Resource Manager doesn't recognize AttemptId: " +
- this.getContext().getApplicationID());
- default:
- ....
- }http://m.nvzi91.cn/zigongai/29352.html
- // 等等一系列处理
- }
- }
复制代码构建请求:
- protected AllocateResponse makeRemoteRequest() throws IOException {
- AllocateRequest allocateRequest =
- AllocateRequest.newInstance(lastResponseID,
- super.getApplicationProgress(), new ArrayList<ResourceRequest>(ask),
- new ArrayList<ContainerId>(release), blacklistRequest);
- AllocateResponse allocateResponse;
- allocateResponse = scheduler.allocate(allocateRequest); // RPC调用ApplicationMasterService的allocate方法
- .....
- }http://m.nvzi91.cn/jiankang/29353.html
复制代码每一次心跳的调用都会刷新AMLivelinessMonitor的时间,代表AM还活着
而且我们通过代码可以看出,资源请求被封装为一个ask,即一个ResourceRequest的ArrayList的资源列表 例如:
- priority:20 host:host9 capability:<memory:2048, vCores:1>
- priority:20 host:host2 capability:<memory:2048, vCores:1>
- priority:20 host:host10 capability:<memory:2048, vCores:1>
- priority:20 host:/rack/rack3203 capability:<memory:2048, vCores:1>
- priority:20 host:/rack/rack3202 capability:<memory:2048, vCores:1>
- priority:20 host:* capability:<memory:2048, vCores:1>
- www.kmrlyy.com
复制代码然而,ask是如何被构造的呢?
RMContainerAllocator中的addMap,addReduce,assign方法中对ask的数据内容进行了修改
- addContainerReq --> addResourceRequest --> addResourceRequestToAsk;
- www.nvzi91.cn
复制代码通过在代码自己添加日志可以看出,资源会被分为local,rack,和any级别去申请资源
最终变为一个ask list发送到RM上:
- ask Capability:<memory:2048, vCores:1> ResourceName:* NumContainers:384 Priority:20 RelaxLocality:true
- ask Capability:<memory:2048, vCores:1> ResourceName:/rack/rack3201 NumContainers:227 Priority:20 RelaxLocality:true
- ask Capability:<memory:2048, vCores:1> ResourceName:/rack/rack3202 NumContainers:231 Priority:20 RelaxLocality:true
- ask Capability:<memory:2048, vCores:1> ResourceName:/rack/rack3203 NumContainers:152 Priority:20 RelaxLocality:true
- ask Capability:<memory:2048, vCores:1> ResourceName:/rack/rack3204 NumContainers:158 Priority:20 RelaxLocality:true
- ask Capability:<memory:2048, vCores:1> ResourceName:host1 NumContainers:46 Priority:20 RelaxLocality:true
- ask Capability:<memory:2048, vCores:1> ResourceName:host5 NumContainers:52 Priority:20 RelaxLocality:true
- ask Capability:<memory:2048, vCores:1> ResourceName:host6 NumContainers:38 Priority:20 RelaxLocality:true
- m.nvzi91.cn
复制代码类似日志为:
- getResources() for application_1438330253091_0004: ask=29 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:0, vCores:0> knownNMs=24
0 0