编写yarn应用
来源:互联网 发布:淘宝网活动 编辑:程序博客网 时间:2024/06/11 10:35
目的:
本文在high level中描述了在YARN中实现应用的方法。
理念和流程
大概的理念是,应用提交client提交一项应用给yarn的Resource Manager。这可以通过建立一个yarn client来实现。yarnclient启动之后,client会建立应用,准备包含application master的container的应用,然后提交应用。你需要提供给本地文件/jar包的信息细节,以便让你的应用来运行,实际上需要运行的命令(命令行+变量),还有OS环境变量等等。
大致上,你需要给你的application master提供linux进程的信息。
YARN的resource manager会发布application master在分配的container中。application master和YARN cluster交互,处理应用的执行。她执行同步的特性,应用发布的时候,application master的主要认为是:和RM通信,分配未来container的资源,和NM交互来发布应用的container。任务可以同时通过AMRMclientAsync来同步,事件处理方法特定一种事件处理器-AMRMClientAsync.CallbackHandler。事件处理器需要明确地设定给client。task bcontainer分配需要发布一个可运行的对象,发布container。在发布container的时候,AM需要指定containerlaunchcontext,来说明发布的信息比如命令行,环境变量等等。
在应用的执行过程中,AM会通过NMClientAsync和NM通信。所有的container时间都有AMCLIENT来处理。传统的回收处理机制,处理client的启动,停止,状态更新和运行错误。AM也会向RM提交执行状况,通过getProgress方法。
我们现在讨论的是异步client,她有更简单使用率的优势,我们本文主要介绍异步client。
除了异步client,特定的workflow也有同步的版本(ARMClient和NMClient)。异步client具有更简单的用法,所以被推荐使用。这篇文章主要介绍异步client。
YARN应用的三个主要协议依然是适用的:application client protocol,application master protocol,container management protocol。这三个client共同为yarn应用提供简单地编程模型。
在很罕有的环境下,码农可能想要直接使用这三个协议来实现一个应用。然而这样的举动不再被鼓励。
编写一个简单的yarn应用:
client需要做的第一步是初始化并且启动一个yarn client
代码如下:
YarnClient yarnClient = YarnClient.createYarnClient(); yarnClient.init(conf); yarnClient.start();
一旦client建立之后,client需要创建一个应用,得到应用的ID
代码:
YarnClientApplication app = yarnClient.createApplication(); GetNewApplicationResponse appResponse = app.getNewApplicationResponse();
yarn client application对于新建一个应用的回应还包括集群最多/最少容量的资源信息。这是为了确保在application master发布的时候,你可以特殊设定container的信息。了解更多信息,请查看GetNewApplicationResponse。
client的关键任务在于建立application submission context,定义了RM发布AM需要了解的信息。client需要设定如下信息:应用的id,名字,队列,优先级,应用提交队列,提交的优先级,提交应用的用户。
containerLaunchContext:信息定义了AM发布和运行的container,定义了要运行比如本地资源,环境设定和命令的应用所需要的所有信息。
代码如下:
// set the application submission contextApplicationSubmissionContext appContext = app.getApplicationSubmissionContext();ApplicationId appId = appContext.getApplicationId();appContext.setKeepContainersAcrossApplicationAttempts(keepContainers);appContext.setApplicationName(appName);// set local resources for the application master// local files or archives as needed// In this scenario, the jar file for the application master is part of the local resourcesMap<String, LocalResource> localResources = new HashMap<String, LocalResource>();LOG.info("Copy App Master jar from local filesystem and add to local environment");// Copy the application master jar to the filesystem// Create a local resource to point to the destination jar pathFileSystem fs = FileSystem.get(conf);addToLocalResources(fs, appMasterJar, appMasterJarPath, appId.toString(), localResources, null);// Set the log4j properties if neededif (!log4jPropFile.isEmpty()) { addToLocalResources(fs, log4jPropFile, log4jPath, appId.toString(), localResources, null);}// The shell script has to be made available on the final container(s)// where it will be executed.// To do this, we need to first copy into the filesystem that is visible// to the yarn framework.// We do not need to set this as a local resource for the application// master as the application master does not need it.String hdfsShellScriptLocation = "";long hdfsShellScriptLen = 0;long hdfsShellScriptTimestamp = 0;if (!shellScriptPath.isEmpty()) { Path shellSrc = new Path(shellScriptPath); String shellPathSuffix = appName + "/" + appId.toString() + "/" + SCRIPT_PATH; Path shellDst = new Path(fs.getHomeDirectory(), shellPathSuffix); fs.copyFromLocalFile(false, true, shellSrc, shellDst); hdfsShellScriptLocation = shellDst.toUri().toString(); FileStatus shellFileStatus = fs.getFileStatus(shellDst); hdfsShellScriptLen = shellFileStatus.getLen(); hdfsShellScriptTimestamp = shellFileStatus.getModificationTime();}if (!shellCommand.isEmpty()) { addToLocalResources(fs, null, shellCommandPath, appId.toString(), localResources, shellCommand);}if (shellArgs.length > 0) { addToLocalResources(fs, null, shellArgsPath, appId.toString(), localResources, StringUtils.join(shellArgs, " "));}// Set the env variables to be setup in the env where the application master will be runLOG.info("Set the environment for the application master");Map<String, String> env = new HashMap<String, String>();// put location of shell script into env// using the env info, the application master will create the correct local resource for the// eventual containers that will be launched to execute the shell scriptsenv.put(DSConstants.DISTRIBUTEDSHELLSCRIPTLOCATION, hdfsShellScriptLocation);env.put(DSConstants.DISTRIBUTEDSHELLSCRIPTTIMESTAMP, Long.toString(hdfsShellScriptTimestamp));env.put(DSConstants.DISTRIBUTEDSHELLSCRIPTLEN, Long.toString(hdfsShellScriptLen));// Add AppMaster.jar location to classpath// At some point we should not be required to add// the hadoop specific classpaths to the env.// It should be provided out of the box.// For now setting all required classpaths including// the classpath to "." for the application jarStringBuilder classPathEnv = new StringBuilder(Environment.CLASSPATH.$$()) .append(ApplicationConstants.CLASS_PATH_SEPARATOR).append("./*");for (String c : conf.getStrings( YarnConfiguration.YARN_APPLICATION_CLASSPATH, YarnConfiguration.DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH)) { classPathEnv.append(ApplicationConstants.CLASS_PATH_SEPARATOR); classPathEnv.append(c.trim());}classPathEnv.append(ApplicationConstants.CLASS_PATH_SEPARATOR).append( "./log4j.properties");// Set the necessary command to execute the application masterVector<CharSequence> vargs = new Vector<CharSequence>(30);// Set java executable commandLOG.info("Setting up app master command");vargs.add(Environment.JAVA_HOME.$$() + "/bin/java");// Set Xmx based on am memory sizevargs.add("-Xmx" + amMemory + "m");// Set class namevargs.add(appMasterMainClass);// Set params for Application Mastervargs.add("--container_memory " + String.valueOf(containerMemory));vargs.add("--container_vcores " + String.valueOf(containerVirtualCores));vargs.add("--num_containers " + String.valueOf(numContainers));vargs.add("--priority " + String.valueOf(shellCmdPriority));for (Map.Entry<String, String> entry : shellEnv.entrySet()) { vargs.add("--shell_env " + entry.getKey() + "=" + entry.getValue());}if (debugFlag) { vargs.add("--debug");}vargs.add("1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/AppMaster.stdout");vargs.add("2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/AppMaster.stderr");// Get final commmandStringBuilder command = new StringBuilder();for (CharSequence str : vargs) { command.append(str).append(" ");}LOG.info("Completed setting up app master command " + command.toString());List<String> commands = new ArrayList<String>();commands.add(command.toString());// Set up the container launch context for the application masterContainerLaunchContext amContainer = ContainerLaunchContext.newInstance( localResources, env, commands, null, null, null);// Set up resource type requirements// For now, both memory and vcores are supported, so we set memory and// vcores requirementsResource capability = Resource.newInstance(amMemory, amVCores);appContext.setResource(capability);// Service data is a binary blob that can be passed to the application// Not needed in this scenario// amContainer.setServiceData(serviceData);// Setup security tokensif (UserGroupInformation.isSecurityEnabled()) { // Note: Credentials class is marked as LimitedPrivate for HDFS and MapReduce Credentials credentials = new Credentials(); String tokenRenewer = conf.get(YarnConfiguration.RM_PRINCIPAL); if (tokenRenewer == null | | tokenRenewer.length() == 0) { throw new IOException( "Can't get Master Kerberos principal for the RM to use as renewer"); } // For now, only getting tokens for the default file-system. final Token<?> tokens[] = fs.addDelegationTokens(tokenRenewer, credentials); if (tokens != null) { for (Token<?> token : tokens) { LOG.info("Got dt for " + fs.getUri() + "; " + token); } } DataOutputBuffer dob = new DataOutputBuffer(); credentials.writeTokenStorageToStream(dob); ByteBuffer fsTokens = ByteBuffer.wrap(dob.getData(), 0, dob.getLength()); amContainer.setTokens(fsTokens);}appContext.setAMContainerSpec(amContainer);
一旦建立过程完成之后,client就可以提交应用(其中包括队列和优先级)
// Set the priority for the application masterPriority pri = Priority.newInstance(amPriority);appContext.setPriority(pri);// Set the queue to which this application is to be submitted in the RMappContext.setQueue(amQueue);// Submit the application to the applications manager// SubmitApplicationResponse submitResp = applicationsManager.submitApplication(appRequest);yarnClient.submitApplication(appContext);
在这种程度上,RM会接受应用,并且在后台中进行分配特殊需求的container,最终在分配的container中建立和发布Application master
client监控任务有很多方法
她可以喝RM进行通信,通过getApplicationReport方法来请求应用的报告。
代码:
// Get application report for the appId we are interested inApplicationReport report = yarnClient.getApplicationReport(appId);
RM收到的应用报告包括以下内容:
(我怎么感觉这根linux下的进程差不多啊,大概都一样)
1:大概的应用信息:id,队列,用户,启动时间
2:application master的信息:AM运行的host,client监听请求的端口,client和application master通信使用的token
3:应用跟踪信息:如果应用支持某种形式上的进度跟踪,他可以设定一个追中的URL,通过ApplicationReport's的getTrackingUrl方法。
4:应用状态:RM可以通过application Report的getYarnApplicationState方法来查看应用状态。如果yarnapplicationstate被设定中介,client回想getFinalApplicationStatus来检查应用实际的状态。
5:如果AM支持,client可以直接查询AM来获得app进度。也可以通过URL状态来获得。
在某种特定的情况下,应用占用太多资源,client会想要停止应用。yarnclient支持killapplication的call,来让client发送停止信号给AM通过RM。
yarnClient.killApplication(appId);
编写一个ApplicationMaster
AM是任务实际的拥有者,RM发布AM,clien提供需要的信息和资源。
AM发布的时候,基于多租户特性,她的container会和其他的container共享一个物理机器,所以没有单独监听的端口
AM启动的时候,很多参数都是可用的,包括containerID,应用提交时间和NM宿主机器的信息。(application constants的参数)
AM和RM交互需要通过一个applicationattemptid,她在container的ID中报关。
代码如下:
Map<String, String> envs = System.getenv();String containerIdString = envs.get(ApplicationConstants.AM_CONTAINER_ID_ENV);if (containerIdString == null) { // container id should always be set in the env by the framework throw new IllegalArgumentException( "ContainerId not set in the environment");}ContainerId containerId = ConverterUtils.toContainerId(containerIdString);ApplicationAttemptId appAttemptID = containerId.getApplicationAttemptId();
一旦AM初始化成功之后,我们可以启动两个client,一个负责RM,一个负责NM,我们使用传统的事件处理器来建立他们,后文细讲。
代码如下:
AMRMClientAsync.CallbackHandler allocListener = new RMCallbackHandler(); amRMClient = AMRMClientAsync.createAMRMClientAsync(1000, allocListener); amRMClient.init(conf); amRMClient.start(); containerListener = createNMCallbackHandler(); nmClientAsync = new NMClientAsyncImpl(containerListener); nmClientAsync.init(conf); nmClientAsync.start();
AM会想RM发送心跳来证明正常工作。过期的时间参数在 YarnConfiguration.RM_AM_EXPIRY_INTERVAL_MS中来设定。AM需要向RM注册自己来启动心跳。
代码如下:
// Register self with ResourceManager// This will start heartbeating to the RMappMasterHostname = NetUtils.getHostname();RegisterApplicationMasterResponse response = amRMClient .registerApplicationMaster(appMasterHostname, appMasterRpcPort, appMasterTrackingUrl);
注册之后,如果包含最大资源容量,可以检查一下应用的请求
// Dump out information about cluster capability as seen by the// resource managerint maxMem = response.getMaximumResourceCapability().getMemory();LOG.info("Max mem capabililty of resources in this cluster " + maxMem);int maxVCores = response.getMaximumResourceCapability().getVirtualCores();LOG.info("Max vcores capabililty of resources in this cluster " + maxVCores);// A resource ask cannot exceed the max.if (containerMemory > maxMem) { LOG.info("Container memory specified above max threshold of cluster." + " Using max value." + ", specified=" + containerMemory + ", max=" + maxMem); containerMemory = maxMem;}if (containerVirtualCores > maxVCores) { LOG.info("Container virtual cores specified above max threshold of cluster." + " Using max value." + ", specified=" + containerVirtualCores + ", max=" + maxVCores); containerVirtualCores = maxVCores;}List<Container> previousAMRunningContainers = response.getContainersFromPreviousAttempts();LOG.info("Received " + previousAMRunningContainers.size() + " previous AM's running containers on AM registration.");
基于任务需要,AM可以申请一系列的container来处理任务,我们可以计算container数量,然后请求这些container
List<Container> previousAMRunningContainers = response.getContainersFromPreviousAttempts();List<Container> previousAMRunningContainers = response.getContainersFromPreviousAttempts();LOG.info("Received " + previousAMRunningContainers.size() + " previous AM's running containers on AM registration.");int numTotalContainersToRequest = numTotalContainers - previousAMRunningContainers.size();// Setup ask for containers from RM// Send request for containers to RM// Until we get our fully allocated quota, we keep on polling RM for// containers// Keep looping until all the containers are launched and shell script// executed on them ( regardless of success/failure).for (int i = 0; i < numTotalContainersToRequest; ++i) { ContainerRequest containerAsk = setupContainerAskForRM(); amRMClient.addContainerRequest(containerAsk);}
在setupContainerAskForm()中,一下的东西需要设置:
1:资源容量,YARN现在的资源需求需要制定内存,要小于最大内存,
2:优先级,一当申请很多container时,AM需要定义优先级。比如说:mapreduce AM会对map任务设置一个较高的优先级,对reduce设定一个较低的优先级
代码如下:
private ContainerRequest setupContainerAskForRM() { // setup requirements for hosts // using * as any host will do for the distributed shell app // set the priority for the request Priority pri = Priority.newInstance(requestPriority); // Set up resource type requirements // For now, memory and CPU are supported so we set memory and cpu requirements Resource capability = Resource.newInstance(containerMemory, containerVirtualCores); ContainerRequest request = new ContainerRequest(capability, null, null, pri); LOG.info("Requested container ask: " + request.toString()); return request;}
application manager发送了container分配请求之后,container会异步发布,通过AMRMClientAsync client。处理器会实现AMRMClientAsync.CallbackHandler接口。
当container分配之后,处理器会设置一个线程,启动代码来发布container,我们使用LaunchContainerRunnable 来证明。
代码如下:
@Overridepublic void onContainersAllocated(List<Container> allocatedContainers) { LOG.info("Got response from RM for container ask, allocatedCnt=" + allocatedContainers.size()); numAllocatedContainers.addAndGet(allocatedContainers.size()); for (Container allocatedContainer : allocatedContainers) { LaunchContainerRunnable runnableLaunchContainer = new LaunchContainerRunnable(allocatedContainer, containerListener); Thread launchThread = new Thread(runnableLaunchContainer); // launch and start the container on a separate thread to keep // the main thread unblocked // as all containers may not be allocated at one go. launchThreads.add(launchThread); launchThread.start(); }}
心跳方面,时间处理器会报告应用的进度
代码:
@Overridepublic float getProgress() { // set progress to deliver to RM on next heartbeat float progress = (float) numCompletedContainers.get() / numTotalContainers; return progress;}
container发布线程会在NM上发布container,之后,她需要执行一个类似的过程,设定containerLaunchContext,来处理在分配的container要运行的最终任务。一旦ContainerLaunchContext定义之后,AM会通过NMClientAsync来启动她。
代码如下:
// Set the necessary command to execute on the allocated containerVector<CharSequence> vargs = new Vector<CharSequence>(5);// Set executable commandvargs.add(shellCommand);// Set shell script pathif (!scriptPath.isEmpty()) { vargs.add(Shell.WINDOWS ? ExecBatScripStringtPath : ExecShellStringPath);}// Set args for the shell command if anyvargs.add(shellArgs);// Add log redirect paramsvargs.add("1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout");vargs.add("2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr");// Get final commmandStringBuilder command = new StringBuilder();for (CharSequence str : vargs) { command.append(str).append(" ");}List<String> commands = new ArrayList<String>();commands.add(command.toString());// Set up ContainerLaunchContext, setting local resource, environment,// command and token for constructor.// Note for tokens: Set up tokens for the container too. Today, for normal// shell commands, the container in distribute-shell doesn't need any// tokens. We are populating them mainly for NodeManagers to be able to// download anyfiles in the distributed file-system. The tokens are// otherwise also useful in cases, for e.g., when one is running a// "hadoop dfs" command inside the distributed shell.ContainerLaunchContext ctx = ContainerLaunchContext.newInstance( localResources, shellEnv, commands, null, allTokens.duplicate(), null);containerListener.addContainer(container.getId(), container);nmClientAsync.startContainerAsync(container, ctx);
NMclientAsync对象,和事件处理器共同处理container的事件,包括container启动停止,更新。
AM决定工作完成之后,需要和AM-RM client 解除注册关系,然后停止client。
代码如下:
try { amRMClient.unregisterApplicationMaster(appStatus, appMessage, null);} catch (YarnException ex) { LOG.error("Failed to unregister application", ex);} catch (IOException e) { LOG.error("Failed to unregister application", e);}amRMClient.stop();
- 编写yarn应用
- 如何编写YARN应用程序
- 如何编写YARN应用程序
- 如何编写YARN应用程序
- 如何编写YARN应用程序
- 如何编写YARN应用程序
- 如何编写YARN应用程序
- 如何编写YARN应用程序
- YARN应用实例
- YARN应用开发流程
- yarn-web应用代理
- YARN应用开发流程
- YARN应用生命周期
- Yarn 应用开发流程
- 通过编写一个YARN Application 来了解Hadoop YARN
- Hadoop -YARN 应用程序设计概述
- spark on yarn 应用笔记
- hadoop YARN分布式程序的编写
- Android 表情功能的完整处理方案
- 随机数
- JS中Null与Undefined的区别
- js 弹出对话框3种方式
- eclipse Formatter配置文件生成过程
- 编写yarn应用
- intell IDE初始用
- 3种解耦方式
- Codeforces 558B Amr and The Large Array(hash)
- Web开发技术十年发展历程
- C++ 语法学习笔记
- 你觉得测试人员的主要工作职责是什么?
- XMLHttpRequest对象的属性和方法
- 局域网共享文件