spark on yarn日志切割
来源:互联网 发布:js获取所有id 像 编辑:程序博客网 时间:2024/06/05 10:16
由于spark on yarn的日志会在stderr里面一直追加,而且streaming又是长应用,那么stderr会一直增长。解决办法就是把stderr按天切割,这样我们就可以删除之前的日志了。
1. 首先,把集群中的hadoop的log4j文件/etc/hadoop/conf/log4j.properties添加如下两行:
log4j.additivity.com.kingsoft = false com.kingsoft是项目的包名
log4j.additivity.org.apache.spark = false
然后把log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n的%c{2}改成%c{30}
最终的结果如下:
log4j.rootLogger=${root.logger}
root.logger=INFO,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{30}: %m%n
log4j.additivity.com.kingsoft = false
log4j.additivity.org.apache.spark = false
2. 编写一个 log4j.properties 文件,内容如下:
log4j.rootCategory=INFO, click_all
# Settings to quiet third party logs that are too verbose
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.appender.click_all = org.apache.log4j.DailyRollingFileAppender
log4j.appender.click_all.File = /sparklogs/click_all.log
log4j.appender.click_all.DatePattern = '.'yyyy-MM-dd_HH
log4j.appender.click_all.Append = true
log4j.appender.click_all.Threshold = INFO
log4j.appender.click_all.layout = org.apache.log4j.PatternLayout
log4j.appender.click_all.layout.ConversionPattern = %d{yy/MM/dd HH:mm:ss} %p %c{30}: %m%n
log4j.additivity.com._58city = false
log4j.logger.com._58city = INFO, click
log4j.appender.click = org.apache.log4j.DailyRollingFileAppender
log4j.appender.click.File = /sparklogs/click.log
log4j.appender.click.DatePattern = '.'yyyy-MM-dd_HH
log4j.appender.click.Append = true
log4j.appender.click.Threshold = INFO
log4j.appender.click.layout = org.apache.log4j.PatternLayout
log4j.appender.click.layout.ConversionPattern = %d{yy/MM/dd HH:mm:ss} %p %c{30}: %m%n
- spark on yarn日志切割
- Spark on YARN配置日志Web UI
- Spark on Yarn部署
- Spark on Yarn
- spark on yarn
- spark on yarn
- Spark on Yarn简介
- spark on yarn
- Spark on YARN 部署
- spark on yarn 配置
- spark on yarn
- spark on yarn 安装
- Spark on Yarn 图
- Spark on yarn
- 源码-Spark on Yarn
- Launching Spark on YARN
- Spark On Yarn 知识点
- Spark 2.0 On Yarn
- 黑马程序员_JavaSE的File和IO流
- matlab常用随机函数randperm
- java transient 详解
- java读取jar包中文件
- arcgis中ArcMAP下栅格图象矢量化的步骤
- spark on yarn日志切割
- 获取已安装或未安装的apk签名
- 反射读取annotation
- PHP安装MongoDB扩展驱动
- matlab矩阵复制函数:repmat
- jquery datatables插件兑现表头固定内容可滚动列表
- IOS保存数据的4种方式
- 读懂这些数据,让你的简历有回应
- 熟练使用IText