程序博客网 > ubuntu 界面

Spark Streaming 通过NC运行日志

来源：互联网 发布：ubuntu 界面 编辑：程序博客网 时间：2024/04/30 11:40
"F:\Program Files\Java\jdk1.7.0_15\bin\java" -Didea.launcher.port=7534 "-Didea.launcher.bin.path=F:\Program Files (x86)\JetBrains\IntelliJIDEASpark\bin" -Dfile.encoding=UTF-8 -classpath "F:\Program Files\Java\jdk1.7.0_15\jre\lib\charsets.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\deploy.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\ext\access-bridge-64.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\ext\dnsns.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\ext\jaccess.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\ext\localedata.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\ext\sunec.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\ext\sunjce_provider.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\ext\sunmscapi.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\ext\zipfs.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\javaws.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\jce.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\jfr.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\jfxrt.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\jsse.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\management-agent.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\plugin.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\resources.jar;F:\Program Files\Java\jdk1.7.0_15\jre\lib\rt.jar;D:\SparkProject\MyRpc\target\classes;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-sql_2.10\1.6.3\spark-sql_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-catalyst_2.10\1.6.3\spark-catalyst_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\scala-lang\scala-reflect\2.10.5\scala-reflect-2.10.5.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\codehaus\janino\janino\2.7.8\janino-2.7.8.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\codehaus\janino\commons-compiler\2.7.8\commons-compiler-2.7.8.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\parquet\parquet-column\1.7.0\parquet-column-1.7.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\parquet\parquet-common\1.7.0\parquet-common-1.7.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\parquet\parquet-encoding\1.7.0\parquet-encoding-1.7.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\parquet\parquet-generator\1.7.0\parquet-generator-1.7.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-codec\commons-codec\1.5\commons-codec-1.5.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\parquet\parquet-hadoop\1.7.0\parquet-hadoop-1.7.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\parquet\parquet-format\2.3.0-incubating\parquet-format-2.3.0-incubating.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\parquet\parquet-jackson\1.7.0\parquet-jackson-1.7.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\codehaus\jackson\jackson-mapper-asl\1.9.11\jackson-mapper-asl-1.9.11.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\codehaus\jackson\jackson-core-asl\1.9.11\jackson-core-asl-1.9.11.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.4.4\jackson-databind-2.4.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.4.0\jackson-annotations-2.4.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\fasterxml\jackson\core\jackson-core\2.4.4\jackson-core-2.4.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\spark-project\spark\unused\1.0.0\unused-1.0.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-streaming_2.10\1.6.3\spark-streaming_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-streaming-flume_2.10\1.6.3\spark-streaming-flume_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-streaming-flume-sink_2.10\1.6.3\spark-streaming-flume-sink_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\flume\flume-ng-core\1.6.0\flume-ng-core-1.6.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\flume\flume-ng-configuration\1.6.0\flume-ng-configuration-1.6.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-io\commons-io\2.1\commons-io-2.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-lang\commons-lang\2.5\commons-lang-2.5.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\avro\avro\1.7.4\avro-1.7.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\avro\avro-ipc\1.7.4\avro-ipc-1.7.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\velocity\velocity\1.7\velocity-1.7.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\joda-time\joda-time\2.1\joda-time-2.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\mortbay\jetty\jetty-util\6.1.26\jetty-util-6.1.26.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\mortbay\jetty\jetty\6.1.26\jetty-6.1.26.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\google\code\gson\gson\2.2.2\gson-2.2.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\mina\mina-core\2.0.4\mina-core-2.0.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\flume\flume-ng-sdk\1.6.0\flume-ng-sdk-1.6.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-streaming-kafka_2.10\1.6.3\spark-streaming-kafka_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\kafka\kafka_2.10\0.8.2.1\kafka_2.10-0.8.2.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\yammer\metrics\metrics-core\2.2.0\metrics-core-2.2.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\kafka\kafka-clients\0.8.2.1\kafka-clients-0.8.2.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\101tec\zkclient\0.3\zkclient-0.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\mysql\mysql-connector-java\5.1.38\mysql-connector-java-5.1.38.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\scala-lang\scala-library\2.10.6\scala-library-2.10.6.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-core_2.10\1.6.3\spark-core_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\avro\avro-mapred\1.7.7\avro-mapred-1.7.7-hadoop2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\avro\avro-ipc\1.7.7\avro-ipc-1.7.7-tests.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\twitter\chill_2.10\0.5.0\chill_2.10-0.5.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\esotericsoftware\kryo\kryo\2.21\kryo-2.21.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\esotericsoftware\reflectasm\reflectasm\1.07\reflectasm-1.07-shaded.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\esotericsoftware\minlog\minlog\1.2\minlog-1.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\objenesis\objenesis\1.2\objenesis-1.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\twitter\chill-java\0.5.0\chill-java-0.5.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\xbean\xbean-asm5-shaded\4.4\xbean-asm5-shaded-4.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-launcher_2.10\1.6.3\spark-launcher_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-network-common_2.10\1.6.3\spark-network-common_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-network-shuffle_2.10\1.6.3\spark-network-shuffle_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\fusesource\leveldbjni\leveldbjni-all\1.8\leveldbjni-all-1.8.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\spark\spark-unsafe_2.10\1.6.3\spark-unsafe_2.10-1.6.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\net\java\dev\jets3t\jets3t\0.7.1\jets3t-0.7.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\curator\curator-recipes\2.4.0\curator-recipes-2.4.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\curator\curator-framework\2.4.0\curator-framework-2.4.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\zookeeper\zookeeper\3.4.5\zookeeper-3.4.5.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\jline\jline\0.9.94\jline-0.9.94.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\google\guava\guava\14.0.1\guava-14.0.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\eclipse\jetty\orbit\javax.servlet\3.0.0.v201112011016\javax.servlet-3.0.0.v201112011016.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\commons\commons-lang3\3.3.2\commons-lang3-3.3.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\commons\commons-math3\3.4.1\commons-math3-3.4.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\slf4j\slf4j-api\1.7.10\slf4j-api-1.7.10.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\slf4j\jul-to-slf4j\1.7.10\jul-to-slf4j-1.7.10.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\slf4j\jcl-over-slf4j\1.7.10\jcl-over-slf4j-1.7.10.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\log4j\log4j\1.2.17\log4j-1.2.17.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\slf4j\slf4j-log4j12\1.7.10\slf4j-log4j12-1.7.10.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\ning\compress-lzf\1.0.3\compress-lzf-1.0.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\xerial\snappy\snappy-java\1.1.2.6\snappy-java-1.1.2.6.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\net\jpountz\lz4\lz4\1.3.0\lz4-1.3.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\roaringbitmap\RoaringBitmap\0.5.11\RoaringBitmap-0.5.11.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-net\commons-net\2.2\commons-net-2.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\typesafe\akka\akka-remote_2.10\2.3.11\akka-remote_2.10-2.3.11.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\typesafe\akka\akka-actor_2.10\2.3.11\akka-actor_2.10-2.3.11.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\typesafe\config\1.2.1\config-1.2.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\io\netty\netty\3.8.0.Final\netty-3.8.0.Final.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\uncommons\maths\uncommons-maths\1.2.2a\uncommons-maths-1.2.2a.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\typesafe\akka\akka-slf4j_2.10\2.3.11\akka-slf4j_2.10-2.3.11.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\json4s\json4s-jackson_2.10\3.2.10\json4s-jackson_2.10-3.2.10.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\json4s\json4s-core_2.10\3.2.10\json4s-core_2.10-3.2.10.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\json4s\json4s-ast_2.10\3.2.10\json4s-ast_2.10-3.2.10.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\scala-lang\scalap\2.10.0\scalap-2.10.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\scala-lang\scala-compiler\2.10.0\scala-compiler-2.10.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\sun\jersey\jersey-server\1.9\jersey-server-1.9.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\asm\asm\3.1\asm-3.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\sun\jersey\jersey-core\1.9\jersey-core-1.9.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\mesos\mesos\0.21.1\mesos-0.21.1-shaded-protobuf.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\io\netty\netty-all\4.0.29.Final\netty-all-4.0.29.Final.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\clearspring\analytics\stream\2.7.0\stream-2.7.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\io\dropwizard\metrics\metrics-core\3.1.2\metrics-core-3.1.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\io\dropwizard\metrics\metrics-jvm\3.1.2\metrics-jvm-3.1.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\io\dropwizard\metrics\metrics-json\3.1.2\metrics-json-3.1.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\io\dropwizard\metrics\metrics-graphite\3.1.2\metrics-graphite-3.1.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\fasterxml\jackson\module\jackson-module-scala_2.10\2.4.4\jackson-module-scala_2.10-2.4.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\thoughtworks\paranamer\paranamer\2.6\paranamer-2.6.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\ivy\ivy\2.4.0\ivy-2.4.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\oro\oro\2.0.8\oro-2.0.8.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\tachyonproject\tachyon-client\0.8.2\tachyon-client-0.8.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\tachyonproject\tachyon-underfs-hdfs\0.8.2\tachyon-underfs-hdfs-0.8.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\tachyonproject\tachyon-underfs-s3\0.8.2\tachyon-underfs-s3-0.8.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\tachyonproject\tachyon-underfs-local\0.8.2\tachyon-underfs-local-0.8.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\net\razorvine\pyrolite\4.9\pyrolite-4.9.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\net\sf\py4j\py4j\0.9\py4j-0.9.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-client\2.6.4\hadoop-client-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-common\2.6.4\hadoop-common-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\xmlenc\xmlenc\0.52\xmlenc-0.52.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-logging\commons-logging\1.1.3\commons-logging-1.1.3.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-configuration\commons-configuration\1.6\commons-configuration-1.6.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-digester\commons-digester\1.8\commons-digester-1.8.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-beanutils\commons-beanutils\1.7.0\commons-beanutils-1.7.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\commons-beanutils\commons-beanutils-core\1.8.0\commons-beanutils-core-1.8.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-auth\2.6.4\hadoop-auth-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\httpcomponents\httpclient\4.2.5\httpclient-4.2.5.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\httpcomponents\httpcore\4.2.4\httpcore-4.2.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\directory\server\apacheds-kerberos-codec\2.0.0-M15\apacheds-kerberos-codec-2.0.0-M15.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\directory\server\apacheds-i18n\2.0.0-M15\apacheds-i18n-2.0.0-M15.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\directory\api\api-asn1-api\1.0.0-M20\api-asn1-api-1.0.0-M20.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\directory\api\api-util\1.0.0-M20\api-util-1.0.0-M20.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\curator\curator-client\2.6.0\curator-client-2.6.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\htrace\htrace-core\3.0.4\htrace-core-3.0.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\tukaani\xz\1.0\xz-1.0.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-hdfs\2.6.4\hadoop-hdfs-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\xerces\xercesImpl\2.9.1\xercesImpl-2.9.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\xml-apis\xml-apis\1.3.04\xml-apis-1.3.04.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-app\2.6.4\hadoop-mapreduce-client-app-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-common\2.6.4\hadoop-mapreduce-client-common-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-yarn-client\2.6.4\hadoop-yarn-client-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-yarn-server-common\2.6.4\hadoop-yarn-server-common-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-shuffle\2.6.4\hadoop-mapreduce-client-shuffle-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-yarn-api\2.6.4\hadoop-yarn-api-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-core\2.6.4\hadoop-mapreduce-client-core-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-yarn-common\2.6.4\hadoop-yarn-common-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\javax\xml\bind\jaxb-api\2.2.2\jaxb-api-2.2.2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\javax\xml\stream\stax-api\1.0-2\stax-api-1.0-2.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\javax\activation\activation\1.1\activation-1.1.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\javax\servlet\servlet-api\2.5\servlet-api-2.5.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\com\sun\jersey\jersey-client\1.9\jersey-client-1.9.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\codehaus\jackson\jackson-jaxrs\1.9.13\jackson-jaxrs-1.9.13.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\codehaus\jackson\jackson-xc\1.9.13\jackson-xc-1.9.13.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-jobclient\2.6.4\hadoop-mapreduce-client-jobclient-2.6.4.jar;C:\Users\Administrator.WIN-20160809ARI\.m2\repository\org\apache\hadoop\hadoop-annotations\2.6.4\hadoop-annotations-2.6.4.jar;F:\Program Files (x86)\JetBrains\IntelliJIDEASpark\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain com.sdcet.WindowReduce
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/02/20 10:07:59 INFO SparkContext: Running Spark version 1.6.3
17/02/20 10:08:01 INFO SecurityManager: Changing view acls to: Administrator
17/02/20 10:08:01 INFO SecurityManager: Changing modify acls to: Administrator
17/02/20 10:08:01 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Administrator); users with modify permissions: Set(Administrator)
17/02/20 10:08:01 INFO Utils: Successfully started service 'sparkDriver' on port 49289.
17/02/20 10:08:02 INFO Slf4jLogger: Slf4jLogger started
17/02/20 10:08:02 INFO Remoting: Starting remoting
17/02/20 10:08:03 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.164.1:49303]
17/02/20 10:08:03 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 49303.
17/02/20 10:08:03 INFO SparkEnv: Registering MapOutputTracker
17/02/20 10:08:03 INFO SparkEnv: Registering BlockManagerMaster
17/02/20 10:08:03 INFO DiskBlockManager: Created local directory at C:\Users\Administrator.WIN-20160809ARI\AppData\Local\Temp\blockmgr-818523a9-ff0d-4c61-81f9-265b3107dff5
17/02/20 10:08:03 INFO MemoryStore: MemoryStore started with capacity 1807.0 MB
17/02/20 10:08:03 INFO SparkEnv: Registering OutputCommitCoordinator
17/02/20 10:08:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/02/20 10:08:03 INFO SparkUI: Started SparkUI at http://192.168.164.1:4040
17/02/20 10:08:04 INFO Executor: Starting executor ID driver on host localhost
17/02/20 10:08:04 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49322.
17/02/20 10:08:04 INFO NettyBlockTransferService: Server created on 49322
17/02/20 10:08:04 INFO BlockManagerMaster: Trying to register BlockManager
17/02/20 10:08:04 INFO BlockManagerMasterEndpoint: Registering block manager localhost:49322 with 1807.0 MB RAM, BlockManagerId(driver, localhost, 49322)
17/02/20 10:08:04 INFO BlockManagerMaster: Registered BlockManager
17/02/20 10:08:05 INFO ReceiverTracker: Starting 1 receivers
17/02/20 10:08:05 INFO ReceiverTracker: ReceiverTracker started
17/02/20 10:08:05 INFO ShuffledDStream: Duration for remembering RDDs set to 16000 ms for org.apache.spark.streaming.dstream.ShuffledDStream@220f1b
17/02/20 10:08:05 INFO MappedDStream: Duration for remembering RDDs set to 16000 ms for org.apache.spark.streaming.dstream.MappedDStream@3c0db5ae
17/02/20 10:08:05 INFO FlatMappedDStream: Duration for remembering RDDs set to 16000 ms for org.apache.spark.streaming.dstream.FlatMappedDStream@2812d14f
17/02/20 10:08:05 INFO SocketInputDStream: Duration for remembering RDDs set to 16000 ms for org.apache.spark.streaming.dstream.SocketInputDStream@7a551df6
17/02/20 10:08:05 INFO ForEachDStream: metadataCleanupDelay = -1
17/02/20 10:08:05 INFO ShuffledDStream: metadataCleanupDelay = -1
17/02/20 10:08:05 INFO WindowedDStream: metadataCleanupDelay = -1
17/02/20 10:08:05 INFO ShuffledDStream: metadataCleanupDelay = -1
17/02/20 10:08:05 INFO MappedDStream: metadataCleanupDelay = -1
17/02/20 10:08:05 INFO FlatMappedDStream: metadataCleanupDelay = -1
17/02/20 10:08:05 INFO SocketInputDStream: metadataCleanupDelay = -1
17/02/20 10:08:05 INFO SocketInputDStream: Slide time = 2000 ms
17/02/20 10:08:05 INFO SocketInputDStream: Storage level = StorageLevel(false, false, false, false, 1)
17/02/20 10:08:05 INFO SocketInputDStream: Checkpoint interval = null
17/02/20 10:08:05 INFO SocketInputDStream: Remember duration = 16000 ms
17/02/20 10:08:05 INFO SocketInputDStream: Initialized and validated org.apache.spark.streaming.dstream.SocketInputDStream@7a551df6
17/02/20 10:08:05 INFO FlatMappedDStream: Slide time = 2000 ms
17/02/20 10:08:05 INFO FlatMappedDStream: Storage level = StorageLevel(false, false, false, false, 1)
17/02/20 10:08:05 INFO FlatMappedDStream: Checkpoint interval = null
17/02/20 10:08:05 INFO FlatMappedDStream: Remember duration = 16000 ms
17/02/20 10:08:05 INFO FlatMappedDStream: Initialized and validated org.apache.spark.streaming.dstream.FlatMappedDStream@2812d14f
17/02/20 10:08:05 INFO MappedDStream: Slide time = 2000 ms
17/02/20 10:08:05 INFO MappedDStream: Storage level = StorageLevel(false, false, false, false, 1)
17/02/20 10:08:05 INFO MappedDStream: Checkpoint interval = null
17/02/20 10:08:05 INFO MappedDStream: Remember duration = 16000 ms
17/02/20 10:08:05 INFO MappedDStream: Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@3c0db5ae
17/02/20 10:08:05 INFO ShuffledDStream: Slide time = 2000 ms
17/02/20 10:08:05 INFO ShuffledDStream: Storage level = StorageLevel(false, true, false, false, 1)
17/02/20 10:08:05 INFO ShuffledDStream: Checkpoint interval = null
17/02/20 10:08:05 INFO ShuffledDStream: Remember duration = 16000 ms
17/02/20 10:08:05 INFO ShuffledDStream: Initialized and validated org.apache.spark.streaming.dstream.ShuffledDStream@220f1b
17/02/20 10:08:05 INFO WindowedDStream: Slide time = 10000 ms
17/02/20 10:08:05 INFO WindowedDStream: Storage level = StorageLevel(false, false, false, false, 1)
17/02/20 10:08:05 INFO WindowedDStream: Checkpoint interval = null
17/02/20 10:08:05 INFO WindowedDStream: Remember duration = 10000 ms
17/02/20 10:08:05 INFO WindowedDStream: Initialized and validated org.apache.spark.streaming.dstream.WindowedDStream@70316d61
17/02/20 10:08:05 INFO ShuffledDStream: Slide time = 10000 ms
17/02/20 10:08:05 INFO ShuffledDStream: Storage level = StorageLevel(false, false, false, false, 1)
17/02/20 10:08:05 INFO ShuffledDStream: Checkpoint interval = null
17/02/20 10:08:05 INFO ShuffledDStream: Remember duration = 10000 ms
17/02/20 10:08:05 INFO ShuffledDStream: Initialized and validated org.apache.spark.streaming.dstream.ShuffledDStream@3d0434d2
17/02/20 10:08:05 INFO ForEachDStream: Slide time = 10000 ms
17/02/20 10:08:05 INFO ForEachDStream: Storage level = StorageLevel(false, false, false, false, 1)
17/02/20 10:08:05 INFO ForEachDStream: Checkpoint interval = null
17/02/20 10:08:05 INFO ForEachDStream: Remember duration = 10000 ms
17/02/20 10:08:05 INFO ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@3da0b822
17/02/20 10:08:05 INFO RecurringTimer: Started timer for JobGenerator at time 1487556486000
17/02/20 10:08:05 INFO JobGenerator: Started JobGenerator at 1487556486000 ms
17/02/20 10:08:05 INFO JobScheduler: Started JobScheduler
17/02/20 10:08:05 INFO ReceiverTracker: Receiver 0 started
17/02/20 10:08:05 INFO StreamingContext: StreamingContext started
17/02/20 10:08:05 INFO DAGScheduler: Got job 0 (start at WindowReduce.scala:26) with 1 output partitions
17/02/20 10:08:05 INFO DAGScheduler: Final stage: ResultStage 0 (start at WindowReduce.scala:26)
17/02/20 10:08:05 INFO DAGScheduler: Parents of final stage: List()
17/02/20 10:08:05 INFO DAGScheduler: Missing parents: List()
17/02/20 10:08:05 INFO DAGScheduler: Submitting ResultStage 0 (Receiver 0 ParallelCollectionRDD[0] at makeRDD at ReceiverTracker.scala:588), which has no missing parents
17/02/20 10:08:05 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 46.0 KB, free 1807.0 MB)
17/02/20 10:08:05 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 15.0 KB, free 1807.0 MB)
17/02/20 10:08:05 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:49322 (size: 15.0 KB, free: 1807.0 MB)
17/02/20 10:08:05 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:05 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (Receiver 0 ParallelCollectionRDD[0] at makeRDD at ReceiverTracker.scala:588)
17/02/20 10:08:05 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/02/20 10:08:05 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2637 bytes)
17/02/20 10:08:05 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/02/20 10:08:05 INFO RecurringTimer: Started timer for BlockGenerator at time 1487556485800
17/02/20 10:08:05 INFO BlockGenerator: Started BlockGenerator
17/02/20 10:08:05 INFO BlockGenerator: Started block pushing thread
17/02/20 10:08:05 INFO ReceiverTracker: Registered receiver for stream 0 from 192.168.164.1:49289
17/02/20 10:08:05 INFO ReceiverSupervisorImpl: Starting receiver
17/02/20 10:08:05 INFO ReceiverSupervisorImpl: Called receiver onStart
17/02/20 10:08:05 INFO SocketReceiver: Connecting to skycloud1:8888
17/02/20 10:08:05 INFO ReceiverSupervisorImpl: Waiting for receiver to be stopped
17/02/20 10:08:05 INFO SocketReceiver: Connected to skycloud1:8888
17/02/20 10:08:06 INFO ShuffledDStream: Time 1487556486000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 2000 ms
17/02/20 10:08:06 INFO JobScheduler: No jobs added for time 1487556486000 ms
17/02/20 10:08:08 INFO ShuffledDStream: Time 1487556488000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 4000 ms
17/02/20 10:08:08 INFO JobScheduler: No jobs added for time 1487556488000 ms
17/02/20 10:08:10 INFO ShuffledDStream: Time 1487556490000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 6000 ms
17/02/20 10:08:10 INFO JobScheduler: No jobs added for time 1487556490000 ms
17/02/20 10:08:12 INFO ShuffledDStream: Time 1487556492000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 8000 ms
17/02/20 10:08:12 INFO JobScheduler: No jobs added for time 1487556492000 ms
17/02/20 10:08:14 INFO ShuffledDStream: Slicing from 1487556490000 ms to 1487556494000 ms (aligned to 1487556490000 ms and 1487556494000 ms)
17/02/20 10:08:14 INFO JobScheduler: Added jobs for time 1487556494000 ms
17/02/20 10:08:14 INFO JobScheduler: Starting job streaming job 1487556494000 ms.0 from job set of time 1487556494000 ms
17/02/20 10:08:14 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:08:14 INFO DAGScheduler: Registering RDD 11 (map at WindowReduce.scala:22)
17/02/20 10:08:14 INFO DAGScheduler: Registering RDD 7 (map at WindowReduce.scala:22)
17/02/20 10:08:14 INFO DAGScheduler: Registering RDD 3 (map at WindowReduce.scala:22)
17/02/20 10:08:14 INFO DAGScheduler: Got job 1 (print at WindowReduce.scala:24) with 1 output partitions
17/02/20 10:08:14 INFO DAGScheduler: Final stage: ResultStage 4 (print at WindowReduce.scala:24)
17/02/20 10:08:14 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1, ShuffleMapStage 2, ShuffleMapStage 3)
17/02/20 10:08:14 INFO DAGScheduler: Missing parents: List()
17/02/20 10:08:14 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[14] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:14 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.0 KB, free 1807.0 MB)
17/02/20 10:08:14 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:14 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:14 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[14] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:08:14 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
17/02/20 10:08:14 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 1, localhost, partition 0,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:14 INFO Executor: Running task 0.0 in stage 4.0 (TID 1)
17/02/20 10:08:14 INFO CacheManager: Partition rdd_4_0 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 10 ms
17/02/20 10:08:14 INFO MemoryStore: Block rdd_4_0 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_4_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO CacheManager: Partition rdd_8_0 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:14 INFO MemoryStore: Block rdd_8_0 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_8_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO CacheManager: Partition rdd_12_0 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:14 INFO MemoryStore: Block rdd_12_0 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_12_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO Executor: Finished task 0.0 in stage 4.0 (TID 1). 1851 bytes result sent to driver
17/02/20 10:08:14 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 1) in 125 ms on localhost (1/1)
17/02/20 10:08:14 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
17/02/20 10:08:14 INFO DAGScheduler: ResultStage 4 (print at WindowReduce.scala:24) finished in 0.138 s
17/02/20 10:08:14 INFO DAGScheduler: Job 1 finished: print at WindowReduce.scala:24, took 0.204492 s
17/02/20 10:08:14 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:08:14 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 82 bytes
17/02/20 10:08:14 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 82 bytes
17/02/20 10:08:14 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 82 bytes
17/02/20 10:08:14 INFO DAGScheduler: Got job 2 (print at WindowReduce.scala:24) with 3 output partitions
17/02/20 10:08:14 INFO DAGScheduler: Final stage: ResultStage 8 (print at WindowReduce.scala:24)
17/02/20 10:08:14 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 5, ShuffleMapStage 6, ShuffleMapStage 7)
17/02/20 10:08:14 INFO DAGScheduler: Missing parents: List()
17/02/20 10:08:14 INFO DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[14] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:14 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.0 KB, free 1807.0 MB)
17/02/20 10:08:14 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:14 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:14 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 8 (MapPartitionsRDD[14] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:08:14 INFO TaskSchedulerImpl: Adding task set 8.0 with 3 tasks
17/02/20 10:08:14 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 2, localhost, partition 1,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:14 INFO TaskSetManager: Starting task 1.0 in stage 8.0 (TID 3, localhost, partition 2,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:14 INFO TaskSetManager: Starting task 2.0 in stage 8.0 (TID 4, localhost, partition 3,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:14 INFO Executor: Running task 1.0 in stage 8.0 (TID 3)
17/02/20 10:08:14 INFO Executor: Running task 0.0 in stage 8.0 (TID 2)
17/02/20 10:08:14 INFO Executor: Running task 2.0 in stage 8.0 (TID 4)
17/02/20 10:08:14 INFO CacheManager: Partition rdd_4_1 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO CacheManager: Partition rdd_4_2 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:14 INFO MemoryStore: Block rdd_4_1 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO CacheManager: Partition rdd_4_3 not found, computing it
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_4_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
17/02/20 10:08:14 INFO MemoryStore: Block rdd_4_2 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 5 ms
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_4_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO CacheManager: Partition rdd_8_2 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:14 INFO MemoryStore: Block rdd_4_3 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_4_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO MemoryStore: Block rdd_8_2 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_8_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO CacheManager: Partition rdd_8_3 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:14 INFO CacheManager: Partition rdd_12_2 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:14 INFO MemoryStore: Block rdd_8_3 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO MemoryStore: Block rdd_12_2 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_8_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_12_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO CacheManager: Partition rdd_8_1 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:14 INFO Executor: Finished task 1.0 in stage 8.0 (TID 3). 1851 bytes result sent to driver
17/02/20 10:08:14 INFO MemoryStore: Block rdd_8_1 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_8_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO CacheManager: Partition rdd_12_1 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:14 INFO MemoryStore: Block rdd_12_1 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_12_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO TaskSetManager: Finished task 1.0 in stage 8.0 (TID 3) in 67 ms on localhost (1/3)
17/02/20 10:08:14 INFO CacheManager: Partition rdd_12_3 not found, computing it
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:14 INFO Executor: Finished task 0.0 in stage 8.0 (TID 2). 1851 bytes result sent to driver
17/02/20 10:08:14 INFO MemoryStore: Block rdd_12_3 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:14 INFO BlockManagerInfo: Added rdd_12_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:14 INFO Executor: Finished task 2.0 in stage 8.0 (TID 4). 1851 bytes result sent to driver
17/02/20 10:08:14 INFO TaskSetManager: Finished task 2.0 in stage 8.0 (TID 4) in 185 ms on localhost (2/3)
17/02/20 10:08:14 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 2) in 196 ms on localhost (3/3)
17/02/20 10:08:14 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 
17/02/20 10:08:14 INFO DAGScheduler: ResultStage 8 (print at WindowReduce.scala:24) finished in 0.201 s
-------------------------------------------
Time: 1487556494000 ms
-------------------------------------------


17/02/20 10:08:14 INFO DAGScheduler: Job 2 finished: print at WindowReduce.scala:24, took 0.257986 s
17/02/20 10:08:14 INFO JobScheduler: Finished job streaming job 1487556494000 ms.0 from job set of time 1487556494000 ms
17/02/20 10:08:14 INFO JobScheduler: Total delay: 0.755 s for time 1487556494000 ms (execution: 0.564 s)
17/02/20 10:08:14 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer()
17/02/20 10:08:14 INFO InputInfoTracker: remove old batch metadata: 
17/02/20 10:08:14 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:49322 in memory (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:14 INFO ContextCleaner: Cleaned accumulator 2
17/02/20 10:08:16 INFO ShuffledDStream: Time 1487556496000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 12000 ms
17/02/20 10:08:16 INFO JobScheduler: No jobs added for time 1487556496000 ms
17/02/20 10:08:18 INFO ShuffledDStream: Time 1487556498000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 14000 ms
17/02/20 10:08:18 INFO JobScheduler: No jobs added for time 1487556498000 ms
17/02/20 10:08:20 INFO ShuffledDStream: Time 1487556500000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 16000 ms
17/02/20 10:08:20 INFO JobScheduler: No jobs added for time 1487556500000 ms
17/02/20 10:08:22 INFO ShuffledDStream: Time 1487556502000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 18000 ms
17/02/20 10:08:22 INFO JobScheduler: No jobs added for time 1487556502000 ms
17/02/20 10:08:24 INFO ShuffledDStream: Slicing from 1487556500000 ms to 1487556504000 ms (aligned to 1487556500000 ms and 1487556504000 ms)
17/02/20 10:08:24 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:08:24 INFO JobScheduler: Added jobs for time 1487556504000 ms
17/02/20 10:08:24 INFO JobScheduler: Starting job streaming job 1487556504000 ms.0 from job set of time 1487556504000 ms
17/02/20 10:08:24 INFO DAGScheduler: Registering RDD 25 (map at WindowReduce.scala:22)
17/02/20 10:08:24 INFO DAGScheduler: Registering RDD 21 (map at WindowReduce.scala:22)
17/02/20 10:08:24 INFO DAGScheduler: Registering RDD 17 (map at WindowReduce.scala:22)
17/02/20 10:08:24 INFO DAGScheduler: Got job 3 (print at WindowReduce.scala:24) with 1 output partitions
17/02/20 10:08:24 INFO DAGScheduler: Final stage: ResultStage 12 (print at WindowReduce.scala:24)
17/02/20 10:08:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 9, ShuffleMapStage 10, ShuffleMapStage 11)
17/02/20 10:08:24 INFO DAGScheduler: Missing parents: List()
17/02/20 10:08:24 INFO DAGScheduler: Submitting ResultStage 12 (MapPartitionsRDD[28] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:24 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 4.0 KB, free 1807.0 MB)
17/02/20 10:08:24 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1807.0 MB)
17/02/20 10:08:24 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:24 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:24 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 12 (MapPartitionsRDD[28] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:08:24 INFO TaskSchedulerImpl: Adding task set 12.0 with 1 tasks
17/02/20 10:08:24 INFO TaskSetManager: Starting task 0.0 in stage 12.0 (TID 5, localhost, partition 0,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:24 INFO Executor: Running task 0.0 in stage 12.0 (TID 5)
17/02/20 10:08:24 INFO CacheManager: Partition rdd_18_0 not found, computing it
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:24 INFO MemoryStore: Block rdd_18_0 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_18_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO CacheManager: Partition rdd_22_0 not found, computing it
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:24 INFO MemoryStore: Block rdd_22_0 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_22_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO CacheManager: Partition rdd_26_0 not found, computing it
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:24 INFO MemoryStore: Block rdd_26_0 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_26_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO Executor: Finished task 0.0 in stage 12.0 (TID 5). 1851 bytes result sent to driver
17/02/20 10:08:24 INFO TaskSetManager: Finished task 0.0 in stage 12.0 (TID 5) in 29 ms on localhost (1/1)
17/02/20 10:08:24 INFO TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 
17/02/20 10:08:24 INFO DAGScheduler: ResultStage 12 (print at WindowReduce.scala:24) finished in 0.032 s
17/02/20 10:08:24 INFO DAGScheduler: Job 3 finished: print at WindowReduce.scala:24, took 0.059108 s
17/02/20 10:08:24 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:08:24 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 3 is 82 bytes
17/02/20 10:08:24 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 4 is 82 bytes
17/02/20 10:08:24 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 5 is 82 bytes
17/02/20 10:08:24 INFO DAGScheduler: Got job 4 (print at WindowReduce.scala:24) with 3 output partitions
17/02/20 10:08:24 INFO DAGScheduler: Final stage: ResultStage 16 (print at WindowReduce.scala:24)
17/02/20 10:08:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 15, ShuffleMapStage 13, ShuffleMapStage 14)
17/02/20 10:08:24 INFO DAGScheduler: Missing parents: List()
17/02/20 10:08:24 INFO DAGScheduler: Submitting ResultStage 16 (MapPartitionsRDD[28] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:24 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 4.0 KB, free 1807.0 MB)
17/02/20 10:08:24 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1807.0 MB)
17/02/20 10:08:24 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:24 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:24 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 16 (MapPartitionsRDD[28] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:08:24 INFO TaskSchedulerImpl: Adding task set 16.0 with 3 tasks
17/02/20 10:08:24 INFO TaskSetManager: Starting task 0.0 in stage 16.0 (TID 6, localhost, partition 1,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:24 INFO TaskSetManager: Starting task 1.0 in stage 16.0 (TID 7, localhost, partition 2,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:24 INFO TaskSetManager: Starting task 2.0 in stage 16.0 (TID 8, localhost, partition 3,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:24 INFO Executor: Running task 1.0 in stage 16.0 (TID 7)
17/02/20 10:08:24 INFO Executor: Running task 0.0 in stage 16.0 (TID 6)
17/02/20 10:08:24 INFO Executor: Running task 2.0 in stage 16.0 (TID 8)
17/02/20 10:08:24 INFO CacheManager: Partition rdd_18_1 not found, computing it
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:24 INFO CacheManager: Partition rdd_18_2 not found, computing it
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:24 INFO MemoryStore: Block rdd_18_1 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_18_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO MemoryStore: Block rdd_18_2 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO CacheManager: Partition rdd_22_1 not found, computing it
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_18_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO CacheManager: Partition rdd_18_3 not found, computing it
17/02/20 10:08:24 INFO MemoryStore: Block rdd_22_1 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:24 INFO CacheManager: Partition rdd_22_2 not found, computing it
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_22_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO CacheManager: Partition rdd_26_1 not found, computing it
17/02/20 10:08:24 INFO MemoryStore: Block rdd_18_3 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:24 INFO MemoryStore: Block rdd_22_2 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_18_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_22_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO MemoryStore: Block rdd_26_1 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_26_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO CacheManager: Partition rdd_22_3 not found, computing it
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:24 INFO Executor: Finished task 0.0 in stage 16.0 (TID 6). 1851 bytes result sent to driver
17/02/20 10:08:24 INFO MemoryStore: Block rdd_22_3 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO CacheManager: Partition rdd_26_2 not found, computing it
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_22_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO MemoryStore: Block rdd_26_2 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_26_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO CacheManager: Partition rdd_26_3 not found, computing it
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:24 INFO MemoryStore: Block rdd_26_3 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:08:24 INFO TaskSetManager: Finished task 0.0 in stage 16.0 (TID 6) in 43 ms on localhost (1/3)
17/02/20 10:08:24 INFO Executor: Finished task 1.0 in stage 16.0 (TID 7). 1851 bytes result sent to driver
17/02/20 10:08:24 INFO BlockManagerInfo: Added rdd_26_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:24 INFO TaskSetManager: Finished task 1.0 in stage 16.0 (TID 7) in 54 ms on localhost (2/3)
17/02/20 10:08:24 INFO Executor: Finished task 2.0 in stage 16.0 (TID 8). 1851 bytes result sent to driver
17/02/20 10:08:24 INFO DAGScheduler: ResultStage 16 (print at WindowReduce.scala:24) finished in 0.061 s
17/02/20 10:08:24 INFO TaskSetManager: Finished task 2.0 in stage 16.0 (TID 8) in 55 ms on localhost (3/3)
17/02/20 10:08:24 INFO TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 
17/02/20 10:08:24 INFO DAGScheduler: Job 4 finished: print at WindowReduce.scala:24, took 0.100034 s
17/02/20 10:08:24 INFO JobScheduler: Finished job streaming job 1487556504000 ms.0 from job set of time 1487556504000 ms
17/02/20 10:08:24 INFO JobScheduler: Total delay: 0.261 s for time 1487556504000 ms (execution: 0.174 s)
-------------------------------------------
Time: 1487556504000 ms
-------------------------------------------


17/02/20 10:08:24 INFO MapPartitionsRDD: Removing RDD 14 from persistence list
17/02/20 10:08:24 INFO BlockManager: Removing RDD 14
17/02/20 10:08:24 INFO PartitionerAwareUnionRDD: Removing RDD 13 from persistence list
17/02/20 10:08:24 INFO BlockManager: Removing RDD 13
17/02/20 10:08:24 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1487556486000 ms)
17/02/20 10:08:24 INFO InputInfoTracker: remove old batch metadata: 
17/02/20 10:08:26 INFO ShuffledDStream: Time 1487556506000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 22000 ms
17/02/20 10:08:26 INFO JobScheduler: No jobs added for time 1487556506000 ms
17/02/20 10:08:28 INFO ShuffledDStream: Time 1487556508000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 24000 ms
17/02/20 10:08:28 INFO JobScheduler: No jobs added for time 1487556508000 ms
17/02/20 10:08:28 INFO MemoryStore: Block input-0-1487556508000 stored as bytes in memory (estimated size 24.0 B, free 1807.0 MB)
17/02/20 10:08:28 INFO BlockManagerInfo: Added input-0-1487556508000 in memory on localhost:49322 (size: 24.0 B, free: 1807.0 MB)
17/02/20 10:08:28 INFO BlockGenerator: Pushed block input-0-1487556508000
17/02/20 10:08:30 INFO ShuffledDStream: Time 1487556510000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 26000 ms
17/02/20 10:08:30 INFO JobScheduler: No jobs added for time 1487556510000 ms
17/02/20 10:08:31 INFO MemoryStore: Block input-0-1487556510800 stored as bytes in memory (estimated size 19.0 B, free 1807.0 MB)
17/02/20 10:08:31 INFO BlockManagerInfo: Added input-0-1487556510800 in memory on localhost:49322 (size: 19.0 B, free: 1807.0 MB)
17/02/20 10:08:31 INFO BlockGenerator: Pushed block input-0-1487556510800
17/02/20 10:08:32 INFO ShuffledDStream: Time 1487556512000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 28000 ms
17/02/20 10:08:32 INFO JobScheduler: No jobs added for time 1487556512000 ms
17/02/20 10:08:33 INFO MemoryStore: Block input-0-1487556513400 stored as bytes in memory (estimated size 24.0 B, free 1807.0 MB)
17/02/20 10:08:33 INFO BlockManagerInfo: Added input-0-1487556513400 in memory on localhost:49322 (size: 24.0 B, free: 1807.0 MB)
17/02/20 10:08:33 INFO BlockGenerator: Pushed block input-0-1487556513400
17/02/20 10:08:34 INFO ShuffledDStream: Slicing from 1487556510000 ms to 1487556514000 ms (aligned to 1487556510000 ms and 1487556514000 ms)
17/02/20 10:08:34 INFO JobScheduler: Added jobs for time 1487556514000 ms
17/02/20 10:08:34 INFO JobScheduler: Starting job streaming job 1487556514000 ms.0 from job set of time 1487556514000 ms
17/02/20 10:08:34 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:08:34 INFO DAGScheduler: Registering RDD 39 (map at WindowReduce.scala:22)
17/02/20 10:08:34 INFO DAGScheduler: Registering RDD 35 (map at WindowReduce.scala:22)
17/02/20 10:08:34 INFO DAGScheduler: Registering RDD 31 (map at WindowReduce.scala:22)
17/02/20 10:08:34 INFO DAGScheduler: Got job 5 (print at WindowReduce.scala:24) with 1 output partitions
17/02/20 10:08:34 INFO DAGScheduler: Final stage: ResultStage 20 (print at WindowReduce.scala:24)
17/02/20 10:08:34 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 19, ShuffleMapStage 17, ShuffleMapStage 18)
17/02/20 10:08:34 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 19, ShuffleMapStage 17, ShuffleMapStage 18)
17/02/20 10:08:34 INFO DAGScheduler: Submitting ShuffleMapStage 17 (MapPartitionsRDD[39] at map at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:34 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 2.6 KB, free 1807.0 MB)
17/02/20 10:08:34 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 1625.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:49322 (size: 1625.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:34 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 17 (MapPartitionsRDD[39] at map at WindowReduce.scala:22)
17/02/20 10:08:34 INFO TaskSchedulerImpl: Adding task set 17.0 with 1 tasks
17/02/20 10:08:34 INFO DAGScheduler: Submitting ShuffleMapStage 18 (MapPartitionsRDD[35] at map at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:34 INFO TaskSetManager: Starting task 0.0 in stage 17.0 (TID 9, localhost, partition 0,NODE_LOCAL, 2006 bytes)
17/02/20 10:08:34 INFO Executor: Running task 0.0 in stage 17.0 (TID 9)
17/02/20 10:08:34 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 2.6 KB, free 1806.9 MB)
17/02/20 10:08:34 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1623.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:49322 (size: 1623.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:34 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 18 (MapPartitionsRDD[35] at map at WindowReduce.scala:22)
17/02/20 10:08:34 INFO TaskSchedulerImpl: Adding task set 18.0 with 1 tasks
17/02/20 10:08:34 INFO DAGScheduler: Submitting ShuffleMapStage 19 (MapPartitionsRDD[31] at map at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:34 INFO TaskSetManager: Starting task 0.0 in stage 18.0 (TID 10, localhost, partition 0,NODE_LOCAL, 2006 bytes)
17/02/20 10:08:34 INFO BlockManager: Found block input-0-1487556513400 locally
17/02/20 10:08:34 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 2.6 KB, free 1806.9 MB)
17/02/20 10:08:34 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 1620.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO Executor: Running task 0.0 in stage 18.0 (TID 10)
17/02/20 10:08:34 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:49322 (size: 1620.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:34 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 19 (MapPartitionsRDD[31] at map at WindowReduce.scala:22)
17/02/20 10:08:34 INFO TaskSchedulerImpl: Adding task set 19.0 with 1 tasks
17/02/20 10:08:34 INFO BlockManager: Found block input-0-1487556510800 locally
17/02/20 10:08:34 INFO TaskSetManager: Starting task 0.0 in stage 19.0 (TID 11, localhost, partition 0,NODE_LOCAL, 2006 bytes)
17/02/20 10:08:34 INFO Executor: Running task 0.0 in stage 19.0 (TID 11)
17/02/20 10:08:34 INFO BlockManager: Found block input-0-1487556508000 locally
17/02/20 10:08:34 INFO Executor: Finished task 0.0 in stage 17.0 (TID 9). 1161 bytes result sent to driver
17/02/20 10:08:34 INFO Executor: Finished task 0.0 in stage 19.0 (TID 11). 1161 bytes result sent to driver
17/02/20 10:08:34 INFO TaskSetManager: Finished task 0.0 in stage 17.0 (TID 9) in 96 ms on localhost (1/1)
17/02/20 10:08:34 INFO TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool 
17/02/20 10:08:34 INFO TaskSetManager: Finished task 0.0 in stage 19.0 (TID 11) in 62 ms on localhost (1/1)
17/02/20 10:08:34 INFO TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool 
17/02/20 10:08:34 INFO DAGScheduler: ShuffleMapStage 17 (map at WindowReduce.scala:22) finished in 0.099 s
17/02/20 10:08:34 INFO DAGScheduler: looking for newly runnable stages
17/02/20 10:08:34 INFO DAGScheduler: running: Set(ResultStage 0, ShuffleMapStage 19, ShuffleMapStage 18)
17/02/20 10:08:34 INFO DAGScheduler: waiting: Set(ResultStage 20)
17/02/20 10:08:34 INFO DAGScheduler: failed: Set()
17/02/20 10:08:34 INFO DAGScheduler: ShuffleMapStage 19 (map at WindowReduce.scala:22) finished in 0.078 s
17/02/20 10:08:34 INFO DAGScheduler: looking for newly runnable stages
17/02/20 10:08:34 INFO DAGScheduler: running: Set(ResultStage 0, ShuffleMapStage 18)
17/02/20 10:08:34 INFO DAGScheduler: waiting: Set(ResultStage 20)
17/02/20 10:08:34 INFO DAGScheduler: failed: Set()
17/02/20 10:08:34 INFO Executor: Finished task 0.0 in stage 18.0 (TID 10). 1161 bytes result sent to driver
17/02/20 10:08:34 INFO TaskSetManager: Finished task 0.0 in stage 18.0 (TID 10) in 104 ms on localhost (1/1)
17/02/20 10:08:34 INFO DAGScheduler: ShuffleMapStage 18 (map at WindowReduce.scala:22) finished in 0.105 s
17/02/20 10:08:34 INFO TaskSchedulerImpl: Removed TaskSet 18.0, whose tasks have all completed, from pool 
17/02/20 10:08:34 INFO DAGScheduler: looking for newly runnable stages
17/02/20 10:08:34 INFO DAGScheduler: running: Set(ResultStage 0)
17/02/20 10:08:34 INFO DAGScheduler: waiting: Set(ResultStage 20)
17/02/20 10:08:34 INFO DAGScheduler: failed: Set()
17/02/20 10:08:34 INFO DAGScheduler: Submitting ResultStage 20 (MapPartitionsRDD[42] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:34 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.0 KB, free 1806.9 MB)
17/02/20 10:08:34 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:34 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:34 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 20 (MapPartitionsRDD[42] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:08:34 INFO TaskSchedulerImpl: Adding task set 20.0 with 1 tasks
17/02/20 10:08:34 INFO TaskSetManager: Starting task 0.0 in stage 20.0 (TID 12, localhost, partition 0,NODE_LOCAL, 2108 bytes)
17/02/20 10:08:34 INFO Executor: Running task 0.0 in stage 20.0 (TID 12)
17/02/20 10:08:34 INFO CacheManager: Partition rdd_32_0 not found, computing it
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
17/02/20 10:08:34 INFO MemoryStore: Block rdd_32_0 stored as bytes in memory (estimated size 154.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_32_0 in memory on localhost:49322 (size: 154.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO CacheManager: Partition rdd_36_0 not found, computing it
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:34 INFO MemoryStore: Block rdd_36_0 stored as bytes in memory (estimated size 153.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_36_0 in memory on localhost:49322 (size: 153.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO CacheManager: Partition rdd_40_0 not found, computing it
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:34 INFO MemoryStore: Block rdd_40_0 stored as bytes in memory (estimated size 172.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_40_0 in memory on localhost:49322 (size: 172.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO Executor: Finished task 0.0 in stage 20.0 (TID 12). 2048 bytes result sent to driver
17/02/20 10:08:34 INFO TaskSetManager: Finished task 0.0 in stage 20.0 (TID 12) in 51 ms on localhost (1/1)
17/02/20 10:08:34 INFO DAGScheduler: ResultStage 20 (print at WindowReduce.scala:24) finished in 0.052 s
17/02/20 10:08:34 INFO TaskSchedulerImpl: Removed TaskSet 20.0, whose tasks have all completed, from pool 
17/02/20 10:08:34 INFO DAGScheduler: Job 5 finished: print at WindowReduce.scala:24, took 0.220163 s
17/02/20 10:08:34 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:08:34 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 6 is 147 bytes
17/02/20 10:08:34 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 7 is 147 bytes
17/02/20 10:08:34 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 8 is 147 bytes
17/02/20 10:08:34 INFO DAGScheduler: Got job 6 (print at WindowReduce.scala:24) with 3 output partitions
17/02/20 10:08:34 INFO DAGScheduler: Final stage: ResultStage 24 (print at WindowReduce.scala:24)
17/02/20 10:08:34 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 21, ShuffleMapStage 22, ShuffleMapStage 23)
17/02/20 10:08:34 INFO DAGScheduler: Missing parents: List()
17/02/20 10:08:34 INFO DAGScheduler: Submitting ResultStage 24 (MapPartitionsRDD[42] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:34 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 4.0 KB, free 1806.9 MB)
17/02/20 10:08:34 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:34 INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:34 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 24 (MapPartitionsRDD[42] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:08:34 INFO TaskSchedulerImpl: Adding task set 24.0 with 3 tasks
17/02/20 10:08:34 INFO TaskSetManager: Starting task 1.0 in stage 24.0 (TID 13, localhost, partition 2,NODE_LOCAL, 2108 bytes)
17/02/20 10:08:34 INFO TaskSetManager: Starting task 2.0 in stage 24.0 (TID 14, localhost, partition 3,NODE_LOCAL, 2108 bytes)
17/02/20 10:08:34 INFO TaskSetManager: Starting task 0.0 in stage 24.0 (TID 15, localhost, partition 1,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:34 INFO Executor: Running task 0.0 in stage 24.0 (TID 15)
17/02/20 10:08:34 INFO Executor: Running task 2.0 in stage 24.0 (TID 14)
17/02/20 10:08:34 INFO Executor: Running task 1.0 in stage 24.0 (TID 13)
17/02/20 10:08:34 INFO CacheManager: Partition rdd_32_2 not found, computing it
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:34 INFO CacheManager: Partition rdd_32_3 not found, computing it
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:34 INFO MemoryStore: Block rdd_32_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO MemoryStore: Block rdd_32_2 stored as bytes in memory (estimated size 173.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_32_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_32_2 in memory on localhost:49322 (size: 173.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO CacheManager: Partition rdd_36_2 not found, computing it
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:34 INFO CacheManager: Partition rdd_32_1 not found, computing it
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:34 INFO MemoryStore: Block rdd_32_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO CacheManager: Partition rdd_36_3 not found, computing it
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:34 INFO MemoryStore: Block rdd_36_2 stored as bytes in memory (estimated size 169.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_32_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO MemoryStore: Block rdd_36_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_36_2 in memory on localhost:49322 (size: 169.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_36_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO CacheManager: Partition rdd_36_1 not found, computing it
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:34 INFO CacheManager: Partition rdd_40_3 not found, computing it
17/02/20 10:08:34 INFO CacheManager: Partition rdd_40_2 not found, computing it
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:34 INFO MemoryStore: Block rdd_36_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO MemoryStore: Block rdd_40_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_36_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_40_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO MemoryStore: Block rdd_40_3 stored as bytes in memory (estimated size 155.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO Executor: Finished task 1.0 in stage 24.0 (TID 13). 2062 bytes result sent to driver
17/02/20 10:08:34 INFO CacheManager: Partition rdd_40_1 not found, computing it
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_40_3 in memory on localhost:49322 (size: 155.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/02/20 10:08:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:34 INFO TaskSetManager: Finished task 1.0 in stage 24.0 (TID 13) in 44 ms on localhost (1/3)
17/02/20 10:08:34 INFO Executor: Finished task 2.0 in stage 24.0 (TID 14). 2002 bytes result sent to driver
17/02/20 10:08:34 INFO MemoryStore: Block rdd_40_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:34 INFO TaskSetManager: Finished task 2.0 in stage 24.0 (TID 14) in 47 ms on localhost (2/3)
17/02/20 10:08:34 INFO BlockManagerInfo: Added rdd_40_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:34 INFO Executor: Finished task 0.0 in stage 24.0 (TID 15). 1851 bytes result sent to driver
17/02/20 10:08:34 INFO TaskSetManager: Finished task 0.0 in stage 24.0 (TID 15) in 51 ms on localhost (3/3)
17/02/20 10:08:34 INFO DAGScheduler: ResultStage 24 (print at WindowReduce.scala:24) finished in 0.055 s
17/02/20 10:08:34 INFO TaskSchedulerImpl: Removed TaskSet 24.0, whose tasks have all completed, from pool 
17/02/20 10:08:34 INFO DAGScheduler: Job 6 finished: print at WindowReduce.scala:24, took 0.081687 s
17/02/20 10:08:34 INFO JobScheduler: Finished job streaming job 1487556514000 ms.0 from job set of time 1487556514000 ms
17/02/20 10:08:34 INFO JobScheduler: Total delay: 0.388 s for time 1487556514000 ms (execution: 0.325 s)
17/02/20 10:08:34 INFO MapPartitionsRDD: Removing RDD 28 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 28
17/02/20 10:08:34 INFO PartitionerAwareUnionRDD: Removing RDD 27 from persistence list
-------------------------------------------
Time: 1487556514000 ms
-------------------------------------------
(wdwfd,1)
(wdwdv,1)
(wdwf,2)
(dwdw,1)
(dwe,1)
(ewdfwe,1)
(sdw,1)
(dwfefg,1)


17/02/20 10:08:34 INFO BlockManager: Removing RDD 27
17/02/20 10:08:34 INFO ShuffledRDD: Removing RDD 8 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 8
17/02/20 10:08:34 INFO ShuffledRDD: Removing RDD 12 from persistence list
17/02/20 10:08:34 INFO ShuffledRDD: Removing RDD 4 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 12
17/02/20 10:08:34 INFO BlockManager: Removing RDD 4
17/02/20 10:08:34 INFO MapPartitionsRDD: Removing RDD 7 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 7
17/02/20 10:08:34 INFO MapPartitionsRDD: Removing RDD 11 from persistence list
17/02/20 10:08:34 INFO MapPartitionsRDD: Removing RDD 3 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 11
17/02/20 10:08:34 INFO BlockManager: Removing RDD 3
17/02/20 10:08:34 INFO MapPartitionsRDD: Removing RDD 6 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 6
17/02/20 10:08:34 INFO MapPartitionsRDD: Removing RDD 10 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 10
17/02/20 10:08:34 INFO MapPartitionsRDD: Removing RDD 2 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 2
17/02/20 10:08:34 INFO BlockRDD: Removing RDD 5 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 5
17/02/20 10:08:34 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[5] at socketTextStream at WindowReduce.scala:19 of time 1487556514000 ms
17/02/20 10:08:34 INFO BlockRDD: Removing RDD 9 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 9
17/02/20 10:08:34 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[9] at socketTextStream at WindowReduce.scala:19 of time 1487556514000 ms
17/02/20 10:08:34 INFO BlockRDD: Removing RDD 1 from persistence list
17/02/20 10:08:34 INFO BlockManager: Removing RDD 1
17/02/20 10:08:34 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[1] at socketTextStream at WindowReduce.scala:19 of time 1487556514000 ms
17/02/20 10:08:34 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1487556490000 ms, 1487556492000 ms, 1487556494000 ms, 1487556488000 ms, 1487556496000 ms)
17/02/20 10:08:34 INFO InputInfoTracker: remove old batch metadata: 1487556492000 ms 1487556494000 ms 1487556490000 ms
17/02/20 10:08:35 INFO MemoryStore: Block input-0-1487556515200 stored as bytes in memory (estimated size 17.0 B, free 1806.9 MB)
17/02/20 10:08:35 INFO BlockManagerInfo: Added input-0-1487556515200 in memory on localhost:49322 (size: 17.0 B, free: 1807.0 MB)
17/02/20 10:08:35 INFO BlockGenerator: Pushed block input-0-1487556515200
17/02/20 10:08:36 INFO ShuffledDStream: Time 1487556516000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 32000 ms
17/02/20 10:08:36 INFO JobScheduler: No jobs added for time 1487556516000 ms
17/02/20 10:08:38 INFO ShuffledDStream: Time 1487556518000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 34000 ms
17/02/20 10:08:38 INFO JobScheduler: No jobs added for time 1487556518000 ms
17/02/20 10:08:40 INFO ShuffledDStream: Time 1487556520000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 36000 ms
17/02/20 10:08:40 INFO JobScheduler: No jobs added for time 1487556520000 ms
17/02/20 10:08:42 INFO ShuffledDStream: Time 1487556522000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 38000 ms
17/02/20 10:08:42 INFO JobScheduler: No jobs added for time 1487556522000 ms
17/02/20 10:08:44 INFO ShuffledDStream: Slicing from 1487556520000 ms to 1487556524000 ms (aligned to 1487556520000 ms and 1487556524000 ms)
17/02/20 10:08:44 INFO JobScheduler: Added jobs for time 1487556524000 ms
17/02/20 10:08:44 INFO JobScheduler: Starting job streaming job 1487556524000 ms.0 from job set of time 1487556524000 ms
17/02/20 10:08:44 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:08:44 INFO DAGScheduler: Registering RDD 53 (map at WindowReduce.scala:22)
17/02/20 10:08:44 INFO DAGScheduler: Registering RDD 49 (map at WindowReduce.scala:22)
17/02/20 10:08:44 INFO DAGScheduler: Registering RDD 45 (map at WindowReduce.scala:22)
17/02/20 10:08:44 INFO DAGScheduler: Got job 7 (print at WindowReduce.scala:24) with 1 output partitions
17/02/20 10:08:44 INFO DAGScheduler: Final stage: ResultStage 28 (print at WindowReduce.scala:24)
17/02/20 10:08:44 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 27, ShuffleMapStage 25, ShuffleMapStage 26)
17/02/20 10:08:44 INFO DAGScheduler: Missing parents: List()
17/02/20 10:08:44 INFO DAGScheduler: Submitting ResultStage 28 (MapPartitionsRDD[56] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:44 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.0 KB, free 1806.9 MB)
17/02/20 10:08:44 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1806.9 MB)
17/02/20 10:08:44 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:44 INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:44 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 28 (MapPartitionsRDD[56] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:08:44 INFO TaskSchedulerImpl: Adding task set 28.0 with 1 tasks
17/02/20 10:08:44 INFO TaskSetManager: Starting task 0.0 in stage 28.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:44 INFO Executor: Running task 0.0 in stage 28.0 (TID 16)
17/02/20 10:08:44 INFO CacheManager: Partition rdd_46_0 not found, computing it
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:44 INFO MemoryStore: Block rdd_46_0 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_46_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO CacheManager: Partition rdd_50_0 not found, computing it
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:44 INFO MemoryStore: Block rdd_50_0 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_50_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO CacheManager: Partition rdd_54_0 not found, computing it
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:44 INFO MemoryStore: Block rdd_54_0 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_54_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO Executor: Finished task 0.0 in stage 28.0 (TID 16). 1851 bytes result sent to driver
17/02/20 10:08:44 INFO TaskSetManager: Finished task 0.0 in stage 28.0 (TID 16) in 21 ms on localhost (1/1)
17/02/20 10:08:44 INFO DAGScheduler: ResultStage 28 (print at WindowReduce.scala:24) finished in 0.022 s
17/02/20 10:08:44 INFO TaskSchedulerImpl: Removed TaskSet 28.0, whose tasks have all completed, from pool 
17/02/20 10:08:44 INFO DAGScheduler: Job 7 finished: print at WindowReduce.scala:24, took 0.045784 s
17/02/20 10:08:44 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:08:44 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 9 is 82 bytes
17/02/20 10:08:44 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 10 is 82 bytes
17/02/20 10:08:44 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 11 is 82 bytes
17/02/20 10:08:44 INFO DAGScheduler: Got job 8 (print at WindowReduce.scala:24) with 3 output partitions
17/02/20 10:08:44 INFO DAGScheduler: Final stage: ResultStage 32 (print at WindowReduce.scala:24)
17/02/20 10:08:44 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 30, ShuffleMapStage 31, ShuffleMapStage 29)
17/02/20 10:08:44 INFO DAGScheduler: Missing parents: List()
17/02/20 10:08:44 INFO DAGScheduler: Submitting ResultStage 32 (MapPartitionsRDD[56] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:44 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 4.0 KB, free 1806.9 MB)
17/02/20 10:08:44 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1806.9 MB)
17/02/20 10:08:44 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:44 INFO SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:44 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 32 (MapPartitionsRDD[56] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:08:44 INFO TaskSchedulerImpl: Adding task set 32.0 with 3 tasks
17/02/20 10:08:44 INFO TaskSetManager: Starting task 0.0 in stage 32.0 (TID 17, localhost, partition 1,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:44 INFO TaskSetManager: Starting task 1.0 in stage 32.0 (TID 18, localhost, partition 2,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:44 INFO TaskSetManager: Starting task 2.0 in stage 32.0 (TID 19, localhost, partition 3,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:44 INFO Executor: Running task 0.0 in stage 32.0 (TID 17)
17/02/20 10:08:44 INFO Executor: Running task 1.0 in stage 32.0 (TID 18)
17/02/20 10:08:44 INFO Executor: Running task 2.0 in stage 32.0 (TID 19)
17/02/20 10:08:44 INFO CacheManager: Partition rdd_46_3 not found, computing it
17/02/20 10:08:44 INFO CacheManager: Partition rdd_46_2 not found, computing it
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:44 INFO CacheManager: Partition rdd_46_1 not found, computing it
17/02/20 10:08:44 INFO MemoryStore: Block rdd_46_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:44 INFO MemoryStore: Block rdd_46_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_46_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_46_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO MemoryStore: Block rdd_46_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO CacheManager: Partition rdd_50_2 not found, computing it
17/02/20 10:08:44 INFO CacheManager: Partition rdd_50_3 not found, computing it
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_46_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:44 INFO MemoryStore: Block rdd_50_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO CacheManager: Partition rdd_50_1 not found, computing it
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_50_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:44 INFO MemoryStore: Block rdd_50_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO CacheManager: Partition rdd_54_3 not found, computing it
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_50_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:44 INFO MemoryStore: Block rdd_50_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO CacheManager: Partition rdd_54_2 not found, computing it
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:44 INFO MemoryStore: Block rdd_54_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_50_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_54_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO MemoryStore: Block rdd_54_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO CacheManager: Partition rdd_54_1 not found, computing it
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_54_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO Executor: Finished task 2.0 in stage 32.0 (TID 19). 1851 bytes result sent to driver
17/02/20 10:08:44 INFO MemoryStore: Block rdd_54_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:44 INFO TaskSetManager: Finished task 2.0 in stage 32.0 (TID 19) in 27 ms on localhost (1/3)
17/02/20 10:08:44 INFO BlockManagerInfo: Added rdd_54_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:44 INFO Executor: Finished task 1.0 in stage 32.0 (TID 18). 1851 bytes result sent to driver
17/02/20 10:08:44 INFO Executor: Finished task 0.0 in stage 32.0 (TID 17). 1851 bytes result sent to driver
17/02/20 10:08:44 INFO TaskSetManager: Finished task 1.0 in stage 32.0 (TID 18) in 31 ms on localhost (2/3)
17/02/20 10:08:44 INFO TaskSetManager: Finished task 0.0 in stage 32.0 (TID 17) in 32 ms on localhost (3/3)
17/02/20 10:08:44 INFO DAGScheduler: ResultStage 32 (print at WindowReduce.scala:24) finished in 0.033 s
17/02/20 10:08:44 INFO TaskSchedulerImpl: Removed TaskSet 32.0, whose tasks have all completed, from pool 
17/02/20 10:08:44 INFO DAGScheduler: Job 8 finished: print at WindowReduce.scala:24, took 0.049153 s
17/02/20 10:08:44 INFO JobScheduler: Finished job streaming job 1487556524000 ms.0 from job set of time 1487556524000 ms
17/02/20 10:08:44 INFO JobScheduler: Total delay: 0.188 s for time 1487556524000 ms (execution: 0.122 s)
17/02/20 10:08:44 INFO MapPartitionsRDD: Removing RDD 42 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 42
17/02/20 10:08:44 INFO PartitionerAwareUnionRDD: Removing RDD 41 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 41
17/02/20 10:08:44 INFO ShuffledRDD: Removing RDD 18 from persistence list
-------------------------------------------
Time: 1487556524000 ms
-------------------------------------------


17/02/20 10:08:44 INFO BlockManager: Removing RDD 18
17/02/20 10:08:44 INFO ShuffledRDD: Removing RDD 26 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 26
17/02/20 10:08:44 INFO ShuffledRDD: Removing RDD 22 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 22
17/02/20 10:08:44 INFO MapPartitionsRDD: Removing RDD 17 from persistence list
17/02/20 10:08:44 INFO MapPartitionsRDD: Removing RDD 25 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 17
17/02/20 10:08:44 INFO BlockManager: Removing RDD 25
17/02/20 10:08:44 INFO MapPartitionsRDD: Removing RDD 21 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 21
17/02/20 10:08:44 INFO MapPartitionsRDD: Removing RDD 16 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 16
17/02/20 10:08:44 INFO MapPartitionsRDD: Removing RDD 24 from persistence list
17/02/20 10:08:44 INFO MapPartitionsRDD: Removing RDD 20 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 24
17/02/20 10:08:44 INFO BlockManager: Removing RDD 20
17/02/20 10:08:44 INFO BlockRDD: Removing RDD 15 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 15
17/02/20 10:08:44 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[15] at socketTextStream at WindowReduce.scala:19 of time 1487556524000 ms
17/02/20 10:08:44 INFO BlockRDD: Removing RDD 23 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 23
17/02/20 10:08:44 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[23] at socketTextStream at WindowReduce.scala:19 of time 1487556524000 ms
17/02/20 10:08:44 INFO BlockRDD: Removing RDD 19 from persistence list
17/02/20 10:08:44 INFO BlockManager: Removing RDD 19
17/02/20 10:08:44 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[19] at socketTextStream at WindowReduce.scala:19 of time 1487556524000 ms
17/02/20 10:08:44 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1487556506000 ms, 1487556502000 ms, 1487556504000 ms, 1487556498000 ms, 1487556500000 ms)
17/02/20 10:08:44 INFO InputInfoTracker: remove old batch metadata: 1487556500000 ms 1487556504000 ms 1487556502000 ms
17/02/20 10:08:46 INFO ShuffledDStream: Time 1487556526000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 42000 ms
17/02/20 10:08:46 INFO JobScheduler: No jobs added for time 1487556526000 ms
17/02/20 10:08:48 INFO ShuffledDStream: Time 1487556528000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 44000 ms
17/02/20 10:08:48 INFO JobScheduler: No jobs added for time 1487556528000 ms
17/02/20 10:08:50 INFO ShuffledDStream: Time 1487556530000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 46000 ms
17/02/20 10:08:50 INFO JobScheduler: No jobs added for time 1487556530000 ms
17/02/20 10:08:52 INFO ShuffledDStream: Time 1487556532000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 48000 ms
17/02/20 10:08:52 INFO JobScheduler: No jobs added for time 1487556532000 ms
17/02/20 10:08:54 INFO ShuffledDStream: Slicing from 1487556530000 ms to 1487556534000 ms (aligned to 1487556530000 ms and 1487556534000 ms)
17/02/20 10:08:54 INFO JobScheduler: Starting job streaming job 1487556534000 ms.0 from job set of time 1487556534000 ms
17/02/20 10:08:54 INFO JobScheduler: Added jobs for time 1487556534000 ms
17/02/20 10:08:54 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:08:54 INFO DAGScheduler: Registering RDD 67 (map at WindowReduce.scala:22)
17/02/20 10:08:54 INFO DAGScheduler: Registering RDD 63 (map at WindowReduce.scala:22)
17/02/20 10:08:54 INFO DAGScheduler: Registering RDD 59 (map at WindowReduce.scala:22)
17/02/20 10:08:54 INFO DAGScheduler: Got job 9 (print at WindowReduce.scala:24) with 1 output partitions
17/02/20 10:08:54 INFO DAGScheduler: Final stage: ResultStage 36 (print at WindowReduce.scala:24)
17/02/20 10:08:54 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 33, ShuffleMapStage 34, ShuffleMapStage 35)
17/02/20 10:08:54 INFO DAGScheduler: Missing parents: List()
17/02/20 10:08:54 INFO DAGScheduler: Submitting ResultStage 36 (MapPartitionsRDD[70] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:54 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 4.0 KB, free 1806.9 MB)
17/02/20 10:08:54 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1806.9 MB)
17/02/20 10:08:54 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:54 INFO SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:54 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 36 (MapPartitionsRDD[70] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:08:54 INFO TaskSchedulerImpl: Adding task set 36.0 with 1 tasks
17/02/20 10:08:54 INFO TaskSetManager: Starting task 0.0 in stage 36.0 (TID 20, localhost, partition 0,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:54 INFO Executor: Running task 0.0 in stage 36.0 (TID 20)
17/02/20 10:08:54 INFO CacheManager: Partition rdd_60_0 not found, computing it
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:54 INFO MemoryStore: Block rdd_60_0 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_60_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO CacheManager: Partition rdd_64_0 not found, computing it
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:54 INFO MemoryStore: Block rdd_64_0 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_64_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO CacheManager: Partition rdd_68_0 not found, computing it
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:54 INFO MemoryStore: Block rdd_68_0 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_68_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO Executor: Finished task 0.0 in stage 36.0 (TID 20). 1851 bytes result sent to driver
17/02/20 10:08:54 INFO TaskSetManager: Finished task 0.0 in stage 36.0 (TID 20) in 20 ms on localhost (1/1)
17/02/20 10:08:54 INFO DAGScheduler: ResultStage 36 (print at WindowReduce.scala:24) finished in 0.022 s
17/02/20 10:08:54 INFO TaskSchedulerImpl: Removed TaskSet 36.0, whose tasks have all completed, from pool 
17/02/20 10:08:54 INFO DAGScheduler: Job 9 finished: print at WindowReduce.scala:24, took 0.037720 s
17/02/20 10:08:54 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:08:54 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 12 is 82 bytes
17/02/20 10:08:54 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 13 is 82 bytes
17/02/20 10:08:54 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 14 is 82 bytes
17/02/20 10:08:54 INFO DAGScheduler: Got job 10 (print at WindowReduce.scala:24) with 3 output partitions
17/02/20 10:08:54 INFO DAGScheduler: Final stage: ResultStage 40 (print at WindowReduce.scala:24)
17/02/20 10:08:54 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 37, ShuffleMapStage 38, ShuffleMapStage 39)
17/02/20 10:08:54 INFO DAGScheduler: Missing parents: List()
17/02/20 10:08:54 INFO DAGScheduler: Submitting ResultStage 40 (MapPartitionsRDD[70] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:08:54 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 4.0 KB, free 1806.9 MB)
17/02/20 10:08:54 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1806.9 MB)
17/02/20 10:08:54 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:54 INFO SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_4_piece0 on localhost:49322 in memory (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:54 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 40 (MapPartitionsRDD[70] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:08:54 INFO TaskSchedulerImpl: Adding task set 40.0 with 3 tasks
17/02/20 10:08:54 INFO TaskSetManager: Starting task 0.0 in stage 40.0 (TID 21, localhost, partition 1,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 5
17/02/20 10:08:54 INFO TaskSetManager: Starting task 1.0 in stage 40.0 (TID 22, localhost, partition 2,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:54 INFO TaskSetManager: Starting task 2.0 in stage 40.0 (TID 23, localhost, partition 3,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:08:54 INFO Executor: Running task 1.0 in stage 40.0 (TID 22)
17/02/20 10:08:54 INFO Executor: Running task 0.0 in stage 40.0 (TID 21)
17/02/20 10:08:54 INFO Executor: Running task 2.0 in stage 40.0 (TID 23)
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:49322 in memory (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 4
17/02/20 10:08:54 INFO CacheManager: Partition rdd_60_2 not found, computing it
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:54 INFO MemoryStore: Block rdd_60_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO CacheManager: Partition rdd_60_1 not found, computing it
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:54 INFO ContextCleaner: Cleaned shuffle 5
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_60_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO CacheManager: Partition rdd_60_3 not found, computing it
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:54 INFO ContextCleaner: Cleaned shuffle 4
17/02/20 10:08:54 INFO CacheManager: Partition rdd_64_2 not found, computing it
17/02/20 10:08:54 INFO MemoryStore: Block rdd_60_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:54 INFO MemoryStore: Block rdd_60_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned shuffle 3
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_60_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_60_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO MemoryStore: Block rdd_64_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO BlockManager: Removing RDD 26
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_64_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO CacheManager: Partition rdd_64_1 not found, computing it
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:54 INFO CacheManager: Partition rdd_68_2 not found, computing it
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:54 INFO CacheManager: Partition rdd_64_3 not found, computing it
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:54 INFO MemoryStore: Block rdd_64_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_64_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO MemoryStore: Block rdd_64_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned RDD 26
17/02/20 10:08:54 INFO MemoryStore: Block rdd_68_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO CacheManager: Partition rdd_68_1 not found, computing it
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_64_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:08:54 INFO BlockManager: Removing RDD 22
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_68_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO MemoryStore: Block rdd_68_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO CacheManager: Partition rdd_68_3 not found, computing it
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:08:54 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:08:54 INFO Executor: Finished task 1.0 in stage 40.0 (TID 22). 1851 bytes result sent to driver
17/02/20 10:08:54 INFO ContextCleaner: Cleaned RDD 22
17/02/20 10:08:54 INFO MemoryStore: Block rdd_68_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_68_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO BlockManagerInfo: Added rdd_68_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO Executor: Finished task 0.0 in stage 40.0 (TID 21). 1851 bytes result sent to driver
17/02/20 10:08:54 INFO BlockManager: Removing RDD 18
17/02/20 10:08:54 INFO Executor: Finished task 2.0 in stage 40.0 (TID 23). 1851 bytes result sent to driver
17/02/20 10:08:54 INFO TaskSetManager: Finished task 1.0 in stage 40.0 (TID 22) in 46 ms on localhost (1/3)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned RDD 18
17/02/20 10:08:54 INFO TaskSetManager: Finished task 2.0 in stage 40.0 (TID 23) in 45 ms on localhost (2/3)
17/02/20 10:08:54 INFO TaskSetManager: Finished task 0.0 in stage 40.0 (TID 21) in 49 ms on localhost (3/3)
17/02/20 10:08:54 INFO TaskSchedulerImpl: Removed TaskSet 40.0, whose tasks have all completed, from pool 
17/02/20 10:08:54 INFO DAGScheduler: ResultStage 40 (print at WindowReduce.scala:24) finished in 0.050 s
-------------------------------------------
Time: 1487556534000 ms
-------------------------------------------


17/02/20 10:08:54 INFO DAGScheduler: Job 10 finished: print at WindowReduce.scala:24, took 0.108602 s
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_2_piece0 on localhost:49322 in memory (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:54 INFO JobScheduler: Finished job streaming job 1487556534000 ms.0 from job set of time 1487556534000 ms
17/02/20 10:08:54 INFO JobScheduler: Total delay: 0.220 s for time 1487556534000 ms (execution: 0.168 s)
17/02/20 10:08:54 INFO MapPartitionsRDD: Removing RDD 56 from persistence list
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 3
17/02/20 10:08:54 INFO BlockManager: Removing RDD 56
17/02/20 10:08:54 INFO PartitionerAwareUnionRDD: Removing RDD 55 from persistence list
17/02/20 10:08:54 INFO ContextCleaner: Cleaned shuffle 2
17/02/20 10:08:54 INFO ContextCleaner: Cleaned shuffle 1
17/02/20 10:08:54 INFO ShuffledRDD: Removing RDD 40 from persistence list
17/02/20 10:08:54 INFO ContextCleaner: Cleaned shuffle 0
17/02/20 10:08:54 INFO BlockManager: Removing RDD 55
17/02/20 10:08:54 INFO BlockManager: Removing RDD 40
17/02/20 10:08:54 INFO ShuffledRDD: Removing RDD 36 from persistence list
17/02/20 10:08:54 INFO BlockManager: Removing RDD 12
17/02/20 10:08:54 INFO ShuffledRDD: Removing RDD 32 from persistence list
17/02/20 10:08:54 INFO ContextCleaner: Cleaned RDD 12
17/02/20 10:08:54 INFO BlockManager: Removing RDD 36
17/02/20 10:08:54 INFO BlockManager: Removing RDD 8
17/02/20 10:08:54 INFO MapPartitionsRDD: Removing RDD 39 from persistence list
17/02/20 10:08:54 INFO BlockManager: Removing RDD 32
17/02/20 10:08:54 INFO BlockManager: Removing RDD 39
17/02/20 10:08:54 INFO MapPartitionsRDD: Removing RDD 35 from persistence list
17/02/20 10:08:54 INFO ContextCleaner: Cleaned RDD 8
17/02/20 10:08:54 INFO BlockManager: Removing RDD 35
17/02/20 10:08:54 INFO MapPartitionsRDD: Removing RDD 31 from persistence list
17/02/20 10:08:54 INFO BlockManager: Removing RDD 4
17/02/20 10:08:54 INFO BlockManager: Removing RDD 31
17/02/20 10:08:54 INFO MapPartitionsRDD: Removing RDD 38 from persistence list
17/02/20 10:08:54 INFO ContextCleaner: Cleaned RDD 4
17/02/20 10:08:54 INFO BlockManager: Removing RDD 38
17/02/20 10:08:54 INFO MapPartitionsRDD: Removing RDD 34 from persistence list
17/02/20 10:08:54 INFO BlockManager: Removing RDD 34
17/02/20 10:08:54 INFO MapPartitionsRDD: Removing RDD 30 from persistence list
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_12_piece0 on localhost:49322 in memory (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 13
17/02/20 10:08:54 INFO BlockManager: Removing RDD 30
17/02/20 10:08:54 INFO BlockRDD: Removing RDD 37 from persistence list
17/02/20 10:08:54 INFO BlockManager: Removing RDD 37
17/02/20 10:08:54 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[37] at socketTextStream at WindowReduce.scala:19 of time 1487556534000 ms
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_11_piece0 on localhost:49322 in memory (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 12
17/02/20 10:08:54 INFO BlockRDD: Removing RDD 33 from persistence list
17/02/20 10:08:54 INFO BlockManager: Removing RDD 33
17/02/20 10:08:54 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[33] at socketTextStream at WindowReduce.scala:19 of time 1487556534000 ms
17/02/20 10:08:54 INFO BlockManagerInfo: Removed input-0-1487556513400 on localhost:49322 in memory (size: 24.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_10_piece0 on localhost:49322 in memory (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:54 INFO BlockRDD: Removing RDD 29 from persistence list
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 11
17/02/20 10:08:54 INFO BlockManagerInfo: Removed input-0-1487556510800 on localhost:49322 in memory (size: 19.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO BlockManager: Removing RDD 29
17/02/20 10:08:54 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[29] at socketTextStream at WindowReduce.scala:19 of time 1487556534000 ms
17/02/20 10:08:54 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1487556514000 ms, 1487556508000 ms, 1487556516000 ms, 1487556512000 ms, 1487556510000 ms)
17/02/20 10:08:54 INFO BlockManagerInfo: Removed input-0-1487556508000 on localhost:49322 in memory (size: 24.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO InputInfoTracker: remove old batch metadata: 1487556514000 ms 1487556512000 ms 1487556510000 ms
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_9_piece0 on localhost:49322 in memory (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 10
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_8_piece0 on localhost:49322 in memory (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 9
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_7_piece0 on localhost:49322 in memory (size: 1620.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 8
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_6_piece0 on localhost:49322 in memory (size: 1623.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 7
17/02/20 10:08:54 INFO BlockManagerInfo: Removed broadcast_5_piece0 on localhost:49322 in memory (size: 1625.0 B, free: 1807.0 MB)
17/02/20 10:08:54 INFO ContextCleaner: Cleaned accumulator 6
17/02/20 10:08:56 INFO ShuffledDStream: Time 1487556536000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 52000 ms
17/02/20 10:08:56 INFO JobScheduler: No jobs added for time 1487556536000 ms
17/02/20 10:08:58 INFO ShuffledDStream: Time 1487556538000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 54000 ms
17/02/20 10:08:58 INFO JobScheduler: No jobs added for time 1487556538000 ms
17/02/20 10:09:00 INFO ShuffledDStream: Time 1487556540000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 56000 ms
17/02/20 10:09:00 INFO JobScheduler: No jobs added for time 1487556540000 ms
17/02/20 10:09:02 INFO ShuffledDStream: Time 1487556542000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 58000 ms
17/02/20 10:09:02 INFO JobScheduler: No jobs added for time 1487556542000 ms
17/02/20 10:09:04 INFO ShuffledDStream: Slicing from 1487556540000 ms to 1487556544000 ms (aligned to 1487556540000 ms and 1487556544000 ms)
17/02/20 10:09:04 INFO JobScheduler: Added jobs for time 1487556544000 ms
17/02/20 10:09:04 INFO JobScheduler: Starting job streaming job 1487556544000 ms.0 from job set of time 1487556544000 ms
17/02/20 10:09:04 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:09:04 INFO DAGScheduler: Registering RDD 81 (map at WindowReduce.scala:22)
17/02/20 10:09:04 INFO DAGScheduler: Registering RDD 77 (map at WindowReduce.scala:22)
17/02/20 10:09:04 INFO DAGScheduler: Registering RDD 73 (map at WindowReduce.scala:22)
17/02/20 10:09:04 INFO DAGScheduler: Got job 11 (print at WindowReduce.scala:24) with 1 output partitions
17/02/20 10:09:04 INFO DAGScheduler: Final stage: ResultStage 44 (print at WindowReduce.scala:24)
17/02/20 10:09:04 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 42, ShuffleMapStage 43, ShuffleMapStage 41)
17/02/20 10:09:04 INFO DAGScheduler: Missing parents: List()
17/02/20 10:09:04 INFO DAGScheduler: Submitting ResultStage 44 (MapPartitionsRDD[84] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:09:04 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 4.0 KB, free 1807.0 MB)
17/02/20 10:09:04 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1807.0 MB)
17/02/20 10:09:04 INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:09:04 INFO SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1006
17/02/20 10:09:04 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 44 (MapPartitionsRDD[84] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:09:04 INFO TaskSchedulerImpl: Adding task set 44.0 with 1 tasks
17/02/20 10:09:04 INFO TaskSetManager: Starting task 0.0 in stage 44.0 (TID 24, localhost, partition 0,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:09:04 INFO Executor: Running task 0.0 in stage 44.0 (TID 24)
17/02/20 10:09:04 INFO CacheManager: Partition rdd_74_0 not found, computing it
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:04 INFO MemoryStore: Block rdd_74_0 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_74_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO CacheManager: Partition rdd_78_0 not found, computing it
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:04 INFO MemoryStore: Block rdd_78_0 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_78_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO CacheManager: Partition rdd_82_0 not found, computing it
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:04 INFO MemoryStore: Block rdd_82_0 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_82_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO Executor: Finished task 0.0 in stage 44.0 (TID 24). 1851 bytes result sent to driver
17/02/20 10:09:04 INFO TaskSetManager: Finished task 0.0 in stage 44.0 (TID 24) in 15 ms on localhost (1/1)
17/02/20 10:09:04 INFO DAGScheduler: ResultStage 44 (print at WindowReduce.scala:24) finished in 0.016 s
17/02/20 10:09:04 INFO TaskSchedulerImpl: Removed TaskSet 44.0, whose tasks have all completed, from pool 
17/02/20 10:09:04 INFO DAGScheduler: Job 11 finished: print at WindowReduce.scala:24, took 0.027560 s
17/02/20 10:09:04 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:09:04 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 15 is 82 bytes
17/02/20 10:09:04 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 16 is 82 bytes
17/02/20 10:09:04 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 17 is 82 bytes
17/02/20 10:09:04 INFO DAGScheduler: Got job 12 (print at WindowReduce.scala:24) with 3 output partitions
17/02/20 10:09:04 INFO DAGScheduler: Final stage: ResultStage 48 (print at WindowReduce.scala:24)
17/02/20 10:09:04 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 45, ShuffleMapStage 46, ShuffleMapStage 47)
17/02/20 10:09:04 INFO DAGScheduler: Missing parents: List()
17/02/20 10:09:04 INFO DAGScheduler: Submitting ResultStage 48 (MapPartitionsRDD[84] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:09:04 INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 4.0 KB, free 1807.0 MB)
17/02/20 10:09:04 INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1807.0 MB)
17/02/20 10:09:04 INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:09:04 INFO SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1006
17/02/20 10:09:04 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 48 (MapPartitionsRDD[84] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:09:04 INFO TaskSchedulerImpl: Adding task set 48.0 with 3 tasks
17/02/20 10:09:04 INFO TaskSetManager: Starting task 0.0 in stage 48.0 (TID 25, localhost, partition 1,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:09:04 INFO TaskSetManager: Starting task 1.0 in stage 48.0 (TID 26, localhost, partition 2,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:09:04 INFO TaskSetManager: Starting task 2.0 in stage 48.0 (TID 27, localhost, partition 3,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:09:04 INFO Executor: Running task 0.0 in stage 48.0 (TID 25)
17/02/20 10:09:04 INFO Executor: Running task 2.0 in stage 48.0 (TID 27)
17/02/20 10:09:04 INFO Executor: Running task 1.0 in stage 48.0 (TID 26)
17/02/20 10:09:04 INFO CacheManager: Partition rdd_74_3 not found, computing it
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:04 INFO MemoryStore: Block rdd_74_3 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO CacheManager: Partition rdd_74_1 not found, computing it
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:04 INFO MemoryStore: Block rdd_74_1 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_74_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_74_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO CacheManager: Partition rdd_78_3 not found, computing it
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:04 INFO CacheManager: Partition rdd_74_2 not found, computing it
17/02/20 10:09:04 INFO CacheManager: Partition rdd_78_1 not found, computing it
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:04 INFO MemoryStore: Block rdd_78_3 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_78_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO MemoryStore: Block rdd_74_2 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO CacheManager: Partition rdd_82_3 not found, computing it
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_74_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:04 INFO MemoryStore: Block rdd_78_1 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO CacheManager: Partition rdd_78_2 not found, computing it
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_78_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:04 INFO MemoryStore: Block rdd_82_3 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO CacheManager: Partition rdd_82_1 not found, computing it
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_82_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:04 INFO MemoryStore: Block rdd_78_2 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_78_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO Executor: Finished task 2.0 in stage 48.0 (TID 27). 1851 bytes result sent to driver
17/02/20 10:09:04 INFO MemoryStore: Block rdd_82_1 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_82_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO CacheManager: Partition rdd_82_2 not found, computing it
17/02/20 10:09:04 INFO TaskSetManager: Finished task 2.0 in stage 48.0 (TID 27) in 29 ms on localhost (1/3)
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:04 INFO Executor: Finished task 0.0 in stage 48.0 (TID 25). 1851 bytes result sent to driver
17/02/20 10:09:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:04 INFO MemoryStore: Block rdd_82_2 stored as bytes in memory (estimated size 4.0 B, free 1807.0 MB)
17/02/20 10:09:04 INFO BlockManagerInfo: Added rdd_82_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:04 INFO TaskSetManager: Finished task 0.0 in stage 48.0 (TID 25) in 32 ms on localhost (2/3)
17/02/20 10:09:04 INFO Executor: Finished task 1.0 in stage 48.0 (TID 26). 1851 bytes result sent to driver
17/02/20 10:09:04 INFO TaskSetManager: Finished task 1.0 in stage 48.0 (TID 26) in 34 ms on localhost (3/3)
17/02/20 10:09:04 INFO DAGScheduler: ResultStage 48 (print at WindowReduce.scala:24) finished in 0.035 s
17/02/20 10:09:04 INFO TaskSchedulerImpl: Removed TaskSet 48.0, whose tasks have all completed, from pool 
17/02/20 10:09:04 INFO DAGScheduler: Job 12 finished: print at WindowReduce.scala:24, took 0.047923 s
17/02/20 10:09:04 INFO JobScheduler: Finished job streaming job 1487556544000 ms.0 from job set of time 1487556544000 ms
17/02/20 10:09:04 INFO MapPartitionsRDD: Removing RDD 70 from persistence list
17/02/20 10:09:04 INFO JobScheduler: Total delay: 0.147 s for time 1487556544000 ms (execution: 0.096 s)
17/02/20 10:09:04 INFO BlockManager: Removing RDD 70
17/02/20 10:09:04 INFO PartitionerAwareUnionRDD: Removing RDD 69 from persistence list
17/02/20 10:09:04 INFO BlockManager: Removing RDD 69
17/02/20 10:09:04 INFO ShuffledRDD: Removing RDD 46 from persistence list
17/02/20 10:09:04 INFO BlockManager: Removing RDD 46
17/02/20 10:09:04 INFO ShuffledRDD: Removing RDD 50 from persistence list
-------------------------------------------
Time: 1487556544000 ms
-------------------------------------------


17/02/20 10:09:04 INFO BlockManager: Removing RDD 50
17/02/20 10:09:04 INFO ShuffledRDD: Removing RDD 54 from persistence list
17/02/20 10:09:04 INFO MapPartitionsRDD: Removing RDD 45 from persistence list
17/02/20 10:09:04 INFO MapPartitionsRDD: Removing RDD 49 from persistence list
17/02/20 10:09:04 INFO BlockManager: Removing RDD 45
17/02/20 10:09:04 INFO BlockManager: Removing RDD 54
17/02/20 10:09:04 INFO MapPartitionsRDD: Removing RDD 53 from persistence list
17/02/20 10:09:04 INFO BlockManager: Removing RDD 49
17/02/20 10:09:04 INFO BlockManager: Removing RDD 53
17/02/20 10:09:04 INFO MapPartitionsRDD: Removing RDD 44 from persistence list
17/02/20 10:09:04 INFO BlockManager: Removing RDD 44
17/02/20 10:09:04 INFO MapPartitionsRDD: Removing RDD 48 from persistence list
17/02/20 10:09:04 INFO BlockManager: Removing RDD 48
17/02/20 10:09:04 INFO MapPartitionsRDD: Removing RDD 52 from persistence list
17/02/20 10:09:04 INFO BlockManager: Removing RDD 52
17/02/20 10:09:04 INFO BlockRDD: Removing RDD 43 from persistence list
17/02/20 10:09:04 INFO BlockManager: Removing RDD 43
17/02/20 10:09:04 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[43] at socketTextStream at WindowReduce.scala:19 of time 1487556544000 ms
17/02/20 10:09:04 INFO BlockRDD: Removing RDD 47 from persistence list
17/02/20 10:09:04 INFO BlockManager: Removing RDD 47
17/02/20 10:09:04 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[47] at socketTextStream at WindowReduce.scala:19 of time 1487556544000 ms
17/02/20 10:09:04 INFO BlockRDD: Removing RDD 51 from persistence list
17/02/20 10:09:04 INFO BlockManager: Removing RDD 51
17/02/20 10:09:04 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[51] at socketTextStream at WindowReduce.scala:19 of time 1487556544000 ms
17/02/20 10:09:04 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1487556524000 ms, 1487556520000 ms, 1487556526000 ms, 1487556522000 ms, 1487556518000 ms)
17/02/20 10:09:04 INFO InputInfoTracker: remove old batch metadata: 1487556520000 ms 1487556522000 ms 1487556524000 ms
17/02/20 10:09:06 INFO ShuffledDStream: Time 1487556546000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 62000 ms
17/02/20 10:09:06 INFO JobScheduler: No jobs added for time 1487556546000 ms
17/02/20 10:09:08 INFO ShuffledDStream: Time 1487556548000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 64000 ms
17/02/20 10:09:08 INFO JobScheduler: No jobs added for time 1487556548000 ms
17/02/20 10:09:08 INFO MemoryStore: Block input-0-1487556548600 stored as bytes in memory (estimated size 20.0 B, free 1807.0 MB)
17/02/20 10:09:08 INFO BlockManagerInfo: Added input-0-1487556548600 in memory on localhost:49322 (size: 20.0 B, free: 1807.0 MB)
17/02/20 10:09:08 INFO BlockGenerator: Pushed block input-0-1487556548600
17/02/20 10:09:10 INFO ShuffledDStream: Time 1487556550000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 66000 ms
17/02/20 10:09:10 INFO JobScheduler: No jobs added for time 1487556550000 ms
17/02/20 10:09:12 INFO ShuffledDStream: Time 1487556552000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 68000 ms
17/02/20 10:09:12 INFO JobScheduler: No jobs added for time 1487556552000 ms
17/02/20 10:09:12 INFO MemoryStore: Block input-0-1487556552000 stored as bytes in memory (estimated size 23.0 B, free 1807.0 MB)
17/02/20 10:09:12 INFO BlockManagerInfo: Added input-0-1487556552000 in memory on localhost:49322 (size: 23.0 B, free: 1807.0 MB)
17/02/20 10:09:12 INFO BlockGenerator: Pushed block input-0-1487556552000
17/02/20 10:09:14 INFO ShuffledDStream: Slicing from 1487556550000 ms to 1487556554000 ms (aligned to 1487556550000 ms and 1487556554000 ms)
17/02/20 10:09:14 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:09:14 INFO JobScheduler: Added jobs for time 1487556554000 ms
17/02/20 10:09:14 INFO JobScheduler: Starting job streaming job 1487556554000 ms.0 from job set of time 1487556554000 ms
17/02/20 10:09:14 INFO DAGScheduler: Registering RDD 95 (map at WindowReduce.scala:22)
17/02/20 10:09:14 INFO DAGScheduler: Registering RDD 91 (map at WindowReduce.scala:22)
17/02/20 10:09:14 INFO DAGScheduler: Registering RDD 87 (map at WindowReduce.scala:22)
17/02/20 10:09:14 INFO DAGScheduler: Got job 13 (print at WindowReduce.scala:24) with 1 output partitions
17/02/20 10:09:14 INFO DAGScheduler: Final stage: ResultStage 52 (print at WindowReduce.scala:24)
17/02/20 10:09:14 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 51, ShuffleMapStage 49, ShuffleMapStage 50)
17/02/20 10:09:14 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 51, ShuffleMapStage 49)
17/02/20 10:09:14 INFO DAGScheduler: Submitting ShuffleMapStage 49 (MapPartitionsRDD[95] at map at WindowReduce.scala:22), which has no missing parents
17/02/20 10:09:14 INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 2.6 KB, free 1807.0 MB)
17/02/20 10:09:14 INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 1623.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added broadcast_16_piece0 in memory on localhost:49322 (size: 1623.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1006
17/02/20 10:09:14 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 49 (MapPartitionsRDD[95] at map at WindowReduce.scala:22)
17/02/20 10:09:14 INFO TaskSchedulerImpl: Adding task set 49.0 with 1 tasks
17/02/20 10:09:14 INFO DAGScheduler: Submitting ShuffleMapStage 51 (MapPartitionsRDD[87] at map at WindowReduce.scala:22), which has no missing parents
17/02/20 10:09:14 INFO TaskSetManager: Starting task 0.0 in stage 49.0 (TID 28, localhost, partition 0,NODE_LOCAL, 2006 bytes)
17/02/20 10:09:14 INFO Executor: Running task 0.0 in stage 49.0 (TID 28)
17/02/20 10:09:14 INFO BlockManager: Found block input-0-1487556552000 locally
17/02/20 10:09:14 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 2.6 KB, free 1806.9 MB)
17/02/20 10:09:14 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 1625.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on localhost:49322 (size: 1625.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006
17/02/20 10:09:14 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 51 (MapPartitionsRDD[87] at map at WindowReduce.scala:22)
17/02/20 10:09:14 INFO TaskSchedulerImpl: Adding task set 51.0 with 1 tasks
17/02/20 10:09:14 INFO TaskSetManager: Starting task 0.0 in stage 51.0 (TID 29, localhost, partition 0,NODE_LOCAL, 2006 bytes)
17/02/20 10:09:14 INFO Executor: Running task 0.0 in stage 51.0 (TID 29)
17/02/20 10:09:14 INFO BlockManager: Found block input-0-1487556548600 locally
17/02/20 10:09:14 INFO Executor: Finished task 0.0 in stage 49.0 (TID 28). 1161 bytes result sent to driver
17/02/20 10:09:14 INFO TaskSetManager: Finished task 0.0 in stage 49.0 (TID 28) in 78 ms on localhost (1/1)
17/02/20 10:09:14 INFO TaskSchedulerImpl: Removed TaskSet 49.0, whose tasks have all completed, from pool 
17/02/20 10:09:14 INFO DAGScheduler: ShuffleMapStage 49 (map at WindowReduce.scala:22) finished in 0.083 s
17/02/20 10:09:14 INFO DAGScheduler: looking for newly runnable stages
17/02/20 10:09:14 INFO DAGScheduler: running: Set(ResultStage 0, ShuffleMapStage 51)
17/02/20 10:09:14 INFO DAGScheduler: waiting: Set(ResultStage 52)
17/02/20 10:09:14 INFO DAGScheduler: failed: Set()
17/02/20 10:09:14 INFO Executor: Finished task 0.0 in stage 51.0 (TID 29). 1161 bytes result sent to driver
17/02/20 10:09:14 INFO TaskSetManager: Finished task 0.0 in stage 51.0 (TID 29) in 82 ms on localhost (1/1)
17/02/20 10:09:14 INFO TaskSchedulerImpl: Removed TaskSet 51.0, whose tasks have all completed, from pool 
17/02/20 10:09:14 INFO DAGScheduler: ShuffleMapStage 51 (map at WindowReduce.scala:22) finished in 0.096 s
17/02/20 10:09:14 INFO DAGScheduler: looking for newly runnable stages
17/02/20 10:09:14 INFO DAGScheduler: running: Set(ResultStage 0)
17/02/20 10:09:14 INFO DAGScheduler: waiting: Set(ResultStage 52)
17/02/20 10:09:14 INFO DAGScheduler: failed: Set()
17/02/20 10:09:14 INFO DAGScheduler: Submitting ResultStage 52 (MapPartitionsRDD[98] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:09:14 INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 4.0 KB, free 1806.9 MB)
17/02/20 10:09:14 INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:09:14 INFO SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1006
17/02/20 10:09:14 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 52 (MapPartitionsRDD[98] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:09:14 INFO TaskSchedulerImpl: Adding task set 52.0 with 1 tasks
17/02/20 10:09:14 INFO TaskSetManager: Starting task 0.0 in stage 52.0 (TID 30, localhost, partition 0,NODE_LOCAL, 2108 bytes)
17/02/20 10:09:14 INFO Executor: Running task 0.0 in stage 52.0 (TID 30)
17/02/20 10:09:14 INFO CacheManager: Partition rdd_88_0 not found, computing it
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:14 INFO MemoryStore: Block rdd_88_0 stored as bytes in memory (estimated size 153.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_88_0 in memory on localhost:49322 (size: 153.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO CacheManager: Partition rdd_92_0 not found, computing it
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:14 INFO MemoryStore: Block rdd_92_0 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_92_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO CacheManager: Partition rdd_96_0 not found, computing it
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:14 INFO MemoryStore: Block rdd_96_0 stored as bytes in memory (estimated size 153.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_96_0 in memory on localhost:49322 (size: 153.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO Executor: Finished task 0.0 in stage 52.0 (TID 30). 2000 bytes result sent to driver
17/02/20 10:09:14 INFO TaskSetManager: Finished task 0.0 in stage 52.0 (TID 30) in 42 ms on localhost (1/1)
17/02/20 10:09:14 INFO DAGScheduler: ResultStage 52 (print at WindowReduce.scala:24) finished in 0.042 s
17/02/20 10:09:14 INFO TaskSchedulerImpl: Removed TaskSet 52.0, whose tasks have all completed, from pool 
17/02/20 10:09:14 INFO DAGScheduler: Job 13 finished: print at WindowReduce.scala:24, took 0.217885 s
17/02/20 10:09:14 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:09:14 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 18 is 146 bytes
17/02/20 10:09:14 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 19 is 82 bytes
17/02/20 10:09:14 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 20 is 146 bytes
17/02/20 10:09:14 INFO DAGScheduler: Got job 14 (print at WindowReduce.scala:24) with 3 output partitions
17/02/20 10:09:14 INFO DAGScheduler: Final stage: ResultStage 56 (print at WindowReduce.scala:24)
17/02/20 10:09:14 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 53, ShuffleMapStage 54, ShuffleMapStage 55)
17/02/20 10:09:14 INFO DAGScheduler: Missing parents: List()
17/02/20 10:09:14 INFO DAGScheduler: Submitting ResultStage 56 (MapPartitionsRDD[98] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:09:14 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 4.0 KB, free 1806.9 MB)
17/02/20 10:09:14 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:09:14 INFO SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1006
17/02/20 10:09:14 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 56 (MapPartitionsRDD[98] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:09:14 INFO TaskSchedulerImpl: Adding task set 56.0 with 3 tasks
17/02/20 10:09:14 INFO TaskSetManager: Starting task 0.0 in stage 56.0 (TID 31, localhost, partition 1,NODE_LOCAL, 2108 bytes)
17/02/20 10:09:14 INFO TaskSetManager: Starting task 1.0 in stage 56.0 (TID 32, localhost, partition 2,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:09:14 INFO TaskSetManager: Starting task 2.0 in stage 56.0 (TID 33, localhost, partition 3,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:09:14 INFO Executor: Running task 2.0 in stage 56.0 (TID 33)
17/02/20 10:09:14 INFO Executor: Running task 0.0 in stage 56.0 (TID 31)
17/02/20 10:09:14 INFO Executor: Running task 1.0 in stage 56.0 (TID 32)
17/02/20 10:09:14 INFO CacheManager: Partition rdd_88_2 not found, computing it
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:14 INFO CacheManager: Partition rdd_88_3 not found, computing it
17/02/20 10:09:14 INFO MemoryStore: Block rdd_88_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO CacheManager: Partition rdd_88_1 not found, computing it
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:14 INFO MemoryStore: Block rdd_88_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_88_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_88_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO MemoryStore: Block rdd_88_1 stored as bytes in memory (estimated size 170.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO CacheManager: Partition rdd_92_2 not found, computing it
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_88_1 in memory on localhost:49322 (size: 170.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:14 INFO CacheManager: Partition rdd_92_3 not found, computing it
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:14 INFO CacheManager: Partition rdd_92_1 not found, computing it
17/02/20 10:09:14 INFO MemoryStore: Block rdd_92_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_92_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO MemoryStore: Block rdd_92_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO MemoryStore: Block rdd_92_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_92_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO CacheManager: Partition rdd_96_2 not found, computing it
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_92_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO CacheManager: Partition rdd_96_1 not found, computing it
17/02/20 10:09:14 INFO CacheManager: Partition rdd_96_3 not found, computing it
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/02/20 10:09:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:14 INFO MemoryStore: Block rdd_96_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_96_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO MemoryStore: Block rdd_96_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO Executor: Finished task 2.0 in stage 56.0 (TID 33). 1851 bytes result sent to driver
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_96_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO MemoryStore: Block rdd_96_1 stored as bytes in memory (estimated size 173.0 B, free 1806.9 MB)
17/02/20 10:09:14 INFO BlockManagerInfo: Added rdd_96_1 in memory on localhost:49322 (size: 173.0 B, free: 1807.0 MB)
17/02/20 10:09:14 INFO Executor: Finished task 1.0 in stage 56.0 (TID 32). 1851 bytes result sent to driver
17/02/20 10:09:14 INFO Executor: Finished task 0.0 in stage 56.0 (TID 31). 2063 bytes result sent to driver
17/02/20 10:09:14 INFO TaskSetManager: Finished task 2.0 in stage 56.0 (TID 33) in 32 ms on localhost (1/3)
17/02/20 10:09:14 INFO TaskSetManager: Finished task 1.0 in stage 56.0 (TID 32) in 35 ms on localhost (2/3)
17/02/20 10:09:14 INFO TaskSetManager: Finished task 0.0 in stage 56.0 (TID 31) in 37 ms on localhost (3/3)
17/02/20 10:09:14 INFO TaskSchedulerImpl: Removed TaskSet 56.0, whose tasks have all completed, from pool 
17/02/20 10:09:14 INFO DAGScheduler: ResultStage 56 (print at WindowReduce.scala:24) finished in 0.038 s
17/02/20 10:09:14 INFO DAGScheduler: Job 14 finished: print at WindowReduce.scala:24, took 0.066221 s
17/02/20 10:09:14 INFO JobScheduler: Finished job streaming job 1487556554000 ms.0 from job set of time 1487556554000 ms
17/02/20 10:09:14 INFO JobScheduler: Total delay: 0.363 s for time 1487556554000 ms (execution: 0.298 s)
17/02/20 10:09:14 INFO MapPartitionsRDD: Removing RDD 84 from persistence list
17/02/20 10:09:14 INFO PartitionerAwareUnionRDD: Removing RDD 83 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 84
17/02/20 10:09:14 INFO ShuffledRDD: Removing RDD 64 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 83
-------------------------------------------
Time: 1487556554000 ms
-------------------------------------------
(edef,2)
(de,1)
(wedfe,1)
(wefef,1)
(ewfef,1)


17/02/20 10:09:14 INFO BlockManager: Removing RDD 64
17/02/20 10:09:14 INFO ShuffledRDD: Removing RDD 60 from persistence list
17/02/20 10:09:14 INFO ShuffledRDD: Removing RDD 68 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 60
17/02/20 10:09:14 INFO MapPartitionsRDD: Removing RDD 63 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 68
17/02/20 10:09:14 INFO MapPartitionsRDD: Removing RDD 59 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 63
17/02/20 10:09:14 INFO BlockManager: Removing RDD 59
17/02/20 10:09:14 INFO MapPartitionsRDD: Removing RDD 67 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 67
17/02/20 10:09:14 INFO MapPartitionsRDD: Removing RDD 62 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 62
17/02/20 10:09:14 INFO MapPartitionsRDD: Removing RDD 58 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 58
17/02/20 10:09:14 INFO MapPartitionsRDD: Removing RDD 66 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 66
17/02/20 10:09:14 INFO BlockRDD: Removing RDD 61 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 61
17/02/20 10:09:14 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[61] at socketTextStream at WindowReduce.scala:19 of time 1487556554000 ms
17/02/20 10:09:14 INFO BlockRDD: Removing RDD 57 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 57
17/02/20 10:09:14 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[57] at socketTextStream at WindowReduce.scala:19 of time 1487556554000 ms
17/02/20 10:09:14 INFO BlockRDD: Removing RDD 65 from persistence list
17/02/20 10:09:14 INFO BlockManager: Removing RDD 65
17/02/20 10:09:14 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[65] at socketTextStream at WindowReduce.scala:19 of time 1487556554000 ms
17/02/20 10:09:14 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1487556536000 ms, 1487556530000 ms, 1487556528000 ms, 1487556534000 ms, 1487556532000 ms)
17/02/20 10:09:14 INFO InputInfoTracker: remove old batch metadata: 1487556532000 ms 1487556530000 ms 1487556534000 ms
17/02/20 10:09:16 INFO ShuffledDStream: Time 1487556556000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 72000 ms
17/02/20 10:09:16 INFO JobScheduler: No jobs added for time 1487556556000 ms
17/02/20 10:09:18 INFO ShuffledDStream: Time 1487556558000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 74000 ms
17/02/20 10:09:18 INFO JobScheduler: No jobs added for time 1487556558000 ms
17/02/20 10:09:20 INFO ShuffledDStream: Time 1487556560000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 76000 ms
17/02/20 10:09:20 INFO JobScheduler: No jobs added for time 1487556560000 ms
17/02/20 10:09:22 INFO ShuffledDStream: Time 1487556562000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 78000 ms
17/02/20 10:09:22 INFO JobScheduler: No jobs added for time 1487556562000 ms
17/02/20 10:09:24 INFO ShuffledDStream: Slicing from 1487556560000 ms to 1487556564000 ms (aligned to 1487556560000 ms and 1487556564000 ms)
17/02/20 10:09:24 INFO JobScheduler: Added jobs for time 1487556564000 ms
17/02/20 10:09:24 INFO JobScheduler: Starting job streaming job 1487556564000 ms.0 from job set of time 1487556564000 ms
17/02/20 10:09:24 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:09:24 INFO DAGScheduler: Registering RDD 109 (map at WindowReduce.scala:22)
17/02/20 10:09:24 INFO DAGScheduler: Registering RDD 105 (map at WindowReduce.scala:22)
17/02/20 10:09:24 INFO DAGScheduler: Registering RDD 101 (map at WindowReduce.scala:22)
17/02/20 10:09:24 INFO DAGScheduler: Got job 15 (print at WindowReduce.scala:24) with 1 output partitions
17/02/20 10:09:24 INFO DAGScheduler: Final stage: ResultStage 60 (print at WindowReduce.scala:24)
17/02/20 10:09:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 57, ShuffleMapStage 58, ShuffleMapStage 59)
17/02/20 10:09:24 INFO DAGScheduler: Missing parents: List()
17/02/20 10:09:24 INFO DAGScheduler: Submitting ResultStage 60 (MapPartitionsRDD[112] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:09:24 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 4.0 KB, free 1806.9 MB)
17/02/20 10:09:24 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1806.9 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:09:24 INFO SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1006
17/02/20 10:09:24 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 60 (MapPartitionsRDD[112] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:09:24 INFO TaskSchedulerImpl: Adding task set 60.0 with 1 tasks
17/02/20 10:09:24 INFO TaskSetManager: Starting task 0.0 in stage 60.0 (TID 34, localhost, partition 0,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:09:24 INFO Executor: Running task 0.0 in stage 60.0 (TID 34)
17/02/20 10:09:24 INFO CacheManager: Partition rdd_102_0 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:24 INFO MemoryStore: Block rdd_102_0 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_102_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO CacheManager: Partition rdd_106_0 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:24 INFO MemoryStore: Block rdd_106_0 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_106_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO CacheManager: Partition rdd_110_0 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:24 INFO MemoryStore: Block rdd_110_0 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_110_0 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO Executor: Finished task 0.0 in stage 60.0 (TID 34). 1851 bytes result sent to driver
17/02/20 10:09:24 INFO TaskSetManager: Finished task 0.0 in stage 60.0 (TID 34) in 24 ms on localhost (1/1)
17/02/20 10:09:24 INFO TaskSchedulerImpl: Removed TaskSet 60.0, whose tasks have all completed, from pool 
17/02/20 10:09:24 INFO DAGScheduler: ResultStage 60 (print at WindowReduce.scala:24) finished in 0.029 s
17/02/20 10:09:24 INFO DAGScheduler: Job 15 finished: print at WindowReduce.scala:24, took 0.061873 s
17/02/20 10:09:24 INFO SparkContext: Starting job: print at WindowReduce.scala:24
17/02/20 10:09:24 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 21 is 82 bytes
17/02/20 10:09:24 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 22 is 82 bytes
17/02/20 10:09:24 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 23 is 82 bytes
17/02/20 10:09:24 INFO DAGScheduler: Got job 16 (print at WindowReduce.scala:24) with 3 output partitions
17/02/20 10:09:24 INFO DAGScheduler: Final stage: ResultStage 64 (print at WindowReduce.scala:24)
17/02/20 10:09:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 63, ShuffleMapStage 61, ShuffleMapStage 62)
17/02/20 10:09:24 INFO DAGScheduler: Missing parents: List()
17/02/20 10:09:24 INFO DAGScheduler: Submitting ResultStage 64 (MapPartitionsRDD[112] at reduceByKeyAndWindow at WindowReduce.scala:22), which has no missing parents
17/02/20 10:09:24 INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 4.0 KB, free 1806.9 MB)
17/02/20 10:09:24 INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 2.2 KB, free 1806.9 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added broadcast_21_piece0 in memory on localhost:49322 (size: 2.2 KB, free: 1807.0 MB)
17/02/20 10:09:24 INFO SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:1006
17/02/20 10:09:24 INFO DAGScheduler: Submitting 3 missing tasks from ResultStage 64 (MapPartitionsRDD[112] at reduceByKeyAndWindow at WindowReduce.scala:22)
17/02/20 10:09:24 INFO TaskSchedulerImpl: Adding task set 64.0 with 3 tasks
17/02/20 10:09:24 INFO TaskSetManager: Starting task 0.0 in stage 64.0 (TID 35, localhost, partition 1,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:09:24 INFO TaskSetManager: Starting task 1.0 in stage 64.0 (TID 36, localhost, partition 2,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:09:24 INFO TaskSetManager: Starting task 2.0 in stage 64.0 (TID 37, localhost, partition 3,PROCESS_LOCAL, 2108 bytes)
17/02/20 10:09:24 INFO Executor: Running task 2.0 in stage 64.0 (TID 37)
17/02/20 10:09:24 INFO Executor: Running task 0.0 in stage 64.0 (TID 35)
17/02/20 10:09:24 INFO Executor: Running task 1.0 in stage 64.0 (TID 36)
17/02/20 10:09:24 INFO CacheManager: Partition rdd_102_1 not found, computing it
17/02/20 10:09:24 INFO CacheManager: Partition rdd_102_3 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:24 INFO CacheManager: Partition rdd_102_2 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:24 INFO MemoryStore: Block rdd_102_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:24 INFO MemoryStore: Block rdd_102_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO MemoryStore: Block rdd_102_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_102_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_102_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_102_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO CacheManager: Partition rdd_106_2 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:24 INFO CacheManager: Partition rdd_106_3 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:24 INFO MemoryStore: Block rdd_106_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO MemoryStore: Block rdd_106_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_106_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_106_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO CacheManager: Partition rdd_106_1 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:24 INFO CacheManager: Partition rdd_110_2 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:24 INFO MemoryStore: Block rdd_106_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_106_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO CacheManager: Partition rdd_110_3 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/02/20 10:09:24 INFO MemoryStore: Block rdd_110_3 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO CacheManager: Partition rdd_110_1 not found, computing it
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 0 blocks
17/02/20 10:09:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_110_3 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO MemoryStore: Block rdd_110_2 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO MemoryStore: Block rdd_110_1 stored as bytes in memory (estimated size 4.0 B, free 1806.9 MB)
17/02/20 10:09:24 INFO Executor: Finished task 2.0 in stage 64.0 (TID 37). 1851 bytes result sent to driver
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_110_2 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO BlockManagerInfo: Added rdd_110_1 in memory on localhost:49322 (size: 4.0 B, free: 1807.0 MB)
17/02/20 10:09:24 INFO TaskSetManager: Finished task 2.0 in stage 64.0 (TID 37) in 32 ms on localhost (1/3)
17/02/20 10:09:24 INFO Executor: Finished task 0.0 in stage 64.0 (TID 35). 1851 bytes result sent to driver
17/02/20 10:09:24 INFO TaskSetManager: Finished task 0.0 in stage 64.0 (TID 35) in 36 ms on localhost (2/3)
17/02/20 10:09:24 INFO Executor: Finished task 1.0 in stage 64.0 (TID 36). 1851 bytes result sent to driver
17/02/20 10:09:24 INFO TaskSetManager: Finished task 1.0 in stage 64.0 (TID 36) in 39 ms on localhost (3/3)
17/02/20 10:09:24 INFO TaskSchedulerImpl: Removed TaskSet 64.0, whose tasks have all completed, from pool 
17/02/20 10:09:24 INFO DAGScheduler: ResultStage 64 (print at WindowReduce.scala:24) finished in 0.041 s
17/02/20 10:09:24 INFO DAGScheduler: Job 16 finished: print at WindowReduce.scala:24, took 0.065333 s
17/02/20 10:09:24 INFO JobScheduler: Finished job streaming job 1487556564000 ms.0 from job set of time 1487556564000 ms
17/02/20 10:09:24 INFO JobScheduler: Total delay: 0.196 s for time 1487556564000 ms (execution: 0.151 s)
17/02/20 10:09:24 INFO MapPartitionsRDD: Removing RDD 98 from persistence list
17/02/20 10:09:24 INFO BlockManager: Removing RDD 98
17/02/20 10:09:24 INFO PartitionerAwareUnionRDD: Removing RDD 97 from persistence list
-------------------------------------------
Time: 1487556564000 ms
-------------------------------------------


17/02/20 10:09:24 INFO BlockManager: Removing RDD 97
17/02/20 10:09:24 INFO ShuffledRDD: Removing RDD 78 from persistence list
17/02/20 10:09:24 INFO ShuffledRDD: Removing RDD 74 from persistence list
17/02/20 10:09:24 INFO BlockManager: Removing RDD 78
17/02/20 10:09:24 INFO ShuffledRDD: Removing RDD 82 from persistence list
17/02/20 10:09:24 INFO BlockManager: Removing RDD 74
17/02/20 10:09:24 INFO BlockManager: Removing RDD 82
17/02/20 10:09:24 INFO MapPartitionsRDD: Removing RDD 77 from persistence list
17/02/20 10:09:24 INFO MapPartitionsRDD: Removing RDD 73 from persistence list
17/02/20 10:09:24 INFO BlockManager: Removing RDD 77
17/02/20 10:09:24 INFO MapPartitionsRDD: Removing RDD 81 from persistence list
17/02/20 10:09:24 INFO BlockManager: Removing RDD 73
17/02/20 10:09:24 INFO BlockManager: Removing RDD 81
17/02/20 10:09:24 INFO MapPartitionsRDD: Removing RDD 76 from persistence list
17/02/20 10:09:24 INFO BlockManager: Removing RDD 76
17/02/20 10:09:24 INFO MapPartitionsRDD: Removing RDD 72 from persistence list
17/02/20 10:09:24 INFO MapPartitionsRDD: Removing RDD 80 from persistence list
17/02/20 10:09:24 INFO BlockManager: Removing RDD 72
17/02/20 10:09:24 INFO BlockManager: Removing RDD 80
17/02/20 10:09:24 INFO BlockRDD: Removing RDD 75 from persistence list
17/02/20 10:09:24 INFO BlockManager: Removing RDD 75
17/02/20 10:09:24 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[75] at socketTextStream at WindowReduce.scala:19 of time 1487556564000 ms
17/02/20 10:09:24 INFO BlockRDD: Removing RDD 71 from persistence list
17/02/20 10:09:24 INFO BlockManager: Removing RDD 71
17/02/20 10:09:24 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[71] at socketTextStream at WindowReduce.scala:19 of time 1487556564000 ms
17/02/20 10:09:24 INFO BlockRDD: Removing RDD 79 from persistence list
17/02/20 10:09:24 INFO BlockManager: Removing RDD 79
17/02/20 10:09:24 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[79] at socketTextStream at WindowReduce.scala:19 of time 1487556564000 ms
17/02/20 10:09:24 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1487556538000 ms, 1487556546000 ms, 1487556542000 ms, 1487556544000 ms, 1487556540000 ms)
17/02/20 10:09:24 INFO InputInfoTracker: remove old batch metadata: 1487556542000 ms 1487556540000 ms 1487556544000 ms
17/02/20 10:09:26 INFO ShuffledDStream: Time 1487556566000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 82000 ms
17/02/20 10:09:26 INFO JobScheduler: No jobs added for time 1487556566000 ms
17/02/20 10:09:28 INFO ShuffledDStream: Time 1487556568000 ms is invalid as zeroTime is 1487556484000 ms and slideDuration is 10000 ms and difference is 84000 ms
17/02/20 10:09:28 INFO JobScheduler: No jobs added for time 1487556568000 ms


Process finished with exit code -1
0 0
ubuntu 界面 ubuntu 界面
原创粉丝点击
热门IT博客
c语言可视化游戏c语言可视化游戏
淘宝动销率计算公式淘宝动销率计算公式
ubuntu 软件安装
ipad保护套 知乎
crf python
抄板软件quickpcb
mysql语句生成器
d3.js 在线拓扑图
杭州做公卫软件
用淘宝充话费多久到账
大学专业书籍阅读软件
天龙八部3挂机软件
北京代办韩国签证 知乎
手机淘宝插件
stc单片机原理图库
淘宝账号
玄冰400和t400 知乎
耽美小说软件
威海seo
屏幕数据记录器
热门问题 老师的惩罚 人脸识别 我在镇武司摸鱼那些年 重生之率土为王 我在大康的咸鱼生活 盘龙之生命进化 天生仙种 凡人之先天五行 春回大明朝 姑娘不必设防，我是瞎子 六小岭童年龄去世2017 小童咳嗽怎么办 小哪吒之魔童降世经典台词 娱乐玩童 小玩童 老玩童西西中国正版 娱乐玩童 金色火烧云 玫瑰小精灵选自哪个童话 他的小玫瑰 热吻小玫瑰 玫瑰花怎么养 热吻玫瑰 小妖精再浪些再咬紧些玫瑰 小玫瑰怎么养 小玫瑰花 网王之第一网球 叶小钗 小王和珊全文免费阅读 温柔的交换小王他老婆珊 海绵宝宝珊迪同人全彩 海绵宝宝珊迪 海绵宝宝h珊迪全彩 海绵宝宝在干珊迪的同人动画 小珠山 小珠山风景区 小珠山野生动物园 小珠山门票 小珠山门票多少钱 小珠山野生动物园门票 应彩云小班优质课 幼儿园小班科学课 幼儿园小班消防课 英语 小班课 学大教育小班课 小球藻片 小球藻价格 小球藻培养 蛋白核小球藻减肥吗 小球藻的价格 小球藻的功效与作用 小球藻粉 蛋白核小球藻的功效

程序博客网，程序员的互联网技术博客家园。csdn论坛精品 msdn技术资料都在这里