spark-1.6.3最新版改动

来源：互联网发布：centos ntfs 挂载编辑：程序博客网时间：2024/05/21 03:59

spark-1.6.3最新版改动

Sub-task(子任务)

[SPARK-16488] - Codegen variable namespace collision for pmod and partitionBy
[SPARK-16491] - Crc32 should use different variable names (not "checksum")
[SPARK-16514] - RegexExtract and RegexReplace crash on non-nullable input

Bug（修复Bug）

[SPARK-6005] - Flaky test: o.a.s.streaming.kafka.DirectKafkaStreamSuite.offset recovery
[SPARK-11301] - filter on partitioned column is case sensitive even the context is case insensitive
[SPARK-14209] - Application failure during preemption.
[SPARK-15541] - SparkContext.stop throws error
[SPARK-15606] - Driver hang in o.a.s.DistributedSuite on 2 core machine
[SPARK-16044] - input_file_name() returns empty strings in data sources based on NewHadoopRDD.
[SPARK-16077] - Python UDF may fail because of six
[SPARK-16078] - from_utc_timestamp/to_utc_timestamp may give different result in different timezone
[SPARK-16182] - Utils.scala -- terminateProcess() should call Process.destroyForcibly() if and only if Process.destroy() fails
[SPARK-16257] - spark-ec2 script not updated for 1.6.2 release
[SPARK-16313] - Spark should not silently drop exceptions in file listing
[SPARK-16353] - Intended javadoc options are not honored for Java unidoc
[SPARK-16375] - [Spark web UI]:The wrong value(numCompletedTasks) has been assigned to the variable numSkippedTasks
[SPARK-16385] - NoSuchMethodException thrown by Utils.waitForProcess
[SPARK-16409] - regexp_extract with optional groups causes NPE
[SPARK-16422] - maven 3.3.3 missing from mirror, breaks older builds
[SPARK-16440] - Undeleted broadcast variables in Word2Vec causing OoM for long runs
[SPARK-16489] - Test harness to prevent expression code generation from reusing variable names
[SPARK-16656] - CreateTableAsSelectSuite is flaky
[SPARK-16664] - Spark 1.6.2 - Persist call on Data frames with more than 200 columns is wiping out the data.
[SPARK-16751] - Upgrade derby to 10.12.1.1 from 10.11.1.1
[SPARK-16873] - force spill NPE
[SPARK-16925] - Spark tasks which cause JVM to exit with a zero exit code may cause app to hang in Standalone mode
[SPARK-16939] - Fix build error by using `Tuple1` explicitly in StringFunctionSuite
[SPARK-17003] - release-build.sh is missing hive-thriftserver for scala 2.11
[SPARK-17038] - StreamingSource reports metrics for lastCompletedBatch instead of lastReceivedBatch
[SPARK-17245] - NPE thrown by ClientWrapper.conf
[SPARK-17356] - A large Metadata filed in Alias can cause OOM when calling TreeNode.toJSON
[SPARK-17404] - [BRANCH-1.6] Broken test: showDF in test_sparkSQL.R
[SPARK-17418] - Spark release must NOT distribute Kinesis related assembly artifact
[SPARK-17465] - Inappropriate memory management in `org.apache.spark.storage.MemoryStore` may lead to memory leak
[SPARK-17531] - Don't initialize Hive Listeners for the Execution Client
[SPARK-17547] - Temporary shuffle data files may be leaked following exception in write
[SPARK-17617] - Remainder(%) expression.eval returns incorrect result
[SPARK-17618] - Dataframe except returns incorrect results when combined with coalesce
[SPARK-17678] - Spark 1.6 Scala-2.11 repl doesn't honor "spark.replClassServer.port"
[SPARK-17696] - Race in CoarseGrainedExecutorBackend shutdown can lead to wrong exit status
[SPARK-17721] - Erroneous computation in multiplication of transposed SparseMatrix with SparseVector
[SPARK-17884] - In the cast expression, casting from empty string to interval type throws NullPointerException

Improvement(改进)

[SPARK-2424] - ApplicationState.MAX_NUM_RETRY should be configurable
[SPARK-15761] - pyspark shell should load if PYSPARK_DRIVER_PYTHON is ipython an Python3
[SPARK-16341] - [SQL] In regexp_replace function column and/or column expression should also allowed as replacement.
[SPARK-16796] - Visible passwords on Spark environment page
[SPARK-17316] - Don't block StandaloneSchedulerBackend.executorRemoved
[SPARK-17378] - Upgrade snappy-java to 1.1.2.6
[SPARK-17485] - Failed remote cached block reads can lead to whole job failure
[SPARK-17649] - Log how many Spark events got dropped in LiveListenerBus

New Feature(新特征)

[SPARK-16956] - Make ApplicationState.MAX_NUM_RETRY configurable

0 0