Hive on Spark: Getting Started

Spark Installation
Configuring Yarn
Configuring Hive
Configuring Spark
Common Issues (Green are resolved, will be removed from this list)
Recommended Configuration
Design documents

Hive on Spark is now part of Hive since Hive 1.1 release. It is still under active development in "spark" branch, and is periodically merged into the "master" branch for Hive. See HIVE-7292 and its subtasks and linked issues.

Spark Installation

Follow instructions to install Spark: http://spark.apache.org/docs/latest/running-on-yarn.html (or https://spark.apache.org/docs/latest/spark-standalone.html, if you are running Spark Standalone mode). Hive on Spark supports Spark on Yarn mode as default. In particular, for the installation you'll need to:

Install Spark (either download pre-built Spark, or build assembly from source).
- Install/build a compatible version. Hive root pom.xml's <spark.version> defines what version of Spark it was built/tested with.
- Install/build a compatible distribution. Each version of Spark has several distributions, corresponding with different versions of Hadoop.
- Once Spark is installed, find and keep note of the <spark-assembly-*.jar> location.
- Note that you must have a version of Spark which does not include the Hive jars. Meaning one which was not built with the Hive profile. To remove Hive jars from the installation, simply use the following command under your Spark repository:
  ./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.4"
Start Spark cluster (both standalone and Spark on YARN are supported).
- Keep note of the <Spark Master URL>. This can be found in Spark master WebUI.

Configuring Yarn

yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

Configuring Hive

There are several ways to add the Spark dependency to Hive:
1. Set the property 'spark.home' to point to the Spark installation:
  hive> set spark.home=/location/to/sparkHome;
2. Define the SPARK_HOME environment variable before starting Hive CLI/HiveServer2:
  export SPARK_HOME=/usr/lib/spark....
3. Link the spark-assembly jar to HIVE_HOME/lib.
Configure Hive execution to Spark:
hive> set hive.execution.engine=spark;
See the Spark section of Hive Configuration Properties for other properties for configuring Hive and the Remote Spark Driver.
Configure Spark-application configs for Hive. See: http://spark.apache.org/docs/latest/configuration.html. This can be done either by adding a file "spark-defaults.conf" with these properties to the Hive classpath, or by setting them on Hive configuration (hive-site.xml). For instance:
hive> set spark.master=<Spark Master URL>

hive> set spark.eventLog.enabled=true;

hive> set spark.eventLog.dir=<Spark event log folder (must exist)>

hive> set spark.executor.memory=512m;

hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;
A little explanation for some of the configuration properties:
- spark.executor.memory: Amount of memory to use per executor process.
- spark.executor.cores: Number of cores per executor.
- spark.yarn.executor.memoryOverhead: The amount of off heap memory (in megabytes) to be allocated per executor, when running Spark on Yarn. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. In addition to the executor's memory, the container in which the executor is launched needs some extra memory for system processes, and this is what this overhead is for.
- spark.executor.instances: The number of executors assigned to each application.
- spark.driver.memory: The amount of memory assigned to the Remote Spark Context (RSC). We recommend 4GB.
- spark.yarn.driver.memoryOverhead: We recommend 400 (MB).

Configuring Spark

Setting executor memory size is more complicated than simply setting it to be as large as possible. There are several things that need to be taken into consideration:

More executor memory means it can enable mapjoin optimization for more queries.
More executor memory, on the other hand, becomes unwieldy from GC perspective.
Some experiments shows that HDFS client doesn’t handle concurrent writers well, so it may face race condition if executor cores are too many.

When running Spark on Yarn mode, we generally recommend setting spark.executor.cores to be 5, 6 or 7, depending on what the typical node is divisible by. For instance, if yarn.nodemanager.resource.cpu-vcores is 19, then 6 is a better choice (all executors can only have the same number of cores, here if we chose 5, then every executor only gets 3 cores; if we chose 7, then only 2 executors are used, and 5 cores will be wasted). If it’s 20, then 5 is a better choice (since this way you’ll get 4 executors, and no core is wasted).

For spark.executor.memory, we recommend to calculate yarn.nodemanager.resource.memory-mb * (spark.executor.cores / yarn.nodemanager.resource.cpu-vcores) then split that between spark.executor.memory and spark.yarn.executor.memoryOverhead. According to our experiment, we recommend setting spark.yarn.executor.memoryOverhead to be around 15-20% of the total memory.

After you’ve decided on how much memory each executor receives, you need to decide how many executors will be allocated to queries. In the GA release Spark dynamic executor allocation will be supported. However for this beta only static resource allocation can be used. Based on the physical memory in each node and the configuration of spark.executor.memory and spark.yarn.executor.memoryOverhead, you will need to choose the number of instances and set spark.executor.instances.

Now a real world example. Assuming 10 nodes with 64GB of memory per node with 12 virtual cores, e.g., yarn.nodemanager.resource.cpu-vcores=12. One node will be used as the master and as such the cluster will have 9 slave nodes. We’ll configure spark.executor.cores to 6. Given 64GB of ram yarn.nodemanager.resource.memory-mb will be 50GB. We’ll determine the amount of memory for each executor as follows: 50GB * (6/12) = 25GB. We’ll assign 20% to spark.yarn.executor.memoryOverhead, or 5120, and 80% to spark.executor.memory, or 20GB.

On this 9 node cluster we’ll have two executors per host. As such we can configure spark.executor.instances somewhere between 2 and 18. A value of 18 would utilize the entire cluster.

Common Issues (Green are resolved, will be removed from this list)

Issue

Cause

Resolution

Error: Could not find or load main class org.apache.spark.deploy.SparkSubmitSpark dependency not correctly set.Add Spark dependency to Hive, see Step 1 above.

org.apache.spark.SparkException: Job aborted due to stage failure:

Task 5.0:0 had a not serializable result: java.io.NotSerializableException: org.apache.hadoop.io.BytesWritable

Spark serializer not set to Kryo.Set spark.serializer to be org.apache.spark.serializer.KryoSerializer, see Step 3 above.

[ERROR] Terminal initialization failed; falling back to unsupported
java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected

Hive has upgraded to Jline2 but jline 0.94 exists in the Hadoop lib.

Delete jline from the Hadoop lib directory (it's only pulled in transitively from ZooKeeper).
export HADOOP_USER_CLASSPATH_FIRST=true
If this error occurs during mvn test, perform a mvn clean install on the root project and itests directory.

Spark executor gets killed all the time and Spark keeps retrying the failed stage; you may find similar information in the YARN nodemanager log.

WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=217989,containerID=container_1421717252700_0716_01_50767235] is running beyond physical memory limits. Current usage: 43.1 GB of 43 GB physical memory used; 43.9 GB of 90.3 GB virtual memory used. Killing container.

For Spark on YARN, nodemanager would kill Spark executor if it used more memory than the configured size of "spark.executor.memory" + "spark.yarn.executor.memoryOverhead".Increase "spark.yarn.executor.memoryOverhead" to make sure it covers the executor off-heap memory usage.

Run query and get an error like:

FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask

In Hive logs, it shows:

java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)

Happens on Mac (not officially supported).

This is a general Snappy issue with Mac and is not unique to Hive on Spark, but workaround is noted here because it is needed for startup of Spark client.

Run this command before starting Hive or HiveServer2:

export HADOOP_OPTS="-Dorg.xerial.snappy.tempdir=/tmp -Dorg.xerial.snappy.lib.name=libsnappyjava.jnilib $HADOOP_OPTS"

Stack trace: ExitCodeException exitCode=1: .../launch_container.sh: line 27: $PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR.../usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:$PWD/__app__.jar:$PWD/*: bad substitution

The key mapreduce.application.classpath in/etc/hadoop/conf/mapred-site.xml contains a variable which is invalid in bash.

From mapreduce.application.classpath remove

:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar

from

/etc/hadoop/conf/mapred-site.xml

Exception in thread "Driver" scala.MatchError: java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/TaskAttemptContext (of class java.lang.NoClassDefFoundError)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:432)

MR is not on the YARN classpath.

If on HDP change from

/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework

to

/hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz#mr-framework

Recommended Configuration

# see HIVE-9153mapreduce.input.fileinputformat.split.maxsize=750000000hive.vectorized.execution.enabled=truehive.cbo.enable=truehive.optimize.reducededuplication.min.reducer=4hive.optimize.reducededuplication=truehive.orc.splits.include.file.footer=falsehive.merge.mapfiles=truehive.merge.sparkfiles=falsehive.merge.smallfiles.avgsize=16000000hive.merge.size.per.task=256000000hive.merge.orcfile.stripe.level=truehive.auto.convert.join=truehive.auto.convert.join.noconditionaltask=truehive.auto.convert.join.noconditionaltask.size=894435328hive.optimize.bucketmapjoin.sortedmerge=falsehive.map.aggr.hash.percentmemory=0.5hive.map.aggr=truehive.optimize.sort.dynamic.partition=falsehive.stats.autogather=truehive.stats.fetch.column.stats=truehive.vectorized.execution.reduce.enabled=falsehive.vectorized.groupby.checkinterval=4096hive.vectorized.groupby.flush.percent=0.1hive.compute.query.using.stats=truehive.limit.pushdown.memory.usage=0.4hive.optimize.index.filter=truehive.exec.reducers.bytes.per.reducer=67108864hive.smbjoin.cache.rows=10000hive.exec.orc.default.stripe.size=67108864hive.fetch.task.conversion=morehive.fetch.task.conversion.threshold=1073741824hive.fetch.task.aggr=falsemapreduce.input.fileinputformat.list-status.num-threads=5spark.kryo.referenceTracking=falsespark.kryo.classesToRegister=org.apache.hadoop.hive.ql.io.HiveKey,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch

Design documents

Hive on Spark: Overall Design from HIVE-7272
Hive on Spark: Join Design (HIVE-7613)
Hive on Spark Configuration (HIVE-9449)
Hive on Spark Explain Plan