mapreduce.{map|reduce}.java.opts vs. mapreduce.{map|reduce}.memory.mb

In Hadoop 2, tasks are run within containers launched by YARN. 
mapreduce.{map|reduce}.memory.mb is used by YARN to set the memory size of the container being used to run the map or reduce task. If the task grows beyond this limit, YARN will kill the container.


To execute the actual map or reduce task, YARN will run a JVM within the container. The hadoop property mapreduce.{map|reduce}.java.opts is intended to pass options to this JVM. This could include ­Xmx to set max heap size of the JVM. However, the subsequent growth in the memory footprint of the JVM due to the settings in mapreduce.{map|reduce}.java.opts is limited by the actual size of the container as set by mapreduce.{map|reduce}.memory.mb.


Consequently, you should ensure that the heap you specify in {map|reduce}.java.opts is set to be less than than the memory specified by {map|reduce}.memory.mb.


For example:


  hadoop jar <jarName> -Dmapreduce.reduce.memory.mb=4096 -Dmapreduce.map.java.opts=-Xmx3276