转自：http://blog.csdn.net/xuguokun1986/article/details/50032063
案例说明1：
步骤一：将测试数据放到hdfs上面
hadoopdfs -put data1/test1.txt /tmp/test1.txt
该测试数据有两行文本：
步骤二：在Spark中创建一个RDD来读取hdfs文件/tmp/test1.txt
步骤三：查看map函数的返回值
得到map函数返回的RDD:
查看map函数的返回值——文件中的每一行数据返回了一个数组对象
步骤四：查看flatMap函数的返回值
得到flatMap函数返回的RDD:
查看flatMap函数的返回值——文件中的所有行数据仅返回了一个数组对象
总结：
- Spark 中 map函数会对每一条输入进行指定的操作，然后为每一条输入返回一个对象；
- 而flatMap函数则是两个操作的集合——正是“先映射后扁平化”：
操作1：同map函数一样：对每一条输入进行指定的操作，然后为每一条输入返回一个对象
操作2：最后将所有对象合并为一个对象
案例说明2：
假设存在如下文件：
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@bluejoe0</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># cat categories.csv </span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,生活用品,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>,数码用品,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>,手机,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>,华为<span class="hljs-constant" style="box-sizing: border-box;">Mate7</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>
每一行的格式为：类别ID，类别名称，父类ID
现在欲输出每个类别的父类别的名称，类似于SQL的自连接，注意到join的外键其实是父类ID。
首先生成“父类ID->子类ID，子类名称”
<code class="hljs coffeescript has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">val categories=sc.textFile(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/root/categories.csv"</span>) val left = categories.map<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(_.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span>))</span>.<span class="hljs-title" style="box-sizing: border-box;">map</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(x=>(x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)->Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"id"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))))</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>
left的内容为：
Array((0,Map(id -> 1, name -> 生活用品)), (1,Map(id -> 2, name -> 数码用品)), (2,Map(id -> 3, name -> 手机)), (3,Map(id -> 4, name -> 华为Mate7)))
接着生成“父类ID->父类ID，父类名称”
<code class="hljs coffeescript has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">val right = categories.map<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(_.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span>))</span>.<span class="hljs-title" style="box-sizing: border-box;">map</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(x=>(x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>)->Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pid"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pname"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))))</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
right的内容为：
Array((1,Map(pid -> 1, pname -> 生活用品)), (2,Map(pid -> 2, pname -> 数码用品)), (3,Map(pid -> 3, pname -> 手机)), (4,Map(pid -> 4, pname -> 华为Mate7)))
接下来，将这两个RDD进行合并，并按照key（key都是父类ID）进行reduce：
<code class="hljs brainfuck has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">val</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">merged</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">=</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">(left</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">+</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">+</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">right)</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">reduceByKey(_</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">+</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">+</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">_)</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
merged的内容为：
Array((4,Map(pid -> 4, pname -> 华为Mate7)), (0,Map(id -> 1, name -> 生活用品)), (1,Map(id -> 2, name -> 数码用品, pid -> 1, pname -> 生活用品)), (2,Map(id -> 3, name -> 手机, pid -> 2, pname -> 数码用品)), (3,Map(id -> 4, name -> 华为Mate7, pid -> 3, pname -> 手机)))
搞定！！
可以采用flatMap来简化以上的写法：
<code class="hljs coffeescript has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">val merged = categories.map<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(_.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span>))</span>.<span class="hljs-title" style="box-sizing: border-box;">flatMap</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">((x)=>Array(x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)->Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"id"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)), x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>)->Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pid"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>),<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pname"</span>->x(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))))</span>.<span class="hljs-title" style="box-sizing: border-box;">reduceByKey</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(_++_)</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
结果是一样一样的！！当然代码的可读性大打折扣了~~~
hkl曰：
看了书之后再补充一下基础的内容，原谅我直接上的截图，因为工作时间比较宝贵，就不手敲了。
补充一下：看下两者的区别
![]()
![]()
使用map以后Array()里还是一个个Array(),
但是使用flatMap的结果就是将所有的输出分区合并为一个。
![]()
![]()
