RDD创建的两种方式

There are two ways to create RDDs: parallelizing an existing collection in your driver program, or referencing a dataset in an external storage system, such as a shared filesystem, HDFS, HBase, or any data source offering a Hadoop InputFormat.

第一种创建方式：并行化的方式val data = Array(1, 2, 3, 4, 5)val distData = sc.parallelize(data)

分为几个RDD和系统的核数是有关的核数越多效率越好。

Once created, the distributed dataset (distData) can beoperated on in parallel.For example, we might call distData.reduce((a, b) => a + b)to add up the elements of the array. We describe operations on distributed datasets later on.One important parameter for parallel collections is the number of partitions to cut the dataset into. Spark willrun one task for each partition of the cluster. Typically you want 2-4 partitions for each CPU in your cluster. Normally,Spark tries to set the number of partitions automatically based on your cluster. However, you can also set it manually by passing it as a second parameter to parallelize (e.g. sc.parallelize(data, 10)).Note: some places in the code use the term slices (a synonym for partitions) to maintain backward compatibility.

第二种创建方式：通过读取外部文件的方式（External Datasets）scala> val distFile = sc.textFile("data.txt")distFile: org.apache.spark.rdd.RDD[String] = data.txt MapPartitionsRDD[10] 

Spark can create distributed datasets from any storage source supported by Hadoop, including yourlocal file system, HDFS, Cassandra, HBase, Amazon S3, etc. Spark supports text files, SequenceFiles, and any other Hadoop InputFormat.