spark代码sc统一配置

之前写了两篇关于mlib的应用，其实都是在一个project内的，在创建sc的过程中，可以通过继承统一的trait简化sc的创建。

val postgprop = new Properties  val path = this.getClass.getResourceAsStream("/config.properties")

读取相关配置文件

key.name=valuejdbcURL=jdbc:mysql://**********:3306/******recResultTable_fpGrowth=ods_fpgrowth_dsrecResultTable_als=ods_commender_dsmysqlusername=*******mysqlpassword=*******localClusterURL=local[2]clusterMasterURL=spark://namenode:7077mode=spark://namenode:7077minSupport=0.2minConfidence=0.8numPartitions=2

通过读取的配置文件的内容，创建sc

/配置sparkconf  val mode = postgprop.getProperty("mode")  val conf = new SparkConf().setMaster(mode)  val sc = new SparkContext(conf)  val sqlContext = new SQLContext(sc)

可以在里边直接创建sqlcontext
mysql的相关配置也可以直接在trait里边写

 //jdbc连接  val jdbcURL = postgprop.getProperty("jdbcURL")  val recResultTable_fpGrowth = postgprop.getProperty("recResultTable_fpGrowth")  val recResultTable_als = postgprop.getProperty("recResultTable_als")  val mysqlusername = postgprop.getProperty("mysqlusername")  val mysqlpassword = postgprop.getProperty("mysqlpassword")  val prop = new Properties  prop.put("driver", "com.mysql.jdbc.Driver")  prop.put("user", mysqlusername)  prop.put("password", mysqlpassword)