Spark和Hive集成使用时mysql驱动包无法加载解决办法

来源：互联网发布：python经纬度画轨迹图编辑：程序博客网时间：2024/06/06 19:09

我们在处理离线数据分析时，一般做法是用正则等处理数据，把最终结果保存到Hive仓库。Spark应用程序调用Hive，用Sprk sql进行查询分析。
第一次集成时，报无法找到合适的驱动错误：

Caused by: java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true        at java.sql.DriverManager.getConnection(DriverManager.java:596)        at java.sql.DriverManager.getConnection(DriverManager.java:187)        at org.datanucleus.store.rdbms.datasource.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)        at org.datanucleus.store.rdbms.datasource.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)        at org.datanucleus.store.rdbms.datasource.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1158)        at org.datanucleus.store.rdbms.datasource.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:108)        at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:501)        at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:298)        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)        at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)        at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)

经检查，mysql驱动包已经放到spark目录下，并且使用 –jars参数加载：

bin/spark-shell  --jars  externaljars/mysql-connector-java-5.1.27-bin.jar   --master local[2]

分析原因，可能是–jars没有加载上驱动，改成 –driver-class-path加载：

bin/spark-shell  --jars  externaljars/mysql-connector-java-5.1.27-bin.jar   --master local[2]

问题解决，启动成功。

阅读全文

0 0