Spark-sql与hive的结合环境配置
来源:互联网 发布:淘宝回收q币是真的吗 编辑:程序博客网 时间:2024/05/21 03:24
转:zx老师
########################################
alter database hive character set latin1;ALTER TABLE hive.* DEFAULT CHARACTER SET latin1;
########################################
windows安装mysql时候,选的字符编码为utf-8, hive只认识latin1。 也可以在新建数据库时候,选择编码格式。
1.安装hive
CREATE USER 'hive'@'%' IDENTIFIED BY '123456';
GRANT all privileges ON hive.* TO 'hive'@'%';
flush privileges;
2.将配置好的hive-site.xml放入$SPARK-HOME/conf目录下
3.启动spark-shell时指定mysql连接驱动位置
bin/spark-shell \
--master spark://node1.itcast.cn:7077 \
--executor-memory 1g \
--total-executor-cores 2 \
--driver-class-path /usr/local/apache-hive-0.13.1-bin/lib/mysql-connector-java-5.1.35-bin.jar
4.使用sqlContext.sql调用HQL
sqlContext.sql("select * from spark.person limit 2")
或使用org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.HiveContext
val hiveContext = new HiveContext(sc)
hiveContext.sql("select * from spark.person")
hive.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.--><configuration><property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://172.16.0.1:3306/hive?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description></property><property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description></property><property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> <description>username to use against metastore database</description></property><property> <name>javax.jdo.option.ConnectionPassword</name> <value>123456</value> <description>password to use against metastore database</description></property></configuration>
阅读全文
0 0
- Spark-sql与hive的结合环境配置
- spark-sql 结合 hive
- spark 2.0.0与HIVE结合的测试
- Spark SQL与Hive的关系
- Spark SQL on Hive配置
- 查看Ambari自动配置的hive与spark sql元数据库
- spark sql与hive整合
- spark+hive+hdfs windows开发环境配置:
- spark+hive win7开发环境配置
- spark sql on hive配置及其使用
- 2.2、配置Spark-sql(连接Hive)
- Spark与Hadoop的结合
- spark与lucene的结合
- spark基础之Spark SQL和Hive的集成以及ThriftServer配置
- Spark SQL 与 Spark SQL on Hive 区别
- Spark与Hive的交互
- 015-Spark SQL与 Hive集成
- spark-sql部署实现与Hive交互
- mybatis一直报空指针异常
- linux命令整理
- mysql 防止主键冲突报错-ignore
- 微服务架构综述
- Ubuntu配置ShadowSocks-Qt5开机启动
- Spark-sql与hive的结合环境配置
- Spring Boot (教程三:使用devtools实现热部署 )
- Linux启动顺序、运行级别及开机启动
- 调试系列2:bugreport实战篇
- 一阶线性微分方程
- Unity入门之更改鼠标图标、添加天空盒子、添加浓雾
- python爬网站20170606(re,write,urlopen)
- Java transient关键字使用小记
- [SVN 同步] org.apache.subversion.javahl.ClientException: Unsupported working copy forma