sqoop1.99.3安装与教程
来源:互联网 发布:什么叫软件开发 编辑:程序博客网 时间:2024/05/16 14:09
测试环境hadoop版本2.4.0,sqoop版本1.99.3,其他高版本1.99.4、1.99.5均安装失败。
安装文档 http://sqoop.apache.org/docs/1.99.3/Installation.html
1.下载sqoop-1.99.3-bin-hadoop200.tar.gz
2.解压 tar -xvf sqoop-<version>-bin-hadoop<hadoop-version>.tar.gz
3.配置文件/home/hadoop/sqoop/server/conf
catalina.properties
修改:(依赖jar)
common.loader=common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar,/usr/lib/hadoop/lib/*.jar,/usr/lib/hadoop-hdfs/*.jar,/usr/lib/hadoop-hdfs/lib/*.jar,/usr/lib/hadoop-mapreduce/*.jar,/usr/lib/hadoop-mapreduce/lib/*.jar,/usr/lib/hadoop-yarn/*.jar,/usr/lib/hadoop-yarn/lib/*.jar
sqoop.properties(设置hadoop配置文件路径)
修改
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/
4.添加数据库驱动包
mysql-connector-java-5.1.26-bin.jar --mysql
ojdbc6.jar --oracle
4.启动 /bin/sqoop.sh server start
停止 ./bin/sqoop.sh server stop
5.添加服务端
set server --host 132.77.210.201 --port 12000 --webapp sqoop
6.验证:
-> ./sqoop.sh client
截图
7. 常用命令: 注意 --jid 表示 job ID --xid 表示 连接ID
create connection --cid 1 --创建连接
delete connection --xid 1
update connection --xid 1
create job --xid 1 --type import 创建job 从数据库导入hdfs --xid 1 表示连接id
create job --xid 1 --type export 创建job 从hdfs导入数据库
status job --jid 2 --查看job状态
start job --jid 2 --执行job
update job -jid 2 --修改job
show job --显示job
delete job --jid 4 --删除job --jid 4 要删除的job id
8.链接oracle:
sqoop:000> create connection --cid 1
Creating connection for connector with id 1
Please fill following values to create new connection object
Name: wgh_test
Connection configuration
JDBC Driver Class: oracle.jdbc.driver.OracleDriver
JDBC Connection String: jdbc:oracle:thin:@132.77.119.185:1528:wghdbts1
Username: wgh
Password: ***
JDBC Connection Properties:
There are currently 0 values in the map:
entry#
Security related configuration options
Max connections: 10
New connection was successfully created with validation status FINE and persistent id 2
9.修改jop/新建
sqoop:000> update job --jid 1 --修改 -- create job --xid 1 --type import --创建 从数据库导入hdfs
Updating job with id 1
Please update job metadata:
Name: first_job
Database configuration
Schema name: wgh --数据库用户名
Table name: eval_template
Table SQL statement:
Table column names:EVAL_TEMPLATE_ID,TEMPLATE_NAME --字段
Partition column name:
Nulls in partition column:
Boundary query:
Output configuration
Storage type:
0 : HDFS
Choose: 0
Output format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
Choose: 0
Compression format:
0 : NONE
1 : DEFAULT
2 : DEFLATE
3 : GZIP
4 : BZIP2
5 : LZO
6 : LZ4
7 : SNAPPY
Choose: 0
Output directory: /test --hdfs文件目录
Throttling resources
Extractors:
Loaders:
Job was successfully updated with status FINE
--创建导入job,从hdfs导入oracle
sqoop:000> create job --xid 1 --type export
Creating job for connection with id 1
Please fill following values to create new job object
Name: export_eval
Database configuration
Schema name: wgh
Table name: eval_template_001
Table SQL statement:
Table column names: EVAL_TEMPLATE_ID,TEMPLATE_NAME
Stage table name:
Clear stage table:
Input configuration
Input directory: /test2
Throttling resources
Extractors:
Loaders:
New job was successfully created with validation status FINE and persistent id 4
sqoop:000> show job
+----+--------------------------+--------+-----------+---------+
| Id | Name | Type | Connector | Enabled |
+----+--------------------------+--------+-----------+---------+
| 1 | first_job | IMPORT | 1 | true |
| 2 | export_vip_customer_info | EXPORT | 1 | true |
| 4 | export_eval | EXPORT | 1 | true |
+----+--------------------------+--------+-----------+---------+
sqoop:000>
日志目录:/home/hadoop/sqoop/@LOGDIR@
server启动日志目录:/home/hadoop/sqoop/server/logs
0 0
- sqoop1.99.3安装与教程
- Sqoop1.99.3安装
- sqoop1.99.3的安装
- sqoop1.99.4安装与应用
- Sqoop2学习(一)—Sqoop1.99.3介绍与安装
- Sqoop1 安装
- Sqoop1.4.6安装与使用(一)
- Sqoop1.4.6安装与使用(二)
- Hadoop2.2.0环境下Sqoop1.99.3安装
- sqoop1安装过程
- sqoop1.4.6安装
- Sqoop1.4.6安装指南
- Sqoop1.99.7安装配置
- sqoop1.4.6安装
- Ubuntu安装Sqoop1
- sqoop1.99.7安装
- sqoop1.4.6安装部署
- sqoop1.99.6安装
- Java的内存机制
- JS绘制数学函数图
- 开发日记(2)drawable和bitmap和byte[]的相互转换
- maven 添加自己的包到本地仓库及到项目中
- 测试序列的特点
- sqoop1.99.3安装与教程
- C++作业4.17
- elasticsearchservicewrapper安装
- 进程间通信之WM_COPYDATA方式反思,回顾和总结
- c语言实现将一个整数转换为字符串
- URI和URL的区别
- Trick Collection
- 【java】微信支付生成签名的过程
- 语音转换成文字