超详细讲解Sqoop2部署过程

摘要:超详细讲解Sqoop2部署过程，1.下载Sqoop2,增加Sqoop2变量，修改Sqoop2配置文件，默认已经安装好了hadoop。另外提供Sqoop2自动部署源码

1.下载解压sqoop2

tar -zxf sqoop-1.99.7-bin-hadoop200.tar.gzsudo mv sqoop-1.99.7-bin-hadoop200 /opt/

2.配置变量

2.1.设置hadoop环境变量

sudo vim /etc/profile.d/hadoop-2.7.3

#!/bin/sh# Author:wangxiaolei 王小雷# Blog: http://blog.csdn.net/dream_an# Github: https://github.com/wangxiaoleiai# Date: 20170729# Path: /etc/profile.d/# Organization: https://github.com/whaleaiexport HADOOP_HOME=/opt/hadoop-2.7.3export PATH=$HADOOP_HOME/bin:$PATHexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

2.2.设置sqoop2环境变量

sudo vim /etc/profile.d/sqoop-1.99.7-bin-hadoop200.sh

# Author:wangxiaolei 王小雷# Blog: http://blog.csdn.net/dream_an# Github: https://github.com/wangxiaoleiai# Date: 201707# Organization: https://github.com/whaleaiexport SQOOP_HOME=/opt/sqoop-1.99.7-bin-hadoop200export PATH=$SQOOP_HOME/bin:$PATHeexport SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/lib

2.3.修改文件sqoop.properties

xiaolei@wang:~$ vim /opt/sqoop-1.99.7-bin-hadoop200/conf/sqoop.properties

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/opt/hadoop-2.7.3/etc/hadoop

2.4.修改hadoop的mapred-site.xml文件

<property>    <name>mapreduce.jobhistory.address</name>    <value>localhost:10020</value></property>

3.sqoop2元初始化和安装验证

3.1.元数据初始化

# 进入到sqoop目录是因为初始化会生成@BASEDIR@和@LOGDIR@xiaolei@wang:~$ cd /opt/sqoop-1.99.7-bin-hadoop200/

# 初始化xiaolei@wang:/opt/sqoop-1.99.7-bin-hadoop200$ sqoop2-tool upgrade

3.2.验证sqoop2-tool verify

xiaolei@wang:/opt/sqoop-1.99.7-bin-hadoop200$ sqoop2-tool verify

3.3.开启server

xiaolei@wang:/opt/sqoop-1.99.7-bin-hadoop200$ sqoop2-server start

3.4.通过jps可以看到SqoopJettyServery已经启动

6026 SqoopJettyServer6077 Jps

3.5. 如何应用超详细讲解Sqoop2应用与实践

4.关闭Sqoop2

sqoop2-server stop

5.添加第三方jdbc

mkdir /opt/sqoop-1.99.7/libwget http://central.maven.org/maven2/org/postgresql/postgresql/42.1.1/postgresql-42.1.1.jarmv postgresql-42.1.1.jar /opt/sqoop-1.99.7/lib/

添加环境变量

export SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/sqoop-1.99.7/lib

Sqoop2自动部署源码