Sqoop 1.99.3 安装

1. 1.安装准备工作：  
2. 已经装好的hadoop环境是hadoop 2.2.0  
3. 下载的sqoop安装包（注意是hadoop200）  
4. http://www.us.apache.org/dist/sqoop/1.99.3/sqoop-1.99.3-bin-hadoop200.tar.gz  
5. 2.解压文件到工作目录:  
6. hadoop@hadoopMaster:$ sudo tar -xvf /opt/hn/hadoop_family/sqoop-1.99.3-bin-hadoop200.tar.gz   
7. hadoop@hadoopMaster:mv /opt/hn/hadoop_family/sqoop-1.99.3-bin-hadoop200 /usr/local/sqoop  
8. 3.修改环境变量：  
9. hadoop@hadoopMaster:~$ vim /etc/profile  
10. 添加如下内容：  
11. #sqoop  
12. export SQOOP_HOME=/usr/local/sqoop  
13. export PATH=$SQOOP_HOME/bin:$PATH  
14. export CATALINA_HOME=$SQOOP_HOME/server  
15. export LOGDIR=$SQOOP_HOME/logs  
16. 保存退出即时生效：  
17. source /etc/profile  
18. 4.修改sqoop配置：  
19. hadoop@hadoopMaster:~$ vim /usr/local/sqoop/server/conf/sqoop.properties   
20. #修改指向我的hadoop安装目录  
21. org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/  
22. #把hadoop目录下的jar包都引进来  
23. hadoop@hadoopMaster:~$ vim /usr/local/sqoop/server/conf/catalina.properties   
24. common.loader=/usr/local/hadoop/share/hadoop/common/*.jar,/usr/local/hadoop/share/hadoop/common/lib/*.jar,/usr/local/hadoop/share/hadoop/hdfs/*.jar,/usr/local/hadoop/share/hadoop/hdfs/lib/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/lib/*.jar,/usr/local/hadoop/share/hadoop/tools/*.jar,/usr/local/hadoop/share/hadoop/tools/lib/*.jar,/usr/local/hadoop/share/hadoop/yarn/*.jar,/usr/local/hadoop/share/hadoop/yarn/lib/*.jar,/usr/local/hadoop/share/hadoop/httpfs/tomcat/lib/*.jar  
25. 5.下载mysql驱动包  
26. mysql-connector-java-5.1.16-bin.jar  
27. 6.启动/停止sqoop200  
28. hadoop@hadoopMaster:/usr/local/sqoop/bin$ ./sqoop.sh server start/stop  
29. 查看启动日志：  
30. hadoop@hadoopMaster:/usr/local/sqoop/server/logs$ vim catalina.out   
31. 7.进入客户端交互目录  
32. hadoop@hadoopMaster:/usr/local/sqoop/bin$ ./sqoop.sh client  
33. +------------------------------------------+  
34. |Sqoop home directory: /usr/local/sqoop    |  
35. |Sqoop Shell: Type 'help' or '\h' for help.|  
36. |sqoop:000>                                |  
37. +------------------------------------------+  
38. 为客户端配置服务器：  
39. +---------------------------------------------------------------------+  
40. |sqoop:000> set server --host hadoopMaster --port 12000 --webapp sqoop|  
41. |Server is set successfully                                           |  
42. +---------------------------------------------------------------------+  
43. 查版本信息：  
44. +-----------------------------------------------------------------+  
45. |sqoop:000> show version --all                                    |  
46. |client version:                                                  |  
47. |  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b |  
48. |  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013           |  
49. |server version:                                                  |  
50. |  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b |  
51. |  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013           |  
52. |Protocol version:                                                |  
53. |  [1]                                                            |  
54. +-----------------------------------------------------------------+  
55. 显示连接器：  
56. +---------------------------------------------------------------------------------------------+  
57. |sqoop:000> show connector --all                                                              |  
58. |1 connector(s) to show:                                                                      |  
59. |Connector with id 1:                                                                         |  
60. |  Name: generic-jdbc-connector                                                               |  
61. | Class: org.apache.sqoop.connector.jdbc.GenericJdbcConnector                                 |  
62. |  Version: 1.99.3                                                                            |  
63. |  Supported job types: [IMPORT, EXPORT]                                                      |  
64. |    Connection form 1:                                                                       |  
65. |      Name: connection                                                                       |  
66. |      Label: Connection configuration                                                        |  
67. |      Help: You must supply the information requested in order to create a connection object.|  
68. |      Input 1:                                                                               |  
69. |        .                                                                                    |  
70. |        .                                                                                    |  
71. |        .                                                                                    |  
72. |   太长了，就拷贝这一点                                                                       |  
73. +---------------------------------------------------------------------------------------------+  
74. 创建数据库连接：  
75. +---------------------------------------------------------------------------------------------+  
76. |sqoop:000> create connection --cid 1                                                         |  
77. |Creating connection for connector with id 1                                                  |  
78. |Please fill following values to create new connection object                                 |  
79. |Name: My first                                                                               |  
80. |                                                                                             |  
81. |Connection configuration                                                                     |  
82. |                                                                                             |  
83. |JDBC Driver Class: com.mysql.jdbc.Driver                                                     |  
84. |JDBC Connection String: jdbc:mysql://localhost:3306/sqoop_stu                                |  
85. |Username: root                                                                               |  
86. |Password: **********                                                                         |  
87. |JDBC Connection Properties:                                                                  |  
88. |There are currently 0 values in the map:                                                     |  
89. |entry#                                                                                       |  
90. |                                                                                             |  
91. |Security related configuration options                                                       |  
92. |                                                                                             |  
93. |Max connections: 100                                                                         |  
94. |New connection was successfully created with validation status FINE and persistent id 1      |  
95. +---------------------------------------------------------------------------------------------+  
96. 创建导入任务  
97. +------------------------------------------------------------------------------------+  
98. |sqoop:001> create job --xid 1 --type import                                         |  
99. |Creating job for connection with id 1                                               |  
100. |Please fill following values to create new job object                               |  
101. |Name: First job                                                                     |  
102. |                                                                                    |  
103. |Database configuration                                                              |  
104. |                                                                                    |  
105. |Schema name: traceweb                                                               |  
106. |Table name: trace_web_application                                                   |  
107. |Table SQL statement:                                                                |  
108. |Table column names:                                                                 |  
109. |Partition column name:                                                              |  
110. |Nulls in partition column:                                                          |  
111. |Boundary query:                                                                     |  
112. |                                                                                    |  
113. |Output configuration                                                                |  
114. |                                                                                    |  
115. |Storage type:                                                                       |  
116. |  0 : HDFS                                                                          |  
117. |Choose: 0                                                                           |  
118. |Output format:                                                                      |  
119. |  0 : TEXT_FILE                                                                     |  
120. |  1 : SEQUENCE_FILE                                                                 |  
121. |Choose: 1                                                                           |  
122. |Compression format:                                                                 |  
123. |  0 : NONE                                                                          |  
124. |  1 : DEFAULT                                                                       |  
125. |  2 : DEFLATE                                                                       |  
126. |  3 : GZIP                                                                          |  
127. |  4 : BZIP2                                                                         |  
128. |  5 : LZO                                                                           |  
129. |  6 : LZ4                                                                           |  
130. |  7 : SNAPPY                                                                        |  
131. |Choose: 0                                                                           |  
132. |Output directory: /opt/sqoop_output                                                 |  
133. |                                                                                    |  
134. |Throttling resources                                                                |  
135. |                                                                                    |  
136. |Extractors:                                                                         |  
137. |Loaders:                                                                            |  
138. |New job was successfully created with validation status FINE  and persistent id 1   |  
139. +------------------------------------------------------------------------------------+  
140. 启动job：  
141. +------------------------------------------------  
142. |sqoop:000>  start job --jid 1    
143. +------------------------------------------------  
144.   
145. 查看导入状态：  
146. +------------------------------------------------  
147. |sqoop:000>  status job --jid 1    
148. |Submission details  
149. |Job ID: 1  
150. |Server URL: http://hadoopMaster:12000/sqoop/  
151. |Created by: hadoop  
152. |Creation date: 2014-05-23 18:51:05 CST  
153. |Lastly updated by: hadoop  
154. |External ID: job_local1566994033_0001  
155. |   http://localhost:8080/  
156. |2014-05-23 18:51:35 CST: UNKNOWN   
157. +------------------------------------------------  
158.   
159. 查看输出目录：  
160. +--------------------------------------------------------------------+  
161. hadoop@hadoopMaster:~$ l /opt/sqoop_output/  
162. 总用量 92  
163. drwxrwxr-x 2 hadoop hadoop 4096  5月 23 18:52 .  
164. drwxr-xr-x 8 hadoop hadoop 4096  5月 23 18:51 ..  
165. -rw-r--r-- 1 hadoop hadoop  209  5月 23 18:51 part-m-00000.seq  
166. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00000.seq.crc  
167. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00001.seq  
168. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00001.seq.crc  
169. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00002.seq  
170. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00002.seq.crc  
171. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00003.seq  
172. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00003.seq.crc  
173. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00004.seq  
174. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00004.seq.crc  
175. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00005.seq  
176. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00005.seq.crc  
177. -rw-r--r-- 1 hadoop hadoop  207  5月 23 18:51 part-m-00006.seq  
178. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00006.seq.crc  
179. -rw-r--r-- 1 hadoop hadoop   86  5月 23 18:51 part-m-00007.seq  
180. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00007.seq.crc  
181. -rw-r--r-- 1 hadoop hadoop  206  5月 23 18:51 part-m-00008.seq  
182. -rw-rw-r-- 1 hadoop hadoop   12  5月 23 18:51 .part-m-00008.seq.crc  
183. -rw-r--r-- 1 hadoop hadoop  682  5月 23 18:51 part-m-00009.seq  
184. -rw-rw-r-- 1 hadoop hadoop   16  5月 23 18:51 .part-m-00009.seq.crc  
185. -rw-r--r-- 1 hadoop hadoop    0  5月 23 18:51 _SUCCESS  
186. -rw-rw-r-- 1 hadoop hadoop    8  5月 23 18:51 ._SUCCESS.crc  
187. +--------------------------------------------------------------------  
188. sqoop:000> show job  
189. +----+------------+--------+-----------+---------+  
190. | Id |    Name    |  Type  | Connector | Enabled |  
191. +----+------------+--------+-----------+---------+  
192. | 1  | First job  | IMPORT | 1         | true    |  
193. | 2  | importHDFS | IMPORT | 1         | true    |  
194. +----+------------+--------+-----------+---------+  
195. sqoop:000> delete job --jid 1  
196. sqoop:000> show job            
197. +----+------------+--------+-----------+---------+  
198. | Id |    Name    |  Type  | Connector | Enabled |  
199. +----+------------+--------+-----------+---------+  
200. | 2  | importHDFS | IMPORT | 1         | true    |  
201. +----+------------+--------+-----------+---------+  
202. sqoop:000> delete job --jid 2  
203. sqoop:000> show job  
204. +----+------+------+-----------+---------+  
205. | Id | Name | Type | Connector | Enabled |  
206. +----+------+------+-----------+---------+  
207. +----+------+------+-----------+---------+  
208.   
209. sqoop:000> show connection  
210.   
211. 批处理模式：  
212. sqoop.sh client /opt/sqoop/script.sqoop  
213. hadoop@hadoopMaster:$ vim  /opt/sqoop/script.sqoop  
214. #指定服务器信息  
215. set server --host hadoopMaster --port 12000 --webapp sqoop  
216. #执行JOB  
217. start job --jid 1   
218. +--------------------------------------------------------------------+  
219. hadoop@hadoopMaster:/usr/local/sqoop/bin$ ./sqoop.sh client /opt/hadoop/mysql/batchModel.sqoop   
220. Sqoop home directory: /usr/local/sqoop  
221. sqoop:000> set server --host hadoopMaster --port 12000 --webapp sqoop  
222. Server is set successfully  
223. sqoop:000> start job --jid 1  
224. Submission details  
225. Job ID: 1  
226. Server URL: http://hadoopMaster:12000/sqoop/  
227. Created by: hadoop  
228. Creation date: 2014-05-30 10:55:10 CST  
229. Lastly updated by: hadoop  
230. External ID: job_local945860799_0003  
231.     http://localhost:8080/  
232. 2014-05-30 10:55:10 CST: BOOTING  - Progress is not available  
233. +--------------------------------------------------------------------+  
234.   
235.   
236.   
237. https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Quickstart#Sqoop2Quickstart-Fullimportdemo  
238.   
239. ================================MYSQL=======================================  
240. hadoop@hadoopMaster:~$ mysql -uroot -pjava  
241. mysql> create database sqoop_stu;  
242. Query OK, 1 row affected (0.03 sec)  
243. mysql> use sqoop_stu;  
244. Database changed  
245. mysql> create table student(id int(3) auto_increment not null primary key, name char(10) not null, address varchar(50));  
246. Query OK, 0 rows affected (0.41 sec)  
247. mysql> insert into student values(1, 'Tom','beijing'),(2, 'Joan','shanghai'), (3, 'Wang', 'shenzheng');  
248. Query OK, 3 rows affected (0.07 sec)  
249. Records: 3  Duplicates: 0  Warnings: 0  
250.   
251.   
252.   
253.   
254.   
255. CREATE TABLE `demo_blog` (`id` int(11) NOT NULL AUTO_INCREMENT,  `blog` varchar(100) NOT NULL,  PRIMARY KEY (`id`)) ENGINE=MyISAM  DEFAULT CHARSET=utf8;  
256. CREATE TABLE `demo_log` (`operator` varchar(16) NOT NULL, `log` varchar(100) NOT NULL) ENGINE=MyISAM  DEFAULT CHARSET=utf8;  
257.   
258.   
259.   
260.   
261.   
262.   
263. https://hbase.apache.org/book/configuration.html#hadoop  
264. http://www.tuicool.com/articles/NVfEVnn