hadoop集群之间迁移分区表

来源：互联网发布：jmeter响应数据乱码编辑：程序博客网时间：2024/06/05 10:29

这里集群的分区表是指的hive/impala表, 表存储格式是parquet.

迁移的时候是指文件的拷贝。下面我做一个案例演示。如果有大量的表要迁移，可以写一个java程序，多线程控制。

1.查看源集群的表位置

[root@slave01 ~]# hadoop fs -du -h /user/hive/warehouse/prestat.db/dt_differ_users_pre_xdr299.0 K  896.9 K  /user/hive/warehouse/prestat.db/dt_differ_users_pre_xdr/day=20170601[root@slave01 ~]#

2.把源集群的文件down到源服务器上

[root@slave01 ~]# hadoop fs -get /user/hive/warehouse/prestat.db/dt_differ_users_pre_xdr/day=20170601/minute=0000 /root

3.把文件通过ftp down到本机或者scp到目标集群。

[root@slave01 ~]# hadoop fs -put /root/dt_differ_users_pre_xdr/ /user/hive/warehouse/prestat.db/[root@slave01 ~]# hadoop fs -du -h /user/hive/warehouse/prestat.db^C[root@slave01 ~]# hadoop fs -du -h /user/hive/warehouse/prestat.db/dt_differ_users_pre_xdr159.7 M  159.7 M  /user/hive/warehouse/prestat.db/dt_differ_users_pre_xdr/day=201709010        0        /user/hive/warehouse/prestat.db/dt_differ_users_pre_xdr/day=201710010        0        /user/hive/warehouse/prestat.db/dt_differ_users_pre_xdr/day=20171101

4.在目标集群建立该表的分区

20170901 有数据，我们创建该月分区（我直接在impala执行的，避免元数据没有同步刷新）

[slave01:21000] >  alter table prestat.dt_differ_users_pre_xdr add IF NOT EXISTS partition(day=20170901,minute=cast('0000' as char(4)));Query: alter table prestat.dt_differ_users_pre_xdr add IF NOT EXISTS partition(day=20170901,minute=cast('0000' as char(4)))Fetched 0 row(s) in 1.63s[slave01:21000] > show partitions prestat.dt_differ_users_pre_xdr ;Query: show partitions prestat.dt_differ_users_pre_xdr+----------+--------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------------------------+| day      | minute | #Rows | #Files | Size     | Bytes Cached | Cache Replication | Format  | Incremental stats | Location                                                                                    |+----------+--------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------------------------+| 20170901 | 0000   | -1    | 2      | 159.72MB | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://myha/user/hive/warehouse/prestat.db/dt_differ_users_pre_xdr/day=20170901/minute=0000 || Total    |        | -1    | 2      | 159.72MB | 0B           |                   |         |                   |                                                                                             |+----------+--------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------------------------+Fetched 2 row(s) in 0.03s[slave01:21000] >

5.查看数据

[slave01:21000] > select day,minute,count(*) from prestat.dt_differ_users_pre_xdr group by day,minute;Query: select day,minute,count(*) from prestat.dt_differ_users_pre_xdr group by day,minuteQuery submitted at: 2017-12-18 16:07:34 (Coordinator: http://slave01:25000)Query progress can be monitored at: http://slave01:25000/query_plan?query_id=124365f0388d9b90:2af6db6900000000+----------+--------+----------+| day      | minute | count(*) |+----------+--------+----------+| 20170901 | 0000   | 3235946  |+----------+--------+----------+Fetched 1 row(s) in 0.14s[slave01:21000] >

6.如果查询没有数据，可以执行invalidate metadata xxx; refresh xxx;

[slave01:21000] > invalidate metadata  prestat.dt_differ_users_pre_xdr;Query: invalidate metadata  prestat.dt_differ_users_pre_xdrQuery submitted at: 2017-12-18 16:09:17 (Coordinator: http://slave01:25000)Query progress can be monitored at: http://slave01:25000/query_plan?query_id=2242885efbd4c93d:a0ea048500000000Fetched 0 row(s) in 0.42s[slave01:21000] > refresh prestat.dt_differ_users_pre_xdr;Query: refresh prestat.dt_differ_users_pre_xdrQuery submitted at: 2017-12-18 16:09:24 (Coordinator: http://slave01:25000)Query progress can be monitored at: http://slave01:25000/query_plan?query_id=f5483ec91c70dd59:fc2a589f00000000Fetched 0 row(s) in 0.67s[slave01:21000] >

阅读全文

0 0