利用eclipse编写自定义hive udf函数

来源：互联网发布：火狐浏览器 mac 编辑：程序博客网时间：2024/05/22 07:57

在做日志分析的过程中，用到了hadoop框架中的hive，不过有些日志处理用hive中的函数处理显得力不从心，就需要用udf来进行扩展处理了

1 在eclipse中新建java project hiveudf 然后新建class package(com.afan) name(UDFLower)

2 添加jar library hadoop-0.20.2-core.jar hive-exec-0.7.0-cdh3u0.jar两个文件到project

3 编写代码

package com.afan;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;
public class UDFLower extends UDF{
public Text evaluate(final Text s){
if (null == s){
return null;
}
return new Text(s.toString().toLowerCase());
}
}

4 编译输出打包文件为 udf_hive.jar

5 将udf_hive.jar放入配置好的linux系统的文件夹中路径为/home/udf/udf_hive.jar

6 打开hive命令行测试

hive> add jar /home/udf/udf_hive.jar;

Added udf_hive.jar to class path
Added resource: udf_hive.jar

创建udf函数
hive> create temporary function my_lower as 'com.afan.UDFLower';

创建测试数据
hive> create table dual (info string);

导入数据文件data.txt

data.txt文件内容为

WHO

HELLO

hive> load data local inpath '/home/data/data.txt' into table dual;

hive> select info from dual;

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201105150525_0003, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201105150525_0003
Kill Command = /usr/local/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:9001 -kill job_201105150525_0003
2011-05-15 06:46:05,459 Stage-1 map = 0%,  reduce = 0%
2011-05-15 06:46:10,905 Stage-1 map = 100%,  reduce = 0%
2011-05-15 06:46:13,963 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201105150525_0003
OK
WHO
AM
I
HELLO

使用udf函数
hive> select my_lower(info) from dual;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201105150525_0002, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201105150525_0002
Kill Command = /usr/local/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=localhost:9001 -kill job_201105150525_0002
2011-05-15 06:43:26,100 Stage-1 map = 0%,  reduce = 0%
2011-05-15 06:43:34,364 Stage-1 map = 100%,  reduce = 0%
2011-05-15 06:43:37,484 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201105150525_0002
OK
who
am
i
hello

经测试成功通过

参考文章http://landyer.iteye.com/blog/1070377

——————————————————————————————————

1、编写函数

package com.example.hive.udf;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;
public final class LowerCase extends UDF {
public Text evaluate(final Text s) {
if (s == null) { return null; }
return new Text(s.toString().toLowerCase());
}
}

2、用eclipse下的fatjar插件进行打包

先下载net.sf.fjep.fatjar_0.0.31.jar插件包，cp至eclipse/plugins目录下，重启eclipse，右击项目选Export，选择用fatjar导出(可以删掉没用的包，不然导出的jar包很大)

3、将导出的hiveudf.jar复制到hdfs上

hadoop fs -copyFromLocal hiveudf.jar hiveudf.jar

4、进入hive，添加jar，

add jar hdfs://localhost:9000/user/root/hiveudf.jar

5、创建一个临时函数

create temporary function my_lower as 'com.example.hive.udf.LowerCase';

6、调用

select LowerCase(name) from teacher;

注：这种方法只能添加临时的函数，每次重新进入hive的时候都要再执行4-6，要使得这个函数永久生效，要将其注册到hive的函数列表

添加函数文件$HIVE_HOME/src/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLowerCase.java

修改$HIVE_HOME/src/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java文件

import org.apache.hadoop.hive.ql.udf.UDFLowerCase;

registerUDF(“LowerCase”, UDFLowerCase.class,false);

(上面这个方法未测试成功)

为了避免每次都有add jar 可以设置hive的'辅助jar路径'
在hive-env.sh中 export HIVE_AUX_JARS_PATH=/home/ckl/workspace/mudf/mudf_fat.jar;

from：http://blog.chinaunix.net/uid-28194925-id-3453844.html