Hive 学习笔记（三）

自定义函数

当写hive UDF时，有两个选择：一是继承 UDF类，二是继承抽象类GenericUDF。这两种实现不同之处是：GenericUDF 可以处理复杂类型参数，并且继承GenericUDF更加有效率，因为UDF class 需要HIve使用反射的方式去实现。

UDF

一个UDF 必须满足两个条件：
1. 必须继承 org.apache.hadoop.hive.ql.exec.UDF类
2. 至少实现了一个 evaluate() 方法

/** * A User-defined function (UDF) for the use with Hive. * * New UDF classes need to inherit from this UDF class. * * Required for all UDF classes: 1. Implement one or more methods named * "evaluate" which will be called by Hive. The following are some examples: * public int evaluate(); public int evaluate(int a); public double evaluate(int * a, double b); public String evaluate(String a, int b, String c); * * "evaluate" should never be a void method. However it can return "null" if * needed. */@UDFType(deterministic = true)public class UDF {  /**   * The resolver to use for method resolution.   */  private UDFMethodResolver rslv;  /**   * The constructor.   */  public UDF() {    rslv = new DefaultUDFMethodResolver(this.getClass());  }  /**   * The constructor with user-provided UDFMethodResolver.   */  protected UDF(UDFMethodResolver rslv) {    this.rslv = rslv;  }  /**   * Sets the resolver.   *   * @param rslv   *          The method resolver to use for method resolution.   */  public void setResolver(UDFMethodResolver rslv) {    this.rslv = rslv;  }  /**   * Get the method resolver.   */  public UDFMethodResolver getResolver() {    return rslv;  }  /**   * These can be overriden to provide the same functionality as the   * correspondingly named methods in GenericUDF.   */  public String[] getRequiredJars() {    return null;  }  public String[] getRequiredFiles() {    return null;  }}

UDF 函数只能处理简单的java标量类型，如int，double，String等类型

GenericUDF

使用Geoip 根据IP定位国家,实现GenericUDF。

public class GeoLocUDF extends GenericUDF {    private LookupService service;    //用于将输入类型转换为目标类型    private ObjectInspectorConverters.Converter[] converters;    /**     * 初始化     * @param arguments     * @return     * @throws UDFArgumentException     */    @Override    public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException {        if (arguments.length !=2){            throw new UDFArgumentLengthException(                    "The function COUNTRY(ip, geolocfile) takes exactly 2 arguments.");        }        converters = new ObjectInspectorConverters.Converter[arguments.length];        for (int i = 0; i < arguments.length; i++) {            //创建一个Converter，可以在evaluate方法中使用它 将参数的原生数据类型转存未Java Strings。            converters[i] = ObjectInspectorConverters.getConverter(arguments[i],                    PrimitiveObjectInspectorFactory.javaStringObjectInspector);        }        // 指定UDF （evaluate函数）返回类型为String        return PrimitiveObjectInspectorFactory                .getPrimitiveJavaObjectInspector(PrimitiveObjectInspector.PrimitiveCategory.STRING);    }    @Override    public Object evaluate(DeferredObject[] arguments) throws HiveException {        if (arguments[0].get() == null || arguments[1].get() == null) {            return null;        }        String ip = (String) converters[0].convert(arguments[0].get());        String filename = (String) converters[1].convert(arguments[1].get());        return lookup(ip, filename);    }    //在出现异常时，会调用这个方法，输出调用信息    @Override    public String getDisplayString(String[] children) {        return "country(" + children[0] + ", " + children[1] + ")";    }    protected String lookup(String ip, String filename) throws HiveException {        try {            if (service == null) {                URL u = getClass().getClassLoader().getResource(filename);                if (u == null) {                    throw new HiveException("Couldn't find geolocation file '" + filename + "'");                }                service =                        new LookupService(u.getFile(), LookupService.GEOIP_MEMORY_CACHE);            }            String countryCode = service.getCountry(ip).getCode();            if ("--".equals(countryCode)) {                return null;            }            return countryCode;        } catch (IOException e) {            throw new HiveException("Caught IO exception", e);        }    }}

在Hive中调用UDF

ADD JAR /path/hive-test.jar;CREATE temporary FUNCTION geoIp AS 'com.hadoop2.hive.GeoLocUDF';

temporary 关键字表示UDF只是为这个Hive会话过程定义的。

深入学习：https://cwiki.apache.org/confluence/display/Hive/GenericUDAFCaseStudy