2013-03-18 0%-%33-0%-33%
来源:互联网 发布:linux内核完全剖析pdf 编辑:程序博客网 时间:2024/06/05 03:59
1 试验以tab键分割是否可以,
改了create table语句,以delimited field by ‘/t'
这样分割出来的日期是带[],所以重写了datetimeconvert的udf
2 某产品查询任务,reduce执行到33%就又变成0%,再变成33%,直到报错:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"2013-03-12 02:00:00","_col1":"\"GET /status HTTP/1.1\""},"value":{"_col0":2},"alias":0}
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:268)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:463)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"2013-03-12 02:00:00","_col1":"\"GET /status HTTP/1.1\""},"value":{"_col0":2},"alias":0}
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
... 3 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public org.apache.hadoop.io.Text com.gongsi.hive.udf.URLConvert.evaluate(java.lang.String) on object com.gongsi.hive.udf.URLConvert@44b09697 of class com.gongsi.hive.udf.URLConvert with arguments {"-":java.lang.String} of size 1
at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:848)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:181)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(ExprNodeGenericFuncEvaluator.java:163)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:76)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:959)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:798)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:724)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
... 3 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:824)
... 13 more
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1931)
at com.gongsi.hive.udf.URLConvert.evaluate(URLConvert.java:9)
... 18 more
发现是之前的转换会把url是"-"直接换成-,而现在则直接是"-",而urlconvert函数中并没有对带引号的"-"的处理。
改了create table语句,以delimited field by ‘/t'
这样分割出来的日期是带[],所以重写了datetimeconvert的udf
2 某产品查询任务,reduce执行到33%就又变成0%,再变成33%,直到报错:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"2013-03-12 02:00:00","_col1":"\"GET /status HTTP/1.1\""},"value":{"_col0":2},"alias":0}
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:268)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:463)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"2013-03-12 02:00:00","_col1":"\"GET /status HTTP/1.1\""},"value":{"_col0":2},"alias":0}
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
... 3 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public org.apache.hadoop.io.Text com.gongsi.hive.udf.URLConvert.evaluate(java.lang.String) on object com.gongsi.hive.udf.URLConvert@44b09697 of class com.gongsi.hive.udf.URLConvert with arguments {"-":java.lang.String} of size 1
at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:848)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:181)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(ExprNodeGenericFuncEvaluator.java:163)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:76)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:959)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:798)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:724)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
... 3 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:824)
... 13 more
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1931)
at com.gongsi.hive.udf.URLConvert.evaluate(URLConvert.java:9)
... 18 more
发现是之前的转换会把url是"-"直接换成-,而现在则直接是"-",而urlconvert函数中并没有对带引号的"-"的处理。
- 2013-03-18 0%-%33-0%-33%
- 2011-10-11 0:33:27
- CHAPTER 03:EX 33
- 苏嵌33 17.01.18
- 33
- 33
- 33
- 33
- 33
- 33
- 33
- 33
- 33
- 33
- 33
- 33
- 33
- 33
- 编程从业五年的十四条经验 句句朴实
- 处理Oracle中杀不掉的锁
- 一手棋与一口气
- sharepoint 命令行安装、部署、删除wsp包
- 将分开的两个List合并成一个并保持其原来的顺序的两种方法
- 2013-03-18 0%-%33-0%-33%
- Unity3D 2D Toolkit (之一)
- file上传文件的获取方式.txt
- getimagesize
- MFC消息响应机制分析
- How to batch executing SQL statements on db2
- 32个SEO方法提高网站的流量:
- NET鼠标右键菜单JS[原]
- IOS 网络检测