导出Oracle文件到hive中遇到的相关问题解决方法
来源:互联网 发布:网络词 背书是什么意思 编辑:程序博客网 时间:2024/06/05 19:46
最近在学习hadoop知识,昨天在做数据抽取的时候遇到很多问题,做了记录,整理在博客中,以供日后查询方便,同时希望可以帮助到有需要的朋友。
导入文件相关命令:
导出oracle文件到hive:
sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --split-by yongdianlb --hive-import
遇到的问题:
1、bleException: IO Error: The Network Adapter could not establish the connection
java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
问题描述:网络适配监听错误
原因源码:sqoop import --connect jdbc:oracle:thin://192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --hive-import
解决办法:查看对应的网络连接是否正确,比如我这里就是将@符号换成了 //,导致不能找到监听。只需要将//改为@ 就好。
解决源码:sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --hive-import
2、 ERROR tool.ImportTool: Imported Failed: Attempted to generate class with no columns!
问题描述:没有对象(自己理解,英语不好,见谅Σ( ° △ °|||)︴)
原因源码:sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username uap --password uap --table YKBZ_DETAIL --hive-import
解决办法:将用户名改为大写。
解决源码:sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --hive-import3、14/12/23 04:54:32 ERROR tool.ImportTool: Error during import: No primary key could be found for table YKBZ_DETAIL. Please specify one with --split-by or perform a sequential import with '-m 1'.
问题描述:略
原因代码:略
解决办法:个人建议参考这个博主的内容,不管是问题描述还是解决办法都非常详细http://blog.sina.com.cn/s/blog_6a67b5c501010gd9.html
sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --split-by yongdianlb --hive-import
sqoop import --connect jdbc:oracle:thin:@192.168.1.136:1521/jist --username UAP --password uap --table YKBZ_DETAIL --split-by yongdianlb --hive-import
4、PriviledgedActionException as:root (auth:SIMPLE)
问题描述:略
原因代码:略
解决办法:在 mappred-site.xml 添加属性
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
5、ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
问题描述:
4/12/23 05:49:58 INFO mapreduce.Job: Job job_1419337232716_0001 completed successfully
14/12/23 05:50:06 INFO mapreduce.Job: Counters: 27
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=367120
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=503
HDFS: Number of bytes written=112626288
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=1558361
Total time spent by all reduces in occupied slots (ms)=0
Map-Reduce Framework
Map input records=158424
Map output records=158424
Input split bytes=503
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=83987
CPU time spent (ms)=56180
Physical memory (bytes) snapshot=315813888
Virtual memory (bytes) snapshot=3356368896
Total committed heap usage (bytes)=179679232
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=112626288
14/12/23 05:50:07 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
14/12/23 05:50:08 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/12/23 05:50:38 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/12/23 05:50:38 ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:http://wiki.apache.org/hadoop/ConnectionRefused
14/12/23 05:50:38 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:416)
14/12/23 05:50:38 ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:http://wiki.apache.org/hadoop/ConnectionRefused
14/12/23 05:50:38 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:416)
原因代码:
解决办法:
./mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR
再次执行,成功。
另外附上网上其它的解决方案:(本人没测试)
hadoop dfsadmin -report 先查看datanode状态和主机名
http://blog.csdn.net/shirdrn/article/details/6562292在datanode修改hostname
/etc/sysconfig/network
然后重启
/etc/rc.d/init.d/network restart
再次执行 OK
0 0
- 导出Oracle文件到hive中遇到的相关问题解决方法
- oracle导出数据到文件中的方法
- Hive中将查询结果导出到指定分隔符的文件中
- Mysql5.6版本导出sql文件数据导入到5.7版本出现的问题解决方法
- oracle导出文件到excel中
- PHP中遇到的时区问题解决方法
- 从Excel导入数据到vi文件中遇到问题解决
- MongoDB安装到windows服务中方法及遇到问题解决
- 安装oracle遇到问题解决方法
- hive导出查询文件到本地文件的2种办法
- hive导出查询文件到本地文件的2种办法
- hive导出查询文件到本地文件的2种办法
- hive导出查询文件到本地文件的2种办法
- Oracle数据导出到文本文件的方法
- 利用sqoop导出hive数据到 oracle
- Hive导出到Mysql中中文乱码的问题
- 关于在eclipse中部署项目到tomcat,相关包无法加载的问题解决方法
- 从数据库导出数据到EXCEL换行的问题解决方法
- leetcode:Excel Sheet Column Title
- Ubuntu中网络图标不见了
- mvn
- 用异常拒绝不合格的传入参数和其他
- ZOJ 3209 Treasure Map(DLX精确覆盖)
- 导出Oracle文件到hive中遇到的相关问题解决方法
- spring no session or session was closed,org.hibernate.LazyInitializationException: failed to lazily
- 【深入浅出IOS开发】使用路径进行绘制图形
- 利用位运算实现加法运算
- db2创建用户及授权的研究
- 找出100~200之间的素数
- [RPC Fault faultString="Cannot invoke method 'saveOrUpdate'." faultCode="Server.ResourceUnavailable"
- Mac下配置Eclipse + Android SDK
- 指针