flume保存文件到hdfs的时候报错
来源:互联网 发布:软件测试bug管理工具 编辑:程序博客网 时间:2024/06/04 22:08
flume配置sink向hdfs中写入文件,在启动的时候遇到的报错问题
1. Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
[ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)] Failed to start agent because dependencies were not found in classpath. Error follows.java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:239) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:413) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:98) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 12 more
这是因为没有相应的的jar包导致的, 进入到/hadoop/share/hadoop/common/*.jar 和/hadoop/share/hadoop/common/lib/*.jar 拷贝到flume安装目录的lib下就可以。
2. HDFS IO error
java.io.IOException: No FileSystem for scheme: hdfs
[WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)] HDFS IO errorjava.io.IOException: No FileSystem for scheme: hdfs at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2385) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
运行的时候还是会报找不到hdfs文件的错误,进入到hadoop的安装目录找到/share/hadoop/hdfs/hadoop-hdfs-2.4.1.jar 同样拷贝到flume的安装目录的lib下。再启动就可以了。
1 0
- flume保存文件到hdfs的时候报错
- flume上传文件到hdfs上
- flume监控目录文件到hdfs
- hadoop 上传文件到HDFS报错
- 通过Flume拉取Kafka数据保存到HDFS
- eclipse myeclipse 保存的时候报错
- flume的导日志数据到hdfs
- Flume:本地文件到HDFS
- MyEclipse 9 在部署和保存文件的时候报错 DeploymentBuilder
- Flume采集数据到HDFS时,生成的文件中,开头信息有乱码
- 用flume提交文件到hdfs系统,并保持原来的文件名信息
- 模拟使用Flume监听日志变化,并且把增量的日志文件写入到hdfs中
- 多个Flume合并一个channel上传文件到Hdfs
- Flume采集数据到HDFS时,文件中有乱码
- 使用flume将avro文件上传到hdfs上
- flume 抽取图片文件数据写入到HDFS
- 【Flume】【源码分析】flume中sink到hdfs,文件系统频繁产生文件,文件滚动配置不起作用?
- 【Flume】【源码分析】flume中sink到hdfs,文件系统频繁产生文件,文件滚动配置不起作用?
- 【OGG】SqlServer同步源端增加trandata报错解决
- 小括号与花括号小括号与花括号
- java线程同步
- 使用For XML PATH 会影响Cross Apply 返回
- 从0开始学复制~我的复制学习历程 (一) 简单的快照复制处理
- flume保存文件到hdfs的时候报错
- 作死的玩一下计算列
- mina推送场景小hello world demo
- SqlServer链接MySql操作步骤
- Rollback 语句 在08R2版本
- 浅谈游标选项 Static|Keyset|DYNAMIC|FAST_FORWARD
- 几何空间类型初探
- Merge 的小技巧
- 空间索引应用