程序博客网 > 飞鹰网络电视破解版

日志系统系列flume-ng+kafka+storm+HDFS搭建实验

来源：互联网 发布：飞鹰网络电视破解版 编辑：程序博客网 时间：2024/05/01 16:18

  最近公司要求搞个日志系统，以前接触不多，最近看了好多资料什么flume flume-ng kafka storm hdfs es 等等，感觉其实要做个小的直接kafka或者flume-ng就可以(现在已经代替了flume)，看了N多大神的文章比如这个《实时系统搭建》就很不错，但是对于我这种新手来说不一定合适，所以自己就准备边搞边记录了！

首先我的技术选型就是flume-ng kafka  storm hdfs 其中每个技术我都会详细写出步骤，省的到时候自己忘记了，原型就是根据Jonathan-Wei大神的文章来的，上边已经说了就不再重复！

 好了下边就来搞搞flume-ng，大家应该都知道与flume的不同之处吧！不知道的小伙伴也不要紧请看转载的文章flume的革命好了，下面我们就来看下flume的搭建

flume-ng的东西在整理下

===================================================================================================================================

hadoop hbase 单机版搭建

后面需要安装集群，整合到hadoop，所以注意选择与hadoop对应的版本：

Table 2.1. Hadoop version support matrix

 HBase-0.92.xHBase-0.94.xHBase-0.96Hadoop-0.20.205SXXHadoop-0.22.xSXXHadoop-1.0.xSSSHadoop-1.1.xNTSSHadoop-0.23.xXSNTHadoop-2.xXSS


S = supported and tested,支持X = not supported,不支持NT = not tested enough.可以运行但测试不充分

由于 HBase 依赖 Hadoop，它配套发布了一个Hadoop jar 文件在它的 lib 下。该套装jar仅用于独立模式。在分布式模式下，Hadoop版本必须和HBase下的版本一致。用你运行的分布式Hadoop版本jar文件替换HBase lib目录下的Hadoop jar文件，以避免版本不匹配问题。确认替换了集群中所有HBase下的jar文件。Hadoop版本不匹配问题有不同表现，但看起来都像挂掉了。


安装：

$ tar xfz hbase-0.90.4.tar.gz
$ cd hbase-0.90.4
我自己使用的是独立的zookeeper 后边连接的是hdfs存储，所以有如下配置 
hbase.rootdir 必须和 hadoop的core-sit.xml里面的地址一致

编辑配置文件hbase-site.xml

  1. vi hbase-site.xml
  2. <configuration> 
  3.   <property> 
  4.     <name>hbase.rootdir</name> 
  5.     <value>hdfs://server01:9000/hbase</value> 
  6.     <description>区域服务器使用存储HBase数据库数据的目录</description> 
  7.   </property> 
  8.   <property> 
  9.     <name>hbase.cluster.distributed</name> 
  10.     <value>true</value> 
  11.     <description>
  12.  指定HBase运行的模式：
  13.  false: 单机模式或者为分布式模式                   
  14.                  true: 全分布模式  
  15.     </description> 
  16.   </property> 
  17.   <property> 
  18.     <name>hbase.zookeeper.quorum</name> 
  19.     <value>server01,server02,server03</value> 
  20.     <description>ZooKeeper集群服务器的位置</description> 
  21.   </property> 
  22. </configuration> 


设置环境变量hbase-env.sh

  1. vi hbase-evn.sh  
  2. export HBASE_MANAGES_ZK=false 

使用独立的ZooKeeper时需要修改HBASE_MANAGES_ZK值为false，为不使用默认ZooKeeper实例。


我的环境变量

==============================================================================



# hadoop
HADOOP_HOME=/home/mq/hadoop/hadoop
# hbase
HBASE_HOME=/home/mq/hbase/hbase-0.98.7
# flume
export FLUME_HOME=/home/mq/flume/flume
# sbt
SBT=/home/mq/kafka/sbt
# kafka
KAFKA=/home/mq/kafka/kafka
JAVA_HOME=/usr/java/jdk1.7.0_60
CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/bin:$JAVA_HOME/lib/tools.jar:$CLASS_PATH:$HBASE_HOME/bin

=============================================================================================================================

单机启动抛出这样一个错误

[mq@localhost hbase-0.98.7]$ ./bin/start-hbase.sh 2014-10-23 10:46:27,658 FATAL [main] conf.Configuration: error parsing conf file: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。Exception in thread "main" java.lang.RuntimeException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243)        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1117)        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)        at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)        at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:594)        at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:69)        at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105)        at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:116)        at org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。        at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:687)        at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:557)        at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)        at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1426)        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2807)        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)        ... 8 morestarting master, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-master-localhost.localdomain.outlocalhost: +======================================================================+localhost: |      Error: JAVA_HOME is not set and Java could not be found         |localhost: +----------------------------------------------------------------------+localhost: | Please download the latest Sun JDK from the Sun Java web site        |localhost: |       > http://java.sun.com/javase/downloads/ <                      |localhost: |                                                                      |localhost: | HBase requires Java 1.6 or later.                                    |localhost: | NOTE: This script will find Sun Java whether you install using the   |localhost: |       binary or the RPM based installer.                             |localhost: +======================================================================+


我这很费解，看样子应该是环境变量问题修改修改conf文件下的hbase-env.sh文件中的java_home的环境变量

修改完以后错误变化

[mq@localhost hbase-0.98.7]$ ./bin/start-hbase.sh 2014-10-23 10:59:39,411 FATAL [main] conf.Configuration: error parsing conf file: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。Exception in thread "main" java.lang.RuntimeException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243)        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1117)        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)        at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)        at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:594)        at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:69)        at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105)        at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:116)        at org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。        at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:687)        at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:557)        at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)        at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1426)        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2807)        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:177)        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1156)        ... 8 morestarting master, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-master-localhost.localdomain.outlocalhost: starting regionserver, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-regionserver-localhost.localdomain.outlocalhost: Exception in thread "main" java.lang.RuntimeException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: 1 字节的 UTF-8 序列的字节 1 无效。localhost:      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1243)localhost:      at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1117)localhost:      at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1053)localhost:      at org.apache.hadoop.conf.Configuration.get(Configuration.java:397)localhost:      at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:594)localhost:      at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:69)localhost:      at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:105)localhost:      at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:116)localhost:      at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2484)
</pre><pre name="code" class="html" style="color: rgb(51, 51, 51); font-family: Arial; font-size: 14px; line-height: 26px;">这个是编码问题，代表你有的配置文件不是utf-8的，仔细检查下吧
还有一些问题比如报警告
<h3 class="t" style="margin: 0px 0px 1px; padding: 0px; list-style: none; font-weight: normal;font-size:14px; line-height: 1.54; color: rgb(51, 51, 51); font-family: arial;"><a target=_blank data-click="{'F':'378717EA','F1':'9D73F1E4','F2':'4CA6DE6B','F3':'54E5243F','T':'1414034754','y':'BF6CF7B5' }" href="http://www.baidu.com/link?url=o9-rxdk2Vum7yUDsx_F4fRvvG_VAwahcWTbiEX1l3f-SFwRNFGTSTCbwvFASirhz9bMWR1SQLuPu_IFXNM8hSa" target="_blank"><span style="color: rgb(204, 0, 0);">WARN</span> <span style="color: rgb(204, 0, 0);">util.NativeCodeLoader:</span> <span style="color: rgb(204, 0, 0);">Unable</span> <span style="color: rgb(204, 0, 0);">to</span> <span style="color: rgb(204, 0, 0);">load</span> <span style="color: rgb(204, 0, 0);">native-hadoop</span> ...</a></h3><span style="word-wrap: normal; word-break: normal; line-height: 27px; color: rgb(50, 62, 50); font-size: 13.5pt; font-family: 微软雅黑, sans-serif; background-color: rgb(145, 157, 163);">Apache</span><span style="word-wrap: normal; word-break: normal; line-height: 27px; color: rgb(50, 62, 50); font-size: 13.5pt; font-family: 微软雅黑, sans-serif; background-color: rgb(145, 157, 163);">提供的<span style="word-wrap: normal; word-break: normal;">hadoop</span>本地库是<span style="word-wrap: normal; word-break: normal;">32</span>位的，而在<span style="word-wrap: normal; word-break: normal;">64</span>位的服务器上就会有问题，因此需要自己编译<span style="word-wrap: normal; word-break: normal;">64</span>位的版本</span>
启动服务 bin/stop-hbase.sh 
<pre name="code" class="plain">starting master, logging to /home/mq/hbase/hbase-0.98.7/bin/../logs/hbase-mq-master-localhost.localdomain.outSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/home/mq/hbase/hbase-0.98.7/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/home/mq/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.


我去掉slf4j-log4j12-1.5.8.jar就解决了问题，没有报错。具体还有待查找
下面我们来验证下是否正确了
发现不对劲，居然没反应报错了，看了下日志
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster        at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)        at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:202)        at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:152)        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:179)        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2786)Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4        at org.apache.hadoop.ipc.Client.call(Client.java:1113)        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)        at com.sun.proxy.$Proxy8.getProtocolVersion(Unknown Source)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:606)        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)        at com.sun.proxy.$Proxy8.getProtocolVersion(Unknown Source)        at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)        at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)        at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:926)        at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:460)        at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.<init>(HMasterCommandLine.java:258)        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)        at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:137)        ... 7 more

无语的问题啊，继续查吧看来下，貌似hadoop和hbase版本不合的问题，突然发型hadoop2.5.1没有core，无语了啊
发现最后还是版本问题，我勒个去啊具体版本可以参考http://www.aboutyun.com/blog-61-62.html
换了个同样的版本的2就可以了，然后zookeeper又不能连接2222端口，本来服务器已经有了个zookeeper想用同一个或者再来一个
编辑hbase-site.xml 加入地址（都在hosts配置好了）和端口2222
<property>
   <name>hbase.zookeeper.quorum</name>
   <value>hd204,hd205,hd206</value>
 </property>
在编辑regionservers加入映射
hd204
hd205
hd206
OK 启动hbase
http://192.168.1.116:60010/master-status# 
查看hbase状态，简单的配置就完成了
后期会把hadoop+flume-ng的东西补上留作参考，骑士hbase还是不全，有时间在整理了



                                             
        0        0           
	
				
		









 飞鹰网络电视破解版
    		飞鹰网络电视破解版



原创粉丝点击



		

热门IT博客


美工的基础知识美工的基础知识




唐诗逸和张傲月 知乎唐诗逸和张傲月 知乎




林少华 知乎




恶意域名检测系统




剑三苏曼莎捏脸数据




四季彩软件下载




python命令行参数传递




淘宝改差评技巧




mac找不到steam文件夹




上海翌创网络科技股份




在线数据库设计




ug8.0编程视频




域名多少钱一年




盗墓笔记坐骑进阶数据




小黑屋写作软件免费版




野蛮人升级数据




数据库代理




校园网络规划设计




网络监控系统工程合同




asp.net cms开源系统



		

热门问题
    老师的惩罚
    人脸识别
    我在镇武司摸鱼那些年
    重生之率土为王
    我在大康的咸鱼生活
    盘龙之生命进化
    天生仙种
    凡人之先天五行
    春回大明朝
    姑娘不必设防，我是瞎子
    项链多少钱
    绿松石项链
    情侣项链专卖店
    项链盒
    紫水晶项链
    月光石吊坠
    青金石项链价格
    大日如来吊坠
    项链女
    吊坠盒
    月光石项链
    托帕石项链价格
    项链断了意味着什么
    貔貅挂件正确戴法
    18krgp项链多少钱
    紫水晶吊坠
    男士吊坠项链
    项链吊坠品牌
    白金项链多少钱
    翡翠貔貅吊坠
    玛瑙项链价格
    黑曜石观音吊坠的寓意
    纯银项链吊坠
    黄金项链吊坠
    水晶项链价格
    男生带的项链
    钻石吊坠定制
    虚空藏菩萨玉吊坠
    男项链
    925银项链多少钱
    玫瑰金吊坠
    水晶吊坠项链
    铂金钻石项链多少钱
    十字架吊坠项链
    黄金项链配什么吊坠好看
    铂金项链配什么吊坠好
    男士玉吊坠
    首饰项链
    翡翠观音吊坠
    欧美项链
    黄水晶貔貅吊坠


	










程序博客网，程序员的互联网技术博客家园。csdn论坛精品 msdn技术资料都在这里