Spark-Lamda架构
来源：互联网 发布：戏剧打击乐器软件下载 编辑：程序博客网 时间：2024/06/07 22:21
日志分析
单机日志分析，适用于小数据量的。（最大10G），awk/grep/sort/join等都是日志分析的利器。
例子：
1、shell得到Nginx日志中访问量最高的前十个IP
<code class="hljs vhdl has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cat <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">access</span>.log<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.10</span> | awk '(a[$<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>]++) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">END</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span>(b <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> a) print b<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>a[b])' | sort -k2 -r | head -n <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
2、python 统计每个IP的地址点击数
<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> re <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> sys contents=sys.argv[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>] <span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">NginxIpHit</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(logfile_path)</span>:</span> ipadd = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">r'\.'</span>.join([<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">r'\d{1,3}'</span>]*<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>) re_ip = re.compile(ipadd) iphitlisting = {} <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> line <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> open(contents): match = re_ip.match(line) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> match: ip = match.group() iphitlisting[ip]=iphitlisting.get(ip,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>)+<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> iphitlisting NginxIpHit(contents)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul>**大规模的日志处理，日志分析指标：
PV、UV、PUPV、漏斗模型和准化率、留存率、用户属性
最终用UI展示各个指标的信息。**
架构
- 1、实时日志处理流线
数据采集：采用Flume NG进行数据采集
数据汇总和转发：用Flume 将数据转发和汇总到实时消息系统Kafka
数据处理：采用spark streming 进行实时的数据处理
结果显示：flask作为可视化工具进行结果显示
- 2、离线日志处理流线
数据采集：通过Flume将数据转存到HDFS
数据处理：使用spark sql进行数据的预处理
结果呈现：结果汇总到mysql上，最后使用flask进行结果的展现
Lamda架构：低响应延迟的组合数据传输环境。
查询过程：一次流处理、一次批处理。对应着实时和离线处理。
项目流程
安装flume
Flume进行日志采集，web端的日志一般Nginx、IIS、Tomcat等。Tomcat的日志在var/data/log
安装jdk
安装Flume
<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">wget http:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//mirrors.cnnic.cn/apache/flume/1.5.0/apache-flume-1.5.0-bin.tar.gz</span>tar –zxvf apache<span class="hljs-attribute" style="box-sizing: border-box;">-flume</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span><span class="hljs-attribute" style="box-sizing: border-box;">-bin</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tar<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>gzmv apache<span class="hljs-attribute" style="box-sizing: border-box;">-flume</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span> –bin apache<span class="hljs-attribute" style="box-sizing: border-box;">-flume</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span>ln <span class="hljs-attribute" style="box-sizing: border-box;">-s</span> apache<span class="hljs-attribute" style="box-sizing: border-box;">-flume</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span> fiume </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>
环境变量配置
<code class="hljs smalltalk has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">Vim</span> /etc/profile <span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">Export</span> <span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">JAVA_HOME</span>=/usr/local/jdk<span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">Export</span> <span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">CLASS_PATH</span> = .:<span class="hljs-char" style="box-sizing: border-box;">$ </span><span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">JAVA_HOME</span>/lib/dt.<span class="hljs-method" style="box-sizing: border-box;">jar:</span> <span class="hljs-char" style="box-sizing: border-box;">$ </span><span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">JAVA_HOME</span>/lib/tools.jar<span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">Export</span> <span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">PATH</span>=<span class="hljs-char" style="box-sizing: border-box;">$ </span><span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">PATH</span>:<span class="hljs-char" style="box-sizing: border-box;">$ </span><span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">JAVA_HOME</span>/bin<span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">Export</span> <span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlUME_HOME</span>=/usr/local/flume<span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">Export</span> <span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlUME_CONF_DIR</span>=<span class="hljs-char" style="box-sizing: border-box;">$ </span><span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlUME_HOME</span>/conf<span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">Export</span> <span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">PATH</span>=<span class="hljs-char" style="box-sizing: border-box;">$ </span><span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">PATH</span>:<span class="hljs-char" style="box-sizing: border-box;">$ </span><span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">FlUME_HOME</span> /bin<span class="hljs-class" style="box-sizing: border-box; color: rgb(102, 0, 102);">Souce</span> /etc/profile </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul>
创建agent配置文件将数据输出到hdfs上，修改flume.conf:
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sources</span> = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">r1</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sinks</span> = k1a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span> =c1描述和配置sources第一步：配置数据源a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sources</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.r</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.type</span> =execa1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sources</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.r</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span> =c1配置需要监控的日志输出目录a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sources</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.r</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.command</span>=tail –f /va/log/data第二步：配置数据输出a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.type</span> =hdfsa1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span> =c1a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.useLocalTimeStamp</span>=truea1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.path</span> =hdfs://<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">192.168</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.11</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.177</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9000</span>/flume/events/%<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">Y</span>/%m/%d/%H/%Ma1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.filePrefix</span> =cmcca1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.minBlockReplicas</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fileType</span> =DataStreama1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.writeFormat</span>=Texta1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rollInterval</span> =<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">60</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rollSize</span> =<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rollCount</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.idleTimeout</span> =<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>配置数据通道a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.type</span>=memorya1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.capacity</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.transactionCapacity</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span>第四步：将三者级联a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.souces</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.r</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span> =c1a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sinks</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channel</span> =c1</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li></ul>
启动Flume Agent
<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cd /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/flumenohup bin/flume<span class="hljs-attribute" style="box-sizing: border-box;">-ng</span> agent –n conf <span class="hljs-attribute" style="box-sizing: border-box;">-f</span> conf/flume<span class="hljs-attribute" style="box-sizing: border-box;">-conf</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>properties<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">&</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>
已经将flume整合到了hdfs中
- 整合Flume、kafka、hhdfs
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#hdfs输出端</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.type</span> =hdfsa1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span> =c1a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.useLocalTimeStamp</span>=truea1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.path</span> =hdfs://<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">192.168</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.11</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.174</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9000</span>/flume/events/%<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">Y</span>/%m/%d/%H/%Ma1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.filePrefix</span> =cmcc-%Ha1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.minBlockReplicas</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.fileType</span> =DataStreama1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rollInterval</span> =<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3600</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rollSize</span> =<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rollCount</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.hdfs</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.idleTimeout</span> =<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#kafka输出端 为了提高性能使用内存通道</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.type</span> =<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">com</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.cmcc</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.chiwei</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Kafka</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.CmccKafkaSink</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span> =c2a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.metadata</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.broker</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.List</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">192.168</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.11</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.174</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9002</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">;192.168.11.175:9092; 192.168.11.174:9092</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.partion</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.key</span> =<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.partioner</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>= <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">com</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.cmcc</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.chiwei</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Kafka</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Cmcc</span> Partiona1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.serializer</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.class</span>= kafka. Serializer<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.StringEncoder</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.request</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.acks</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.cmcc</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.encoding</span>=UTF-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.cmcc</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.topic</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.name</span>=cmcca1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.producer</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.type</span> =asynca1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sink</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.k</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.batchSize</span> =<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sources</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.r</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.selector</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.type</span>=replicatinga1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sources</span> = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">r1</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sinks</span> = k1 k2a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span> =c1 c2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#c1</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.type</span>=filea1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.checkpointDir</span>=/home/flume/flumeCheckpointa1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.dataDir</span>=/home/flume/flumeData, /home/flume/flumeDataExta1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.capacity</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2000000</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.transactionCapacity</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#c2</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.type</span>=memorya1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.capacity</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2000000</span>a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.channels</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.c</span>2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.transactionCapacity</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li></ul>
用Kafka将日志汇总
<code class="hljs axapta has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span> Tar –zxvf kafka_2<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.10</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.8</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span>.tgz<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span> 配置kafka和zookeeper文件配置zookeeper.propertiesdataDir=/tmp/zookeeper<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">client</span>.Port=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>maxClientCnxns = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>initLimit = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>syncLimit = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">##</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">server</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.43</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10.190</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.182</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.43</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2888</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3888</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">server</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.38</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10.190</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.182</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.38</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2888</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3888</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">server</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.33</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10.190</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.182</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.33</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2888</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3888</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul>
配置zookeeper myid
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">在每个服务器dataDir 创建 myid文件 写入本机id//server<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.43</span> myid 本机编号<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">43</span>echo “<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">43</span>” > /tmp/ zookeeper/myid配置kafka文件, config/server<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.properties</span>每个节点根据不同主机名配置broker<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.id</span> ：<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">43</span><span class="hljs-label" style="box-sizing: border-box;">host.name:</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10.190</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.172</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.43</span>zookeeper<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connect</span>=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10.190</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.172</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.43</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10.190</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.172</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.33</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span>，<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10.190</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.172</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.38</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2181</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul>
启动zookeeper
kafka通过zookeeper存储元数据，先启动它，提供kafka相应的连接地址
Kafka自带的zookeeper
在每个节点 bin/zookeeper-server-start.sh config/zookeeper. properties
启动Kafka
<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">Bin/Kafka<span class="hljs-attribute" style="box-sizing: border-box;">-server</span><span class="hljs-attribute" style="box-sizing: border-box;">-start</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>sh</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
创建和查看topic
Topic和flume中的要一致，spark streming 也用的这个
<code class="hljs brainfuck has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">Bin/</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">Kafka</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">topics</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">sh</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">create</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">zookeeper</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">10</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">190</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">172</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">43:2181</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">replication</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">factor</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">1</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">partions</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">1</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">topic</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">KafkaTopic</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
查看下：
<code class="hljs brainfuck has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">Bin/</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">Kafka</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">topics</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">sh</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">describe</span> <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span><span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">-</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">zookeeper</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">10</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">190</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">172</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">.</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">43:2181</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
整合kafka sparkstreming
<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">Buid<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>sbtSpark<span class="hljs-attribute" style="box-sizing: border-box;">-core</span>Spark<span class="hljs-attribute" style="box-sizing: border-box;">-streming</span>Spark<span class="hljs-attribute" style="box-sizing: border-box;">-streamng</span><span class="hljs-attribute" style="box-sizing: border-box;">-kafka</span>kafka</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>
- Spark streming 实时分析
数据收集和中转已经好了，kafka给sparkstreming - Spark sql 离线分析
- Flask可视化
代码
移步: github.com/jinhang
0 0
- Spark-Lamda架构
- Spark-再接着上次的Lamda架构
- Lamda
- Spark架构
- spark架构
- Spark 架构
- Spark 架构
- spark架构
- Spark架构
- Spark架构
- Spark架构
- Spark生态和Spark架构
- Spark生态和Spark架构
- “戏”说Spark---Spark架构
- spark+java1.8+lamda wordCount 实例,并且实现按单词出现的次数的倒序排序
- Spark架构图
- Spark架构及运行机制
- spark架构实战
- 学习积累——js
- 【bzoj 1030】 [JSOI2007]文本生成器（AC自动机+dp）
- Android 进程保活都在这里
- [LeetCode] Longest Substring Without Repeating Characters 解题报告
- Iterator浅析
- Spark-Lamda架构
- Rocchio文本分类
- mysql insert into select插入表中的数据与select的数据不一样
- LightOJ1013-Love Calculator-dp
- 题目报错总结
- FastDFS架构介绍
- Leetcode-415. Add Strings
- Java基础复习---数组
- zz
