scribe 研究- 一种分布式日志系统(facebook)
来源:互联网 发布:iphone 电话录音软件 编辑:程序博客网 时间:2024/05/22 20:36
scribe 研究- 一种分布式日志系统(facebook)
Installing Scribe For Log Collection
Scribe is a newly released log collection tool that dumps log files from various nodes in a cluster to Scribe servers, where the logs are stored for further use. Facebook describes their usage of Scribe by saying, “[Scribe] runs on thousands of machines and reliably delivers tens of billions of messages a day.” It turns out that Scribe is rather difficult to install, so the hope of this post is to help those of you attempting to install Scribe. The first step is to get dependencies installed.
Dependencies
Scribe has many dependencies that must be installed in order for Scribe to be built properly. They are listed here:
- ruby (ruby and ruby-dev)
- python (python and python-dev)
- libevent (libevent and libevent-dev)
- boost v1.36
- Thrift
- fb303 (included in Thrift in contrib/fb303)
The order in which these are installed is important. First, you must install libevent, then libevent-dev, then boost, then Thrift, and finally fb303. I installed libevent and libevent-dev from RPMs, whereas boost, Thrift, and fb303 were installed from source. I was unable to get Scribe, Thrift, and fb303 to locate the boost libraries and includes correctly with the default boost install directories, so I installed boost in /usr/local/boost, /usr/local/boost/bin, /usr/local/boost/lib, and /usr/local/boost/include. Run ‘./configure –help’ when configuring boost to see how to specify these options.
When configuring Thrift and fb303, you must specify your location of boost with the “–with-boost=/path/to/boost/root” option and also set your BOOST_ROOT environment variable.
Finally, you must make sure your LD_LIBRARY_PATH environment variable contains the lib folders that house the Thrift, fb303, boost, and libevent C++ libraries. LD_LIBRARY_PATH follows the same pattern as the PATH variable. That is, directories that contain libraries are separated by colons. If you forget to set your LD_LIBRARY_PATH variable, then you’ll get the following error when running scribed:
scribed: error while loading shared libraries: libboost_system-gcc41-mt-1_36.so.1.36.0: cannot open shared object file: No such file or directory
./configure –with-boost=/usr/local/boost –with-boost-system=boost_system-gcc41-mt-1_36 –with-boost-filesystem=boost_filesystem-gcc41-mt-1_36
We installed Scribe from trunk instead of releases/scribe-2.0.
I hope this tutorial was helpful! Send us an email if you have any issues. We’ll make a follow-up post later talking about more in-depth Scribe configurations and benchmarking.
- scribe 研究- 一种分布式日志系统(facebook)
- 分布式日志收集系统(一):Facebook Scribe
- 分布式日志收集系统:Facebook Scribe
- 分布式日志收集系统:Facebook Scribe
- 分布式日志收集系统: Facebook Scribe之日志收集方案
- 分布式日志收集系统: Facebook Scribe之配置文件
- facebook scribe日志搜集系统
- 分布式日志收集系统: Facebook Scribe之结构及源码分析
- Facebook Scribe日志聚合框架
- scribe日志收集系统
- scribe日志收集系统
- Scribe日志收集系统
- 开源日志系统比较:Facebook-scribe、Hadoop-chukwa、kafka、Cloudera-flume
- Facebook开源LogDevice:一种用于日志的分布式数据存储系统
- scribe日志系统安装笔记
- scribe日志系统安装笔记
- Scribe日志收集系统介绍
- 分享:Thrift+Scribe分布式日志系统的创建.ppt,欢迎交流
- JAVA的StringBuffer类
- SQL Server常见问题解决办法
- 状态压缩DP总结【POJ3254】【POJ1185】【POJ3311】【HDU3001】【POJ2288】【ZOJ4257】【POJ2411】【HDU3681】
- Effective C++
- 事务隔离等级划分
- scribe 研究- 一种分布式日志系统(facebook)
- 最小公约数-辗转相除法&stein算法
- 一位程序员的一个LBS应用的想法
- 自己动手开发编译器(一)编译器的模块化工程
- 程序员个修练之道(转)
- SQL1052N 数据库路径*不存在问题解决
- Unable to compile class for JSP
- 论文:Hadoop在Facebook的实时应用
- 错误602 未能在sysindexes中找到数据库ID8中对象ID1的索引ID1对应的行。请对sysindexes运行DBCC CHECKTABLE