scribe 研究

来源：互联网发布：滚筒洗衣机知乎编辑：程序博客网时间：2024/05/16 14:39

scribe 研究- 一种分布式日志系统（facebook）

一张scribe结构图

先转一下，留作后用。

Installing Scribe For Log Collection

Scribe is a newly released log collection tool that dumps log files from various nodes in a cluster to Scribe servers, where the logs are stored for further use. Facebook describes their usage of Scribe by saying, “[Scribe] runs on thousands of machines and reliably delivers tens of billions of messages a day.” It turns out that Scribe is rather difficult to install, so the hope of this post is to help those of you attempting to install Scribe. The first step is to get dependencies installed.

Dependencies
Scribe has many dependencies that must be installed in order for Scribe to be built properly. They are listed here:

ruby (ruby and ruby-dev)
python (python and python-dev)
libevent (libevent and libevent-dev)
boost v1.36
Thrift
fb303 (included in Thrift in contrib/fb303)

The order in which these are installed is important. First, you must install libevent, then libevent-dev, then boost, then Thrift, and finally fb303. I installed libevent and libevent-dev from RPMs, whereas boost, Thrift, and fb303 were installed from source. I was unable to get Scribe, Thrift, and fb303 to locate the boost libraries and includes correctly with the default boost install directories, so I installed boost in /usr/local/boost, /usr/local/boost/bin, /usr/local/boost/lib, and /usr/local/boost/include. Run ‘./configure –help’ when configuring boost to see how to specify these options.

When configuring Thrift and fb303, you must specify your location of boost with the “–with-boost=/path/to/boost/root” option and also set your BOOST_ROOT environment variable.

Finally, you must make sure your LD_LIBRARY_PATH environment variable contains the lib folders that house the Thrift, fb303, boost, and libevent C++ libraries. LD_LIBRARY_PATH follows the same pattern as the PATH variable. That is, directories that contain libraries are separated by colons. If you forget to set your LD_LIBRARY_PATH variable, then you’ll get the following error when running scribed:

scribed: error while loading shared libraries: libboost_system-gcc41-mt-1_36.so.1.36.0: cannot open shared object file: No such file or directory

Install Scribe

Once you’ve successfully installed all dependencies, installing Scribe is easy. Scribe ships with a fairly comprehensive README file, but the instructions involving boost’s configuration are slightly incorrect. I needed to pass the “–with-boost=/usr/local/boost” options while configuring. However, the README file says to use “–with-boost /usr/local/boost”. Here is my full configure statement:

./configure –with-boost=/usr/local/boost –with-boost-system=boost_system-gcc41-mt-1_36 –with-boost-filesystem=boost_filesystem-gcc41-mt-1_36

We installed Scribe from trunk instead of releases/scribe-2.0.

Configure Scribe

Scribe ships with a few good examples in $SCRIBE_ROOT/examples; just take a look at the README in the examples directory, and you should be ready to rock. However, the README doesn’t document that the scribe_ctrl and scribe_cat programs are in the examples directory.

I hope this tutorial was helpful! Send us an email if you have any issues. We’ll make a follow-up post later talking about more in-depth Scribe configurations and benchmarking.

Tuesday, October 28th, 2008 at 9:13 pm by Alex Loddengaard, filed under data collection.

转自：

http://www.cloudera.com/blog/2008/10/28/installing-scribe-for-log-collection/

centos 5 安装

在开源Thrift，Hive和Cassandra之后，Facebook于2008年10月29日开源了其分布式日志收集框架Scribe，而且一上来就是stable的2.0版本。Scribe是一个用于分布式收集日志的Thrift服务，Facebook描述它”运行于数以千计的节点之上，每天可靠的发送数以百亿的消息“。

Scribe本身的代码量不大，但安装相当烦琐，本文的安装环境是CentOS 5，其他distribution如Ubuntu的安装类似。Scribe安装的依赖项包括：

* ruby (ruby and ruby-dev)
* python (python and python-dev)
* libevent (libevent and libevent-dev)
* boost 1.36
* thrift
* fb303

首先是安装ruby, python和libevent，这些比较容易，
yum install ruby ruby-devel
yum install python python-devel
yum install libevent libevent-devel

然后是boost，Scribe要求1.36版本，先去boost官方(http://www.boost.org/users/download/)上下一个安装包，如果没装g++，

yum install gcc-c++

假设boost安装的目录为/usr/local/boost

tar xzvf boost_1_36_0.tar.gz
cd boost_1_36_0
./configure --prefix=/usr/local/boost
make install

boost编译需要的时间比较长，安装完boost就可以装Thrift了，注意要先thrift依赖automake,libtool等

yum install automake byacc libtool flex bison
wget -O thrift.tgz "http://gitweb.thrift-rpc.org/?p=thrift.git;a=snapshot;h=HEAD;sf=tgz"
tar -xzf thrift.tgz
cd thrift
./configure --with-boost=/usr/local/boost
make
make install

然后安装fb303,facebook的支持库,源文件在thrift目录下的contrib/fb303
./configure --with-boost=/usr/local/boost
make
make install

终于可以装Scribe了
设置环境变量BOOST_ROOT为boost安装路径
export BOOST_ROOT=/usr/local/boost

设置环境变量LD_LIBRARY_PATH，指定包含Thrift, fb303, boost, libevent的路径
export LD_LIBRARY_PATH=/usr/local/boost/lib::/usr/lib:/usr/local/lib

安装
./configure --with-boost=/usr/local/boost
make
make install

安装后可以试一下scribe自带的例子，在目录examples下。

转自：

http://www.hadoopchina.com/index.php?option=com_content&task=view&id=67&Itemid=39