[转]KFS官方部署手册
来源:互联网 发布:国泰安并购数据库 编辑:程序博客网 时间:2024/05/22 17:13
野史终归不如官方版本的,转自http://sourceforge.net/apps/trac/kosmosfs/wiki/HowToCompile
Required Packages¶
To compile and run KFS, you need to have the following software packages installed on your machine:
- Boost (preferably, version 1.34 or higher)
- cmake (preferably, version 2.4.6 or higher)
- log4cpp (preferably, version 1.0)
- gcc version 4.1 (or higher)
- xfs devel RPMs on Linux
This document assumes that you have downloaded the source to directory: ~/code/kfs. We assume that you want to build the source in ~/code/kfs/build.
There are few parts to compiling the source:
- Build the C++ side to get the metaserver/chunkserver binaries, tools, and the C++ client library.
- Build the Java side to get a kfs.jar file which contains the wrapper calls to native C++ via JNI; this allows Java apps to access files stored in KFS.
- Building Python extension module for Python support (optional)
Building C++ Components¶
It is preferable to build KFS binaries with debugging symbols. The sequence of steps are:
cd ~/code/kfsmkdir buildcd buildcmake -D CMAKE_BUILD_TYPE=RelWithDebInfo ~/code/kfs/gmakegmake install
The binaries will be installed in:
- Executables will be in: ~/code/kfs/build/bin
- Libraries will be in: ~/code/kfs/build/lib
Building Java Components¶
To build Java support setup:
cd ~/code/kfsant jar
This will produce the following set of files:
- ~/code/kfs/build/classes --- This will contain the Java class files
- ~/code/kfs/build/kfs-{version}.jar --- The jar file containing the Java classes
Add .jar file to your CLASSPATH environment variable
export CLASSPATH=${CLASSPATH}:~/code/kfs/build/kfs-[version].jar
Building Python Support¶
Build the KFS client library (see above). Let the path to the shared libraries is ~/code/kfs/build/lib. Now, to build the Python extension module:
cd to ~/code/kfs/src/cc/accessEdit kfs_setup.py and setup the include path. Specifically, kfsext = Extension('kfs', include_dirs ['kfs/src/cc/', '<path to boost>'])python kfs_setup.py ~/code/kfs/build/lib/ build
This will build a shared object library _kfs.so_. The kfs.so library needs to be installed either in the site-packages for python or in an alternate location. To install in site-packages for python:
python kfs_setup.py ~/code/kfs/build/lib/ install
To install in alternate location such as, ~/code/kfs/build/lib
python kfs_setup.py ~/code/kfs/build/lib install --home=~/code/kfs/build/lib
If you installed in alternate location, update PYTHONPATH environment variables:
export PYTHONPATH=${PYTHONPATH}:~/code/kfs/build/lib/lib64/python
Also, update the LD_LIBRARY_PATH environment variable:
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:~/code/kfs/build/lib
Using Python Client on the Mac (Leopard)¶
For the Mac, update the DYLD_LIBRARY_PATH environment variable (so that libkfsClient.dylib will be found):
export DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:~/code/kfs/build/lib
Requirements¶
The KFS package comes with scripts for installing/starting/stopping KFS servers on a set of machines. The scripts use ssh to login and execute commands (either on a single node or in a cluster). The scripts require that the user have "no password" ssh access to every affected server. For every host affected, be sure you can "ssh host true" without being prompted for a password.
Defining Machine Configuration for KFS-0.2.3 (or higher)¶
With KFS-0.2.3. (or higher), the system configuration can be defined using two files. One that defines the enviroment (such as, paths, port numbers to etc) for all the chunkservers and another file that lists the nodes on which the chunkservers should be started. The set of machines on which KFS servers have to be deployed is defined in a configuration file that follows the Python config file format. For example, the config file machines.cfg:
[metaserver]node: machine1clusterkey: kfs-test-clusterrundir: /mnt/kfs/metabaseport: 20000loglevel: INFOnumservers: 2[chunkserver_defaults]rundir: /mnt/kfs/chunkchunkDir: /mnt/kfs/chunk/bin/kfschunkbaseport: 30000space: 3400 Gloglevel: INFONext, assuming that there are 3 nodes in the cluster, their names should be listed in a separate file, machines.txt. The format of this file is one entry per line. For example,
10.2.3.110.2.3.210.2.3.3Defining Machine Configuration (Old style format)¶
The set of machines on which KFS servers have to be deployed is defined in a configuration file that follows the Python config file format. For example,
[metaserver]node: machine1clusterkey: kfs-test-clusterrundir: /mnt/kfs/metabaseport: 20000loglevel: INFOnumservers: 2[chunkserver1]node: machine2rundir: /mnt/kfs/chunkchunkDir: /mnt/kfs/chunk/bin/kfschunkbaseport: 30000space: 3400 Gloglevel: INFO[chunkserver2]node: machine2rundir: /mnt/kfs/chunkchunkDir: /mnt/kfs/chunk/bin/kfschunkbaseport: 30000space: 3400 Gloglevel: INFOThis file defines a configuration for a deployment comprising of a metaserver and two chunkservers. The various fields:
- node: This defines the machine name where the binary should run
- rundir: This defines the directory on the machine where KFS binaries will be installed.
- baseport: This port at which the metaserver/chunkserver process will listen for connection from clients
- loglevel: The level for outputting messages. Since KFS uses log4cpp, the values are INFO/DEBUG
- space: The storage space exported by a chunkserver for storing chunks (units are 'G' for GigaBytes and 'M' for MegaBytes)
- chunkDir : The list of directories used to store chunk files on the chunkserver nodes. For a JBOD configuration, this would be a space separated list of directory names.
- clusterkey : A key that is shared between metaserver/chunkserver. On startup, each chunkserver announces itself to the metaserver and provides the cluster key. This key should match the value that the metaserver; only then is the chunkserver part of the KFS deployment.
Script for install/launching processes¶
Two sets of scripts are checked into the repository:
- Linux: Use ~/code/kfs/scripts
- Solaris, Mac OSX: Use ~/code/kfs/scripts.solaris
The instructions in this section are for Linux platform. Use the corresponding script in scripts.solaris for Solaris/Mac platforms. When you use any script which will transfer files (particularly kfssetup) on the Macintosh, you will need the additional option --tar=tar; the scripts assume that GNU tar is named "gtar" otherwise.
Installing KFS Binaries¶
Define the configuration in machines.cfg. Then, run the install script. The install script runs a set of processes concurrently. This works well when the servers need to be configured for a distributed setting. For a single node setup, the processes need to be launched sequentially:
- When all the servers are on a single host
cd ~/code/kfs/scriptspython kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webui -s- MacOSX:
cd ~/code/kfs/scripts.solarispython kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webui --tar=tar -s- When the servers are on multiple hosts
cd ~/code/kfs/scriptspython kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webuiNOTE: The "-m" option is supported only with KFS-0.2.3 or higher.
Upgrading from prior KFS 0.2.x releases¶
Stop the servers. Upgrade the binaries and relaunch:
- Stopping the servers:
cd ~/code/kfs/scriptspython kfslaunch.py -f machines.cfg -S
- When all the servers are on a single host
cd ~/code/kfs/scriptspython kfssetup.py -f machines.cfg -b ../build -w ../webui -u -s
- When the servers are on multiple hosts
cd ~/code/kfs/scriptspython kfssetup.py -f machines.cfg -b ../build -w ../webui -uStarting KFS servers¶
To launch the KFS servers, run the launch script:
cd ~/code/kfs/scriptspython kfslaunch.py -f machines.cfg -m machines.txt -sChecking System Status¶
To verify that the servers started up and are connected to the metaserver, use the kfsping tool:
cd ~/code/kfs/build/bin/toolskfsping -m -s <metaserver host> -p <metaserver port>If you using KFS-0.2.2 or higher, you can use a Web browser to monitor the servers. The KFS package now includes a simple python-based web server that shows the set of servers that are currently connected to the metaserver. The KFS web server runs on the same machine as the KFS metaserver. The web server's port is metaserver's baseport + 50. For example,
metaserver runs on machine node1 and uses port 20000the web server runs on machine node1 at port 20050. Point the browser at http://node1:20050The KFS web server uses a file "all-machines.txt" to track where the chunkservers should be running. This file should be manually created and placed in ~/code/kfs/webui before deploying KFS binaries. The format of this file is the name of the chunkserver machines, one entry per line. For example.
10.2.3.110.2.3.210.2.3.3When you open the page, http://node1:20050/cluster-view, the web server will return a web page that lists:
- where the chunkservers are currently running
- where the chunkservers could not be started (due to ssh failures, etc.)
- where the chunkservers have failed
Stopping KFS servers¶
To stop the KFS servers, run the launch script:
cd ~/code/kfs/scriptspython kfslaunch.py -f machines.cfg -m machines.txt -SAdding new chunkservers¶
It is not necessary to stop the KFS servers to add new chunkservers. Simply update the machines.cfg file and start the servers (see Starting KFS servers).
Un-installing KFS Binaries¶
To format the filesystem/uninstall the binaries, run the install script:
cd ~/code/kfs/scriptspython kfssetup.py -f machines.cfg -m machines.txt -b ../build/bin -U
- [转]KFS官方部署手册
- [转]KFS的部署与简单使用
- KFS
- [转]HDFS和KFS 比较
- gitlab安装手册官方修正版(转)
- KFS(kosmosfs)的编译和安装部署 (step by step)
- [转] KFS,一个克隆GFS的文件系统
- mod_python官方手册翻译
- XPath官方手册中文版
- PhysX官方手册翻译
- vim官方中文版手册
- emacs官方中文手册
- Twig官方手册
- velocity 官方手册
- PhysX官方手册翻译
- boost官方手册
- 官方手册集锦
- docker官方手册
- 小游戏,拾火柴
- VC 打开外部文件
- 和按钮一样的层的制作
- 使用servlet下载
- Flex 转向URL
- [转]KFS官方部署手册
- 页面崩溃了
- WEP、WPA无线加密对比
- 常见问题及解决办法 整理之9(动态语句语法)
- “灵捷网络智能办公OA系统”打包安装详解三
- 程序中变量的缩写
- 第四节 UNIX标准文件编程库
- 哲哲很淘气
- HDU 1856(并查集)