分布式架构实战--FastDFS分布式文件系统集群安装

跟踪服务器：


Tracker Server1 IP:  10.10.16.201 FASTDFS-TRACKER-01


Tracker Server2 IP:  10.10.16.202 FASTDFS-TRACKER-02


 


存储服务器：


Storage Server1 IP:  10.10.16.203 FASTDFS-STORAGE-01


Storage Server2 IP:  10.10.16.204 FASTDFS-STORAGE-02


 


 


集群中各操作系统环境设置：


(1) fastdfs安装目录：/usr/local/fastdfs


[root@FASTDFS-TRACKER-01 local]# mkdir /usr/local/fastdfs


[root@FASTDFS-TRACKER-02 local]# mkdir /usr/local/fastdfs


[root@FASTDFS-STORAGE-01 local]# mkdir /usr/local/fastdfs


[root@FASTDFS-STORAGE-02 local]# mkdir /usr/local/fastdfs


 


(2)创建fastdfs用户组及fastdfs用户（四台机同时配置）:


[root@xxx]# groupadd fastdfs


[root@xxx]# useradd -g fastdfs fastdfs


[root@xxx]# passwd fastdfs


## 设置fastdfs用户密码为gzzyzz.com （上生产前要改）


 


(3) 创建数据存储目录:


#创建tracker目录保存运行日志(Tracker服务器)


[fastdfs@FASTDFS-TRACKER-01 ~]$ mkdir -p /home/fastdfs/tracker


[fastdfs@FASTDFS-TRACKER-02 ~]$ mkdir -p /home/fastdfs/tracker


#创建Storage目录保存运行日志及其data数据（Storage服务器）


[fastdfs@FASTDFS-STORAGE-01 ~] $ mkdir -p /home/fastdfs/storage


[fastdfs@FASTDFS-STORAGE-02 ~] $ mkdir -p /home/fastdfs/storage


 


 


注：因为要源码编译安装软件，因而各系统要事先安装好:make cmake gcc gcc-c++


[root@FASTDFS-TRACKER-01 yum.repos.d]# yum install make cmake gcc gcc-c++


 


1、 首先安装libevent（集群中所有服务器都执相同的安装）:


fastdfs在编译源程序时fastdfs内部调用libevent的处理机制,需要用到libevent一些依赖文件,否则编译fastdfs会出错。


# 卸载系统自带libevent,自带版本过低,要不然安装fastdfs会出错


[root@FASTDFS-TRACKER-01 fastdfs]# rpm -qa|grep libevent


libevent-1.4.13-4.el6.x86_64


（由上可以系统自带了libevent-1.4.13-4.el6.x86_64）


卸载：


[root@FASTDFS-TRACKER-01 fastdfs]# rpm -e --nodeps libevent-1.4.13-4.el6.x86_64


 


 


#下载安装libevent


进入/usr/local/src目录


# cd /usr/local/src/


[root@FASTDFS-TRACKER-01 src]#


wget http://cloud.github.com/downloads/libevent/libevent/libevent-2.0.19-stable.tar.gz


 


[root@FASTDFS-TRACKER-01 src]# tar -zxvf libevent-2.0.19-stable.tar.gz


[root@FASTDFS-TRACKER-01 src]# cd libevent-2.0.19-stable


#make clean;


[root@FASTDFS-TRACKER-01 libevent-2.0.19-stable]$ ./configure --prefix=/usr/local/libevent


[root@FASTDFS-TRACKER-01 libevent-2.0.19-stable]# make


[root@FASTDFS-TRACKER-01 libevent-2.0.19-stable]# make && make install


#为libevent创建软链接到/lib库下，64位系统对应/lib64


[root@FASTDFS-TRACKER-01 libevent-2.0.19-stable]#


ln -s /usr/local/libevent/lib/libevent-2.0.so.5 /usr/lib/libevent-2.0.so.5


[root@FASTDFS-TRACKER-01 libevent-2.0.19-stable]#


ln -s /usr/local/libevent/lib/libevent-2.0.so.5 /usr/lib64/libevent-2.0.so.5


 


   


2、 安装FastDFS（集群中所有服务器都执相同的安装）:


进入/usr/local/src目录


[root@FASTDFS-TRACKER-01 src]#


wget https://fastdfs.googlecode.com/files/FastDFS_v4.06.tar.gz


[root@FASTDFS-TRACKER-01 src]# tar -zxvf FastDFS_v4.06.tar.gz


[root@FASTDFS-TRACKER-01 src]# cd FastDFS


 


#由于定义/usr/local/fastdfs为fastdfs安装目录,所以需要修改make.sh


[gw@register2 FastDFS]$ vi make.sh


# /etc/fdfs 全部替换为 /usr/local/fastdfs/conf


# TARGET_PREFIX=/usr/local 修改为 /usr/local/fastdfs


# TARGET_CONF_PATH=/etc/fdfs 修改为 /usr/local/fastdfs/conf


附件：


 


#安装


[root@FASTDFS-TRACKER-01 FastDFS]$


./make.sh C_INCLUDE_PATH=/usr/local/libevent/include LIBRARY_PATH=/usr/local/libevent/lib


# 切换到超级管理员


[root@FASTDFS-TRACKER-01 FastDFS]# ./make.sh install


 


注意：如果安装时提示找不到命令，请查看是不是没有装perl安装环境


wget  http://www.cpan.org/src/5.0/perl-5.18.2.tar.gz


tar -zxvf  perl-5.18.2.tar.gz


cd perl-5.18.2


./Configure


make


make install


 


3、 配置Tracker Server（10.10.16.201，10.10.16.202）:


进入/usr/local/fastdfs/conf


#修改tracker.conf配置


[root@FASTDFS-TRACKER-01 conf]# vi /usr/local/fastdfs/conf/tracker.conf


绑定IP:


bind_addr=10.10.16.201 和 bind_addr=10.10.16.202


 


#设置日志目录


由base_path=/home/yuqing/fastdfs 改为 base_path=/home/fastdfs/tracker


store_group=group1


run_by_group= 改为 run_by_group=fastdfs


run_by_user= 改为 run_by_user=fastdfs


trunk_create_file_space_threshold = 20G


#开启自trunk_create_file_space_threshold定义server ID取代ip形式,方便内部网络服务器更换ip#**此方式要重点理解，4.0以后新特性


use_storage_id = true #使用server ID作为storage server标识


storage_ids_filename = storage_ids.conf  #


id_type_in_filename = id #文件名反解析中包含server ID,以前是ip


 


复制storage_ids.conf文件


[root@FASTDFS-TRACKER-01 conf]#


cp /usr/local/src/FastDFS/conf/storage_ids.conf /usr/local/fastdfs/conf/


#编辑storage服务器ID与IP地址的对应关系


[root@register1 conf]# vi /usr/local/fastdfs/conf/storage_ids.conf


#


100001  group1        10.10.16.203


100002  group1        10.10.16.204


 


防火墙打开8080和22122端口：


# vi /etc/sysconfig/iptables


增加：


-A INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT


-A INPUT -m state --state NEW -m tcp -p tcp --dport 22122 -j ACCEPT


重启防火墙:


# /etc/init.d/iptables restart


 


自定义tracker服务启动、关闭、重启脚本：


# su fastdfs


切换回fastdfs用户后


$ cd /home/fastdfs/tracker/


(1)启动脚本：


[fastdfs@FASTDFS-TRACKER-01 tracker]$ vi start_tracker.sh


/usr/local/fastdfs/bin/fdfs_trackerd /usr/local/fastdfs/conf/tracker.conf


 


(2)关闭脚本（使用FastDFS自带的stop.sh脚本）：


[fastdfs@FASTDFS-TRACKER-01 tracker]$ vi stop_tracker.sh


/usr/local/fastdfs/bin/stop.sh /usr/local/fastdfs/bin/fdfs_trackerd /usr/local/fastdfs/conf/tracker.conf


 


(3)重启脚本（使用FastDFS自带的restart.sh脚本）：


[fastdfs@FASTDFS-TRACKER-01 tracker]$ vi restart_tracker.sh


/usr/local/fastdfs/bin/restart.sh /usr/local/fastdfs/bin/fdfs_trackerd /usr/local/fastdfs/conf/tracker.conf


 


注意：千万不要使用kill -9参数强杀进程，否则可能会导致binlog数据丢失的问题


 


给自定义脚本赋予可执行权限：


[fastdfs@FASTDFS-TRACKER-01 tracker]$ chmod +x *_tracker.sh


 


启动tracker服务：


[fastdfs@FASTDFS-TRACKER-01 tracker]$ ./start_tracker.sh


查看是否启用成功：


[fastdfs@FASTDFS-TRACKER-01 tracker]$ ps -ef | grep fastdfs


root     27253 26511  0 10:03 pts/0    00:00:00 su fastdfs


fastdfs  27254 27253  0 10:03 pts/0    00:00:00 bash


fastdfs  27580     1  0 10:25 ?        00:00:00 /usr/local/fastdfs/bin/fdfs_trackerd /usr/local/fastdfs/conf/tracker.conf


fastdfs  27588 27254  5 10:25 pts/0    00:00:00 ps -ef


fastdfs  27589 27254  0 10:25 pts/0    00:00:00 grep fastdfs


[fastdfs@FASTDFS-TRACKER-01 tracker]$


（首次正常启动后会在/home/fastdfs/tracker目录下会产生data目录和logs目录）


[2014-01-12 16:33:54] INFO - local_host_ip_count: 2,  127.0.0.1  10.10.16.201


[2014-01-12 16:34:11] INFO - file: tracker_service.c, line: 920, the tracker leader is 10.10.16.202:22122


停用tracker服务：


[fastdfs@FASTDFS-TRACKER-01 tracker]$ ./stop_tracker.sh


 


重启tracker服务：


[fastdfs@FASTDFS-TRACKER-01 tracker]$ ./restart_tracker.sh


 


 


 


4、 配置Storage Server(10.10.16.203、10.10.16.204，两台机执行相同的配置操作)：


# cd /usr/local/fastdfs/conf


# vi storage.conf


绑定IP分别为:


bind_addr=10.10.16.203 和 bind_addr=10.10.16.204


 


base_path=/home/yuqing/fastdfs 改为 base_path=/home/fastdfs/storage


store_path0=/home/yuqing/fastdfs 改为 store_path0=/home/fastdfs/storage


tracker_server=192.168.209.121:22122 改为


tracker_server=10.10.16.201:22122


tracker_server=10.10.16.202:22122


（这里我们配了两个跟踪器服务）


run_by_group= 改为 run_by_group=fastdfs


run_by_user= 改为 run_by_user=fastdfs


# 与Nginx端口相同


http.server_port=80


 


防火墙打开23000和8888端口：


# vi /etc/sysconfig/iptables


增加：


-A INPUT -m state --state NEW -m tcp -p tcp --dport 23000 -j ACCEPT


-A INPUT -m state --state NEW -m tcp -p tcp --dport 8888 -j ACCEPT


重启防火墙:


# /etc/init.d/iptables restart


 


 


自定义storage服务启动、关闭、重启脚本：


# su fastdfs


切换回fastdfs用户后


$ cd /home/fastdfs/storage/


(1)启动脚本：


[fastdfs@FASTDFS-STORAGE-01 storage]$ vi start_storage.sh


/usr/local/fastdfs/bin/fdfs_storaged /usr/local/fastdfs/conf/storage.conf


 


(2)关闭脚本（使用FastDFS自带的stop.sh脚本）：


[fastdfs@FASTDFS-STORAGE-01 storage]$ vi stop_tracker.sh


/usr/local/fastdfs/bin/stop.sh /usr/local/fastdfs/bin/fdfs_storaged /usr/local/fastdfs/conf/storage.conf


 


(3)重启脚本（使用FastDFS自带的restart.sh脚本）：


[fastdfs@FASTDFS-STORAGE-01 storage]$ vi restart_storage.sh


/usr/local/fastdfs/bin/restart.sh /usr/local/fastdfs/bin/fdfs_storaged /usr/local/fastdfs/conf/storage.conf


 


注意：千万不要使用kill -9参数强杀进程，否则可能会导致binlog数据丢失的问题


 


给自定义脚本赋予可执行权限：


[fastdfs@FASTDFS-STORAGE-01 storage]$ chmod +x *_storage.sh


 


 


启动Storage Server:


[fastdfs@FASTDFS-STORAGE-01 storage]$ ./start_storage.sh


data path: /home/fastdfs/storage/data, mkdir sub dir...


mkdir data path: 00 ...


mkdir data path: 01 ...


mkdir data path: 02 ...


mkdir data path: 03 ...


（首次成功启动会初始化数据存储目录）


 


重启storage服务：


[fastdfs@FASTDFS-STORAGE-01 storage]$ ./restart_storage.sh


 


停止storage服务：


[fastdfs@FASTDFS-STORAGE-01 storage]$ ./stop_storage.sh


 






Storage Server（10.10.16.203和10.10.16.204）安装Nginx:


#安装Nginx使用fastdfs用户


#创建nginx日志目录


# mkdir -p /home/fastdfs/nginx/logs


# chmod a+w /home/fastdfs/nginx/logs


# chown -R fastdfs:fastdfs /home/fastdfs/nginx/logs


 


 


安装nginx1.4.4:


# cd /usr/local/src


#下载 nginx


[root@FASTDFS-STORAGE-01 src]#


wget http://nginx.org/download/nginx-1.4.4.tar.gz


解压：


# tar -zxvf nginx-1.4.4.tar.gz


# cd nginx-1.4.4


 


# ./configure --user=fastdfs --group=fastdfs --prefix=/usr/local/nginx --with-http_stub_status_module


 


如果产生如下错误提示：


checking for PCRE library ... not found


checking for PCRE library in /usr/local/ ... not found


checking for PCRE library in /usr/include/pcre/ ... not found


checking for PCRE library in /usr/pkg/ ... not found


checking for PCRE library in /opt/local/ ... not found


 


./configure: error: the HTTP rewrite module requires the PCRE library.


You can either disable the module by using --without-http_rewrite_module


option, or install the PCRE library into the system, or build the PCRE library


statically from the source with nginx by using --with-pcre=


 


（大概意思是：HTTP重定向模块的安装需要PCRE库，网上搜了一下，是需要安装pcre-devel与openssl-devel）


# yum install pcre-devel openssl openssl-devel


（RHEL6.3 x64 桌面版中已自带以上包）


 


再重新执行：


# ./configure --user=fastdfs --group=fastdfs --prefix=/usr/local/nginx --with-http_stub_status_module


 


#--with-http_stub_status_module 用来监控nginx的当前状态


Configuration summary


  + using system PCRE library


  + OpenSSL library is not used


  + md5: using system crypto library


  + sha1: using system crypto library


  + using system zlib library


 


  nginx path prefix: "/usr/local/nginx"


  nginx binary file: "/usr/local/nginx/sbin/nginx"


  nginx configuration prefix: "/usr/local/nginx/conf"


  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"


  nginx pid file: "/usr/local/nginx/logs/nginx.pid"


  nginx error log file: "/usr/local/nginx/logs/error.log"


  nginx http access log file: "/usr/local/nginx/logs/access.log"


  nginx http client request body temporary files: "client_body_temp"


  nginx http proxy temporary files: "proxy_temp"


  nginx http fastcgi temporary files: "fastcgi_temp"


  nginx http uwsgi temporary files: "uwsgi_temp"


  nginx http scgi temporary files: "scgi_temp"


 


执行安装：# make && make install


安装fastdfs-nginx-module插件：


# cd /usr/local/src


# wget http://fastdfs.googlecode.com/files/fastdfs-nginx-module_v1.15.tar.gz


# tar -zxvf fastdfs-nginx-module_v1.15.tar.gz


 


#修改插件配置文件


# vi /usr/local/src/fastdfs-nginx-module/src/config


ngx_addon_name=ngx_http_fastdfs_module


HTTP_MODULES="$HTTP_MODULES ngx_http_fastdfs_module"


NGX_ADDON_SRCS="$NGX_ADDON_SRCS $ngx_addon_dir/ngx_http_fastdfs_module.c"


CORE_INCS="$CORE_INCS /usr/local/fastdfs/include/fastdfs /usr/local/fastdfs/include/fastcommon/"


CORE_LIBS="$CORE_LIBS -L/usr/local/fastdfs/lib -lfastcommon -lfdfsclient"


CFLAGS="$CFLAGS -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='\"/usr/local/fastdfs/conf/mod_fastdfs.conf\"'"


 


#复制mod_fastdfs.conf到/usr/local/fastdfs/conf/目录下


# cp /usr/local/src/fastdfs-nginx-module/src/mod_fastdfs.conf /usr/local/fastdfs/conf/


 


上面编译时使用的动态链接库：


#启动nginx报错


#./nginx: error while loading shared libraries: libfastcommon.so: cannot open shared object file: No such file or directory


#解决办法 ---> 将/usr/local/fastdfs/lib 加入系统文件/etc/ld.so.conf中


# vi /etc/ld.so.conf


include ld.so.conf.d/*.conf


/usr/local/fastdfs/lib


 


#更新库文件缓存ld.so.cache


# /sbin/ldconfig -v


 


编译fastdfs-nginx-module模块：


# 重新编译安装nginx


# cd /usr/local/src/nginx-1.4.4


# ./configure --add-module=/usr/local/src/fastdfs-nginx-module/src


 


# make && make install


 


修改mod_fastdfs.conf配置:


# vi /usr/local/fastdfs/conf/mod_fastdfs.conf


 






 


# connect timeout in seconds


# default value is 30s


connect_timeout=30


 


# the base path to store log files


base_path=/tmp


 


# if load FastDFS parameters from tracker server


# since V1.12


# default value is false


load_fdfs_parameters_from_tracker=true


 


# if use storage ID instead of IP address


# same as tracker.conf


# valid only when load_fdfs_parameters_from_tracker is false


# default value is false


# since V1.13


use_storage_id = true


 


# FastDFS tracker_server can ocur more than once, and tracker_server format is


#  "host:port", host can be hostname or ip address


# valid only when load_fdfs_parameters_from_tracker is true


tracker_server=10.10.16.201:22122


tracker_server=10.10.16.202:22122


 


# the port of the local storage server


# the default value is 23000


storage_server_port=23000


 


# the group name of the local storage server


group_name=group1


 


# if the url / uri including the group name


# set to false when uri like /M00/00/00/xxx


# set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx


# default value is false


url_have_group_name = true


 


# path(disk or mount point) count, default value is 1


# must same as storage.conf


store_path_count=1


 


# store_path#, based 0, if store_path0 not exists, it's value is base_path


# the paths must be exist


# must same as storage.conf


store_path0=/home/fastdfs/storage


 


# set the log filename, such as /usr/local/apache2/logs/mod_fastdfs.log


# empty for output to stderr (apache and nginx error_log file)


log_filename=/home/fastdfs/nginx/logs/mod_fastdfs.log


 


 


 注意, 下载时如发现老报404. 将nginx.conf第一行 usr nobody 修改为 user root 重新启动后解决.


 


给 storage 的存储目录做一个软连接


# ln -s /home/fastdfs/storage/data /home/fastdfs/storage/data/M00


 


访问路径带group名（storage对应有多个group的情况），如/group1/M00/00/00/xxx:
      location ~ /group([0-9])/M00 {
            ngx_fastdfs_module;
        }


 


 


nginx配置简洁版本:


# vi /usr/local/nginx/conf/nginx.conf


user root root;


worker_processes  2;


error_log  /home/fastdfs/nginx/logs/error.log  notice;


pid        /home/fastdfs/nginx/logs/nginx.pid;


worker_rlimit_nofile 5120;


 


events {


    use epoll;


    worker_connections  5120;


}


 


http {


    include       mime.types;


    default_type  application/octet-stream;


    sendfile        on;


    tcp_nopush     on;


    keepalive_timeout  65;


    tcp_nodelay on;


    server {


        listen       80;


        server_name  localhost;


        #charset koi8-r;


        location ~/group([0-9])/M00 {


            #alias /home/fastdfs/storage/data;


            ngx_fastdfs_module;


        }


    }


}


 


 


#启动nginx：


# /usr/local/nginx/sbin/nginx


 


#重启nginx：


# /usr/local/nginx/sbin/nginx -s reload