How To Setup MogileFS

refer: http://durrett.net/mogilefs_setup.html

 

HowTo Setup MogileFS

Version:0.04, October 7, 2006

Author:Brett G. Durrett (first name at last name dot net)

Source:http://durrett.net/mogilefs_setup.html

 

Note:These directions were written for MogileFS < 2.0 – I havenot done a 2.0 install but I have been told by very good sources that2.0 is the way to go now as it is backward compatible (making sureyou don't update the schema if you are already running < 2.0 in amixed environment) and it is in production by LiveJournal (oh, and itis better code with better features).

Overview

 

Thisdocument explains how to setup and configure a basic MogileFSinstallation. It is intended for the novice systems administrator andshould enable anybody with the skills to install a basic Linux distroto get MogileFS up and running on it.

I amgoing to assume you have three roles for this setup. A machine shouldbe able to handle more than one role. I will use hostsmogiledb.yourdomain.com, mogiletracker.yourdomain.com andmogilestorage.yourdomain.com – rename these to meet your needs.

Getting MogileFS

If youlook around you can find MogileFS directly on the Danga website, inan abandoned CVS repository and in the current SVN repository. If youpull from the website you will not have admin tools to make setupeasier. If you pull from CVS you will have a version from October,2004. If you pull from SVN you will have a shiny new version thatworks best with this documentation.

Usingsvn (http://subversion.tigris.org/),checkout the latest and greatest version (the example below pullsfrom the trunk).

$ mkdir mogilefs-src
$ cd mogilefs-src
$ svn checkout http://code.sixapart.com/svn/mogilefs/trunk

Installation

Creating aDatabase

Settingup a MySQL server is beyond the scope of this document – thereare packages available for most distributions, find one that suitsyour needs. Once you have a MySQL server up and running on hostmogiledb.yourdomain.com, create a table and user for MogileFS. Somelibraries don't play well with new MySQL passwords – if you usethese, set the password using the “OLD_PASSWORD”function. Make sure you change the password to something better thanthe example.

# mysql
mysql> CREATE DATABASE mogilefs;
mysql> GRANT ALL ON mogilefs.* TO 'mogile'@'%';
mysql> SET PASSWORD FOR 'mogile'@'%' = OLD_PASSWORD( 'sekrit' );
mysql> FLUSH PRIVILEGES;
mysql> quit

You will also need to create theschema – that is covered later in this document.

 

Setting up the Trackers and StorageServers

You needto obtain and install the MogileFS-specific perl modules and theirdependencies before MogileFS will install successfully. Hopefully youpulled the MogileFS source from the SVN repository – if so, theserver components will be located in the trunk/server directory. Youprobably need to install dependent modules first (see below) but onceall dependencies have been met, you can install Mogile with thefollowing commands (starting in the top of the SVN directory youpulled):

# cd trunk/server/
# perl Makefile.PL
# make
# make test
# make install

If you get any errors during thisprocess it will probably be errors telling you that a dependentmodule is missing. If during the 'make test' step you get the error,"t/00-startup....DBI connect('mysql','root',...) failed: Can'tconnect to local MySQL server through socket'/var/run/mysqld/mysqld.sock' (2) at t/lib/mogtestlib.pl line 16"it can probably be ignored – it should not be necessary forMySQL to be running on any host other than mogiledb.yourdomain.com.

Youprobably want to install some helpful utilities on each tracker orstorage server as well (these will be needed for laterconfiguration). These are located in the trunk/utils directory andcan be installed with the following commands (starting in the top ofthe SVN directory you pulled):

# cd trunk/utils/
# perl Makefile.PL
# make
# make test
# make install

You also want the API – theutilities will require this. These are located in the trunk/api/perldirectory and can be installed with the following commands (startingin the top of the SVN directory you pulled):

# cd trunk/api/perl
# perl Makefile.PL
# make
# make test
# make install

Installing Perl Module Dependencies

As forPerl modules, you need (at least) Perlbal and Danga-Socket and anydependencies. If you are not sure how to locate the dependentmodules, consider using CPAN (http://cpan.org/)as it can install a module and all of the dependencies. If youalready have a reasonable Perl installation you probably have most ofthe non-MogileFS modules already.

Thebasic process for installing most perl modules is:

extractthe module:

# tar -xvzf Linux-AIO-1.9.tar.gz

Create the makefile:

# cd Linux-AIO-1.9
# perl Makefile.PL 

Build the code:

# make 

Test the code (some modules may nothave this step but it is okay to run anyway):

# make test 

Install it:

# make install

If you see any errors stating that amodule dependency is missing, obtain and install the missing modulefirst and try again.

Setup

DatabaseConfiguration

Thedatabase is empty and will need a schema applied. The ' trunk/server'directory has a utility named 'mogdbsetup' to make this processsimple. By default it assumes the database is located on localhost soif you are running it from a different host you will need to providethe host name on the command line.

# ./mogdbsetup --dbhost=mogiledb.yourdomain.com --dbname=mogilefs --dbuser=mogile --dbpassword=sekrit 

Again, make sure you replace the hostand password so that they match you database configuration fromabove.

Themogdbsetup utility does not specify a table type by default so yourtables will match the defaults for your database. In many cases thiswill mean that you end up with MyISAM tables. If you prefer InnoDBtables you will either need to make sure your database defaults toInnoDB or you can manually convert the tables (both of these areoutside of the scope of this document but there are plenty ofexamples out there).

 

TrackerConfiguration

On eachtracker server (mogiletracker.yourdomain.com), create a configurationfile at /etc/mogilefs/mogilefsd.conf with the following:

db_dsn DBI:mysql:mogilefs:mogiledb.yourdomain.com
db_user mogile
db_pass sekrit
conf_port 6001
listener_jobs 5

db_dsn points to your databaseinstance. If you are running the database on the same machine as thestorage server you can omit ":mogiledb.yourdomain.com: and itwill use the local machine. db_user and db_pass should match the userand password you configured when setting up your database.

Theprogram 'mogilefsd' will not run as root so you will need to run thisas a non-root user. To create a user for this, enter the followingcommand and follow the prompts to create the "mogile" user:

# adduser mogile

In order to use the tools to setup thestorage servers you will need to have the trackers running. Refer to"Starting Trackers", below.

 

Storage ServerConfiguration

On eachstorage server, create the storage directory (make sure it has accesspermissions for the user you will use to run mogstored):

# mkdir /var/mogdata

Configure it:

On eachstorage server, create a configuration file at/etc/mogilefs/mogstored.conf with the following:

httplisten=0.0.0.0:7500
mgmtlisten=0.0.0.0:7501
docroot=/var/mogdata

Use 'mogadm' to add each storageserver to the database. This requires that the trackers arealready running so if you have not already started them, refer to"Starting Trackers",below. You need to supply the Perl lib path which has the'MogileFS.pm' perl module installed – this was installed if youinstalled the API in the "Setting up the Trackers and StorageServers" section above. The following example would add the hostmogilestorage.yourdomain.com as a storage server, assuming thatmogilestorage.yourdomain.com had an IP address of 192.168.42.3(listening on port 7500) and your tracker had an IP address of192.168.42.1 (listening on port 6001):

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001 host add mogilestorage --ip=192.168.42.3 --port=7500 --status=alive

You can confirm that your host(s) wereadded with the following command;

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001 host list

You also need to add devices for eachstorage host. If you don't provide a device id mogadm is supposed todo it for you... in practice I find it dies with an error, so youwill need to manually add a unique device id after the host:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001 device add mogilestorage 1

Finally, add the correctly-name device(folder) to each storage host. I have been unable to get the tools tohandle this well, so I am probably doing something wrong. As aworkaround, I used the modadm device list command to see what devicenames were assigned and then I added the folders to my storage hosts.Run the following command:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001 device list

It will list each host and the devicename followed by its status and storage available. Here is exampleoutput:

mogilestorage [1]: alive
used(G) free(G) total(G)
dev1: alive 0.892 67.772 68.664

This means "mogilestorage"has a host id of "1" and it has one device named "dev1"on it and that device is in the "alive" state (your otherstatistics will probably be zeros). Using the example output above,you would simply create the directory onmogilestorage.yourdomain.com:

# mkdir -p /var/mogdata/dev1

Finally, confirm your devices areconfigured:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001 device list

Running MogileFS

 

Starting StorageServers

Starteach storage server (mogilestorage.yourdomain.com) by running thefollowing command as root:

# mogstored --daemon

StartingTrackers

Trackerswill not run as root so you will need to run them as another user. Ifyou created the "mogile" user when seetingup the trackers,the following commands will work (assumes you start logged in tomogiletracker.yourdomain.com as root):

# su mogile
$ mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon
$ exit

You canconfirm that the trackers are running with the following command:

# ps aux | grep mogilefsd

If you don't get a list of runningprocesses the trackers are not running.

Try It

 

Do a Quick SanityTest

The'mogadm' tool can be used to make sure your trackers are functioning.You need to supply the Perl lib path which has the 'MogileFS.pm' perlmodule installed – this was installed if you installed the APIin the "Settingup the Trackers and Storage Servers" section above. Thefollowing example would check all mogile components using thetrackers at IP address 192.168.42.1 and 192.168.42.2, both listeningon port 6001:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001,192.168.42.2:6001 check

 

Try it with RealData

Create adomain:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001,192.168.42.2:6001 domain add testdomain

Add a class to the domain:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001,192.168.42.2:6001 class add testdomain testclass

Troubleshooting

Thissection is still very incomplete.  Let me know if you have othercommon problems that need to be added.

WhenI run mogadm I get "Unable to retrieve host information fromtracker(s)":

mogadmrequires the tarckers to be running before it is run.

Whenstarting the storage daemon I get "ERROR: Directory not foundfor service mogstored":

You didnot create a storage directory or you are starting the mogstored as auser that does not have access to the directory.

ProblemsConnecting:

Makesure your firewall is open. Using the examples in this document, port7500 and 7501 needed on storage servers, 6001 on trackers.

Whiletesting I get “MogileFS::Backend: couldn't connect to mogilefsdbackend at /usr/local/share/perl/5.8.4/MogileFS.pm line 56”:

Makesure your tracker connects to the database:

# su mogile
$ mogilefsd -c /etc/mogilefs/mogilefsd.conf

I get a "REQUEST FAILURE"on "Checking devices..." when doing a mogadm check:

Confirmthe devices (folders) exist in the /var/mogdata directory and thatthe use running the mogstored process has full permissions to thesedirectories. If the device does not exist, add it – it willtake a few seconds for mogadm check to reflect the fixed directory.For example, if mogilestorage.yourdomain.com had the device "dev1"on it, you would add the directory:

# mkdir -p /var/mogdata/dev1

Document RevisionHistory

2006-10-07Version 0.04 Brett G. Durrett: changed two typos of /vag to /var,added plug for 2.0

2006-07-03Version 0.03 Brett G. Durrett: highlighted need to start trackersbefore using mogadm (and added troubleshooting reference), formattingchanges.
2006-06-26 Version 0.02 Brett G. Durrett: minor typocorrections and fixed characters trashed in converting document tohtml
2006-06-25 Version 0.01Brett G. Durrett: created initial document