[Celery]Run celery as daemon

This HOWTO is based on instructions provided at celeryd systemd configure and celery in daemon

In my previous HOWTO, we’ve configured celery and are able to run it using celery -A proj worker -l info manually. But in many case, you’d want to run it as daemon. Now let’s start to work.

NOTE:
I’m using another project in this article, please make changes accordingly.

Create a configuration file for celery daemon

My configuration file is /NewHorizons/celery.conf

CELERY_APP="NewHorizons"# Name of nodes to start, here we have a single nodeCELERYD_NODES="w1"# Where to chdir at start.CELERYD_CHDIR="/NewHorizons"# Python interpreter from environment, if using virtualenvENV_PYTHON="/usr/local/bin/python2.7"# How to call "manage.py celeryd_multi"CELERYD_MULTI="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryd_multi"# How to call "manage.py celeryctl"CELERYCTL="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryctl"# Extra arguments to celerydCELERYD_OPTS="--time-limit=300 --concurrency=8"# Name of the celery config module, don't change this.CELERY_CONFIG_MODULE="celeryconfig"# %n will be replaced with the nodename.CELERYD_LOG_FILE="/var/log/celery/%n.log"CELERYD_PID_FILE="/var/run/celery/%n.pid"CELERYD_LOG_LEVEL="INFO"# Workers should run as an unprivileged user.CELERYD_USER="web"CELERYD_GROUP="web"# Set any other env vars here too!#PROJET_ENV="PRODUCTION"# Name of the projects settings module.# in this case is just settings and not the full path because it will change the dir to# the project folder first.#export DJANGO_SETTINGS_MODULE="settings"

Create systemd service file

My service file is /usr/lib/systemd/system/celery.service

[Unit]Description=Celery workersAfter=network.target[Service]Type=forkingUser=webGroup=webEnvironmentFile=-/NewHorizons/celery.conf# **if you are using other environment files, for example, you configure you django.settings to read from env, include these files too**EnvironmentFile=-/LOCATION_OF_YOUR_ENV_FILEWorkingDirectory=/NewHorizons# run ExecStartPre as priviledged user and set up /var/runPermissionsStartOnly=trueExecStartPre=/usr/bin/mkdir -p /var/run/celeryExecStartPre=/usr/bin/chown web:web /var/run/celeryExecStart=/usr/local/bin/celery multi start $CELERYD_NODES \    -A $CELERY_APP --pidfile=${CELERYD_PID_FILE} \    --logfile=${CELERYD_LOG_FILE} --loglevel="${CELERYD_LOG_LEVEL}" \    $CELERYD_OPTSExecStop=/usr/local/bin/celery multi stopwait $CELERYD_NODES \    --pidfile=${CELERYD_PID_FILE}ExecReload=/usr/local/bin/celery multi restart $CELERYD_NODES \    -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \    --logfile=${CELERYD_LOG_FILE} --loglevel="${CELERYD_LOG_LEVEL}" \    $CELERYD_OPTS[Install]WantedBy=multi-user.target

Make directories for log and pid file

According to what you specified in your celery.conf, you should make sure the directories for CELERYD_LOG_FILE and CELERYD_PID_FILE exists before starting celery.service

As root, run

mkdir /var/log/celerymkdir /var/run/celery

Also make sure the User and Group you specified in celery.service have the permission to write to the above directories.

As root, run

chown web:web /var/log/celerychown web:web /var/run/celery

Now we can verify whether celery.service can be started.

systemctl start celery.servicesystemctl status celery.service