Django-Celery In Daemon

By
arruda
– 09/04/2012Posted in: Django, programação, Python, Selecionados, Web

Hello everyone, I’m doing this post because I just got stuck in this a while ago and thought it would be nice to share.

Django-Celery

First of all you need to install it and configure the basics(this all is well explained in the docs), so basically you can do:

Then you need to install a broker server(like RabbitMQ):

After this is done, you need to put some basic configurations for celery in your settings file, like this:

Also don’t forget to run syncdb or migrate(if using south).

Now if you want to run celery, you do it like:

Daemonizing…Not so fast!

This part is not so intuitive for those who are starting to use this tool, but after you get what need to be done, it’s all fast and furious.
This is, in my opinion, because of some misleading details in the celery daemon docs. So I’ll try to make it as explained as I can.

Create a Configuration File For The Daemon

In this file there will be some variable used to run the celery as a daemon, like:
Where is the project folder;
How many workers;
Where will be the log files…
And many others.

But it will also contain any environment variable that you need in your project!
So if you have a different settings for development and for production using an env var, you’ll have to set this in the configuration file as well.
Or else you’ll might and up having the celery working with the sqlite3.

You create this file as: /etc/default/celeryd
And put something like this inside it:

Get a init.d script

Yeah, this is strange, but after some time I understood this.
So basically the file that they tell you to run in the docs, you need to get and put it there yourself…
And this init.d script will run the celeryd using the configurations that you’ll set in the /etc/default/celeryd.

You can get the file in:
https://raw.github.com/ask/celery/1da3aa43d1e6de525beeda398d0acb8841d5b4d2/contrib/generic-init.d/celeryd

This file must go to /etc/init.d/celeryd
And don’t forget to make it executable: chmod +x /etc/init.d/celeryd

Using the Daemon

To use it, just do like this:

Some Details

So, if you run the celery without the daemon and have no problem, but when running the daemon it tells you that some celery table was not created, then probably your celeryd is not using the correct database.

Also don’t forget that the CELERYD_USER and CELERYD_GROUP must have permission to use the manage.py, and to access the CELERYD_LOG_FILE and CELERYD_PID_FILE folders!

And of course. You need to create the folders for CELERYD_LOG_FILE and CELERYD_PID_FILE.

And again, don’t change the CELERY_CONFIG_MODULE=”celeryconfig”. Since you’re using django-celery there is no celeryconfig.py in your project, but you need to set this so that it will get the default one(that it’s in the celery package).

That’s it! Hope this will help others that where lost in this(especially in the CELERY_CONFIG_MODULE) and in the init.d script.

Tags: , , , , , , ,

About arruda

Adoro programar, descobrir novas frameworks ágeis e suas diversas aplicações.