Archive for February 2nd, 2009

Running long processes in Django

February 2nd, 2009 8 comments

My original issue was that I had this piece of code in my Django project that was taking way too long to run, sometimes even leading to a load time out. The particular situation happens very infrequently (at most once a week), so it was just a matter of getting the process to run successfully. Along the way, though, I learned a lot about spawning processes, distributed computing, and the different options we have in the Python community. The different approaches are just ways to get the processing to be done outside the Django process, of course.

cron and queue

I will first start with the recommended process of taking care of this issue. You can setup a cron job to run some code that checks every minute to see if there’s anything to process and then run the appropriate code. Cron jobs are pretty simple to setup and pretty effective. All you need to do is edit the crontab with the time intervals you want the service to be run and your code takes care of the rest.

The cron part of this solution takes care of when the processing happens, but what handles why it happens? So for that aspect of it you’ll need some way to know when there is processing to be done. There are of course multiple ways to handle this. Update a table in your database, update a file, or a folder… One way is to use django-queue-service. This method requires you to run the queue service as another django instance and then make requests to it. The sample code from the projects page looks as such:

import httplib2
client = httplib2.Http()
body_string = "message=%s" % (message_to_be_inserted_in_queue,)
uri = 'http://dqshost:8000/q/queue_name_here/put'
(response,content) =client.request(uri=uri,
if response['status'] != 200:
print "Queue didn't accept the input"

While this method does make the most use of Django of all the methods I’ll discuss, I really have problems with it. It’s heavy handed, unnecessary, and I can’t even tell if there are security concerns. Let’s say that someone set this method up improperly and exposed this django instance to the outside world…

queue: Python module
There was some Python module I came across, which I can’t seem to find. When I find it I’ll post a link, but the way it worked is that it was a file based queue. It would add files to folders. And there were five different folders based on the status of the item of the queue. Ready, active, complete… This was a better way to handle the queue than the django-queue in my opinion, but still seemed a bit uncomfortable.

Read more…

Categories: Django, Python Tags:

Pinax update: breaks external dependencies

February 2nd, 2009 5 comments

Have you been working off the Pinax trunk only to be surprised by the following message after an update:
 Error: No module named notification

Basically what happened is summarized on the django-hotclub page, which I apparently should start paying closer attention to:

What this all means is that you will need to change your setup a bit to get Pinax to work in its new setup… What I had to do first was install pip and virtualenv
sudo easy_install pip
sudo easy_install virtualenv

If you need to setup easy_install or you have any trouble with the following step, try updating python-setuptools
sudo apt-get install python-setuptools

You will also need to install git and bzr as mentioned in the wiki page. On my Ubuntu box, it was simple as installing git-core and bzr
sudo apt-get install git-core
sudo apt-get install bzr

After that you need to run the following command to download the apps (Feb 6, typo fixed, thanks Vernon):
sudo pip install -r pinax/requirements/external_apps.txt

And then back to your normal:
python syncdb
python runserver

(of course this is just on your dev box…)

hope that helps.

Categories: Pinax, Ubuntu Tags: