Archive

Archive for the ‘Django’ Category

Satchmo installation

November 14th, 2011 5 comments

Been playing with Satchmo today. One issue I ran into, is I installed the Trunk version of Django, which Satchmo apparently isn’t compatible with; it apparently expects Django 1.3. The easiest way to get started with a very fresh install of Satchmo is to go through the following steps.

Setup and activate virtual environment:

~/projects$ virtualenv test
~/projects$ cd test
~/projects/test$ source bin/activate

Make sure you have PIL installed:

sudo apt-get install python-imaging

Install Satchmo:

~/projects/test$  hg clone https://chris1610@bitbucket.org/chris1610/satchmo
~/projects/test$ cd satchmo
~/projects/test/satchmo$ python setup.py install

Install all the other requirements:

~/projects/test/satchmo$ pip install -r scripts/requirements.txt

Setup sample store

/projects/test$ python source/satchmo/scripts/clonesatchmo.py
/projects/test$ cd store
/projects/test/store$ python manage.py runserver

Then just shoot your browser to:

http://localhost:8000
Categories: Django, Satchmo Tags:

Web framework focus

September 7th, 2011 6 comments

Over the years I’ve used, read about, and played with lots of different web frameworks. One of the things I’ve concluded is that none of them are perfect. As with the languages they live on top of, each of these frameworks is a tool that fits certain tasks better than others.

One other thing I’ve realized is that the focus of the community effects the type of web sites that are made. In some frameworks there’s a huge emphasis on Ajax integration (such as web2py), then there are others that are more focused with data manipulation and control of flow (such as Django). Now one other interesting thing is that this focus leads to different types of sites being made.

Two mammoth examples that come to mind are Amazon and Google. While they’re not the most perfect examples for these purposes, I bring them up because they both are layered more than the average site we work on. There’s Java or C++ behind doing all the number crunching. Then layers of other languages in between. One thing about sites like these is, although they all have Ajaxy components, not one of those pieces is superfluous. It takes so much work to tie one of these features in that there just won’t be anything extra.

With a site made with Lift, Wicket or Web2Py I think you get the sense that they’re giving away Ajax for free and they feel that they have to make use of it. Rails sites also seem to be heavier on Ajax than the average site. Django sites on the other hand for the most part are lighter on this feature since it’s not as simple as to tie in these pieces.

These are just observations I’ve made and I’m making no judgment call on what the right amount of Ajax is, or the right framework. It’s just interesting to see how tools seem to be effecting what gets produced even though you could essentially produce the same exact sites with all of these tools.

Categories: Django Tags:

Impressive Satchmo Sites

August 29th, 2011 2 comments

I’m starting a project for a client in Satchmo and wanted to show him some sample stores produced with the technology so that he’d feel more comfortable with what it’s capable of. The following are the standouts after going through all the items on this list:

  1. blacklockjewellery.com
  2. qwertee.com
  3. livestrongfitness.com
  4. jeepcollins.com
  5. spoonnyc.com
  6. snowsportsonline.com
  7. zutano.com
  8. abodeliving.co.uk
  9. shopjoielle.com
Quick note: while scanning through these sites it became very obvious how important is for a site to have a quick response time. When I clicked some of the links on the stores that I went to the site just wouldn’t respond to my requests. It just feels a lot better when the site is snappy and ready to serve you.
Categories: Django, Satchmo Tags:

Backing up your database

June 23rd, 2011 2 comments

Let me start by saying that there are MANY ways to backup databases and this might not be the absolute best way, but it definitely works. I’m backing up my PostgreSQL database that happens to be populated by Django, but you can muck with things as you see fit.

So, one of the decisions I had to make is whether to save a representation of the data or to just dump the database data itself. The first option would be something like this:

python manage.py dumpdata --format=yaml

Second option is to talk directly to the database. The benefit of this option is you get everything about the database in its current setup. Let’s say you made some changes to the schema directly within PostgreSQL. If you dumped the data into YAML or JSON you wouldn’t get those changes to the database itself. So it’s probably prefereable to go straight to the DB like so:

pg_dump <DATABASE NAME>

What I ended up with is a nightly backup on the database server. Then I pull that data from onto a second server using rsync.

 

Steps it took:
1. Setup e-mail sending, so that I get cron e-mails.

$ sudo apt-get install mailutils
$ sudo apt-get install exim4-config
$ sudo dpkg-reconfigure exim4-config

Test sending of mail from server

$ echo Test | mail -s Test <YOUR E-MAIL>

E-mail sending log lives here:

/var/log/exim4/mainlog

 

2. Setting up cron job on server to stash database schema and data. What I did is ran the cron job as the postgres user so that it would have access to the database.

$ su postgres

$ crontab -e
MAILTO="<YOUR E-MAIL>"
0 1 * * * pg_dump > "<PATH TO BACKUPS>/$(date +\%Y-%m-%d).psql"

 

3. Setting up pulling data onto another machine; setup e-mail just as you did on server machine

$ crontab -e
MAILTO="<YOUR E-MAIL>"
0 3 * * * rsync -vr --size-only<USERNAME>@<SERVER>:<PATH TO BACKUPS ON SERVER> <PATH TO BACKUPS ON SECOND MACHINE>
Categories: Django, PostgreSQL, Ubuntu Tags:

Revisiting long running processes in Django

March 2nd, 2009 7 comments

So in my last post on this topic I discussed different ways to handle the problem of running code that wold otherwise cripple your web server, by pushing them off to another process.

It was mentioned that there may be other ways to attack the problem, either through Python threads or Django’s signals. This post is a review of those two suggestions.


Django’s signals

I’ll attack the simpler option first, since the response to whether this solution is viable takes all of one word: no. Signals are not asynchronous by nature and therefore they do not work as a solution to the problem. It is true that you can setup signals to work asynchronously, but you’d have to employ one of the methods discussed to handle that. So if you were to use signals to handle some asynchronous work then all they would really be doing is changing the layout of that code. You’d still need something else to actually handle the asynchronous work.

I’ll offer up a simple example of how signal code looks just for completeness.

There are three pieces to signals.
1.The signal itself
2.The sending of the signal
3.The listeners of the signal

1. Setting up the signal is as simple as including a signals.py file in your application.

import django.dispatch
test_sig = django.dispatch.Signal()

2. All you need to do to send out the request to all the listeners is call “send” off the signal. In the following code snippet I’m calling the send from one of the view functions.

from django.shortcuts import render_to_response
from django.template import RequestContext
from my_project.my_app import signals
def home(request):
    signals.test_sig.send(sender=None)
    return render_to_response('home.html',
                              context_instance=RequestContext(request),)

3. To setup a listener you call the connect function on the signal and to have link the listener to an instance method or function. Beware: the connect call has to come after the function definition. This code can really sit anywhere within your project.

def signal_func(*args, **kwargs):
    print "Do something here"
test_sig.connect(receiver=signal_func)

Python’s threads

(Update: See Malcom’s comments below, which kind of nullify the positive things I wrote about threads and this problem.)

Threads are actually a very fitting way to handle this scenario. Of course the thought of using threads did cross my mind when first attacking this problem, but it’s the kind of thing I’ve ignored as an option because I know you can get yourself in trouble when you start fiddling with an animal like this, but at the same time they are very powerful and useful if used properly.

It’s really a very simple way to resolve this problem, just push off the work to another thread within the same Python application. And in Python making use of threads is actually quite simple.

Here’s a simple intro into Python threads. There are two levels of thread libraries. The lower level object is called ‘thread’ and the higher level threading module is call ‘threading’. In most cases you will probably be working with the ‘threading’ module.

Another thing to know about is the GIL (Global interpreter lock). Since Python is an interpreted language and threads seem to all be working at the same time, there has to be a way to insure that they are not both using the same objects at the same time. The GIL locks the interpreter so that each thread can safely use Python’s internal data structures. The lock keeps moving between threads pretty frequently (I think it’s something like every 100 bytes).

Important functions to know about from threading:
__init__: initializes thread
start: starts the thread
run: code that actually runs when thread is activated
join: when called waits for thread to finish before continuing

A very simple example:

from django.shortcuts import render_to_response
from django.template import RequestContext
import threading

class TestThread(threading.Thread):
    def run(self):
        print "%s starts" % (self.getName(),)
        import time
        time.sleep(5)
        print "%s ends" % (self.getName(),)

def threadView(request):
    testThread = TestThread()
    testThread.start()
#    testThread.join() # if you remove the first pound sign on this line the becomes synchronous.
    print "Prints right when requested just to show that the other thread is off on it's own"
    
    return render_to_response('test.html',
                              context_instance=RequestContext(request),)
Categories: Django, Python Tags:

Running long processes in Django

February 2nd, 2009 8 comments

My original issue was that I had this piece of code in my Django project that was taking way too long to run, sometimes even leading to a load time out. The particular situation happens very infrequently (at most once a week), so it was just a matter of getting the process to run successfully. Along the way, though, I learned a lot about spawning processes, distributed computing, and the different options we have in the Python community. The different approaches are just ways to get the processing to be done outside the Django process, of course.


cron and queue

cron
I will first start with the recommended process of taking care of this issue. You can setup a cron job to run some code that checks every minute to see if there’s anything to process and then run the appropriate code. Cron jobs are pretty simple to setup and pretty effective. All you need to do is edit the crontab with the time intervals you want the service to be run and your code takes care of the rest.

django-queue-service
The cron part of this solution takes care of when the processing happens, but what handles why it happens? So for that aspect of it you’ll need some way to know when there is processing to be done. There are of course multiple ways to handle this. Update a table in your database, update a file, or a folder… One way is to use django-queue-service. This method requires you to run the queue service as another django instance and then make requests to it. The sample code from the projects page looks as such:

import httplib2
client = httplib2.Http()
body_string = "message=%s" % (message_to_be_inserted_in_queue,)
uri = 'http://dqshost:8000/q/queue_name_here/put'
(response,content) =client.request(uri=uri,
method='POST',
body=body_string)
if response['status'] != 200:
print "Queue didn't accept the input"

While this method does make the most use of Django of all the methods I’ll discuss, I really have problems with it. It’s heavy handed, unnecessary, and I can’t even tell if there are security concerns. Let’s say that someone set this method up improperly and exposed this django instance to the outside world…

queue: Python module
There was some Python module I came across, which I can’t seem to find. When I find it I’ll post a link, but the way it worked is that it was a file based queue. It would add files to folders. And there were five different folders based on the status of the item of the queue. Ready, active, complete… This was a better way to handle the queue than the django-queue in my opinion, but still seemed a bit uncomfortable.


Read more…

Categories: Django, Python Tags:

Pinax update: breaks external dependencies

February 2nd, 2009 5 comments

Have you been working off the Pinax trunk only to be surprised by the following message after an update:
 Error: No module named notification

Basically what happened is summarized on the django-hotclub page, which I apparently should start paying closer attention to:
http://code.google.com/p/django-hotclub/wiki/MovingToDistutils

What this all means is that you will need to change your setup a bit to get Pinax to work in its new setup… What I had to do first was install pip and virtualenv
sudo easy_install pip
sudo easy_install virtualenv

If you need to setup easy_install or you have any trouble with the following step, try updating python-setuptools
sudo apt-get install python-setuptools

You will also need to install git and bzr as mentioned in the wiki page. On my Ubuntu box, it was simple as installing git-core and bzr
sudo apt-get install git-core
sudo apt-get install bzr

After that you need to run the following command to download the apps (Feb 6, typo fixed, thanks Vernon):
sudo pip install -r pinax/requirements/external_apps.txt

And then back to your normal:
python manage.py syncdb
python manage.py runserver

(of course this is just on your dev box…)

hope that helps.

Categories: Pinax, Ubuntu Tags:

Initial DB data

September 2nd, 2008 No comments

Within your application on the same level as your models place a folder named “fixtures”

In my current setup that means “app/models/fixtures/”, but in a normal setup that would be “app/fixtures/”

Within that file all you need to do is add a file with the extension “xml” or “json”. The syntax looks a lot nicer for “json”:

[
{
"model": "app_name.model_name",
"pk": 1,
"fields": {
"name": "A",
"value": 5
}
},
{
"model": "app_name.model_name",
"pk": 3,
"fields": {
"name": "C",
"value": 15
}
}
]

When you’re all set with your data all you need to do is run “python manage.py loaddata json_filename” (leaving off the “json” extension)

Categories: Django Tags:

Code bloat in your Django projects: Models

August 28th, 2008 No comments

My current project’s getting a bit large, so I figured I should start subdividing the code a bit. The project’s already split up into multiple applications, but my main application is growing…

The first thing I’ve done is split up my models.py into multiple files and moved them into a folder named models.

Folder structure is: project/application/models/model_file.py

All that’s left to be done is to add an _init_.py to that folder and import all your models within the __init__.py. You’ll also need to make sure the app_label set on each of your models.

For example:
class Shape(models.Model):
value = models.CharField ()
class Meta:
app_label = "my_project"

This may turn into a series…lets see if there’s anything else that needs better organization.

Categories: Django Tags:

Django: messaging

August 5th, 2008 3 comments

When completing something like deleting an item or adding a user, it’s usually nice to report a message to the user to let them know what just happened. I also find it useful to send the user somewhere they’d “want” to go. For example, if they just created a user, send them to the list of users page, so they see the change they just made.

I’m also a big advocate of keeping URLs clean.

Because of these concerns the best answer in Django is to do an HttpResponseRedirect. Now the problem with that is that you can’t pass data to a redirect. To get around this there are messages that you can send to the users session itself.

The mechanism isn’t that clean, but it does work, and it does do the job.

So if you have a buycbdproducts in user all you need to do to add messages is:
request.user.message_set.create(message='User successfully created.')

Now the flip side to the equation is that you “get” the message(s). So how this works is when you set a message you’re actually adding a message to a list. When you”get” the message, you’re actually getting the list (may be more than one message). And if you don’t “get” the message at the right time, the messages will just build up. But for the most part this system works just fine. To get the messages you just do:
messages = request.user.get_and_delete_messages()

Then just pass ‘messages’ to the view and show the list to the user.

Categories: Django Tags: