Iraniblog

I wrote a Python script that I want to run every five minutes through a crontab. The script ran fine and linked with my local libraries until I ran it through the crontab and it couldn’t find my local libraries. After a bit of thought, I realized that the crontab was not running through the same shell environment as I expected it to. Apparently the shell is set in /etc/crontab file and there it was set to bash. That in turn was calling the wrong version of Python, and that was why my local Python scripts weren’t being found.

There are four different ways around this:
1) Modify /etc/crontab to hit the right shell; in my case (first line modified):
/etc/crontab

SHELL=/bin/tcsh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

crontab -e

*/5 * * * * python /my/script.py

2) Add the shell value to the top of the custom crontab (crontab -e); in my case:

SHELL=/bin/tcsh

*/5 * * * * python /my/script.py

3) Run the actual command through the tcsh as so:

*/5 * * * * tcsh -c "python /my/script.py"

4) Directly request the right version of Python:

*/5 * * * * /tools/bin/python /my/script.py

Most people don’t know this, but you can also run emacs from within your shell (like vi). There are some moments where it just doesn’t make sense to load the emacs GUI, and it’s just nice to know that we don’t have to use vi in those circumstances. All you need to do is add the “-nw” tag when running emacs to load it up in the shell. I also add the “-Q” so that there’s no fancy color schemes introduced.

emacs -nw -Q

The most common usecase for doing this is when you’re committing something from the shell and want to add a comment. Usually the default is vi in these circumstances, but it doesn’t really have to be…

Apparently documentation is important sometimes… I’m writing this Python script that’s run in both Linux and Windows. My development happens on a Linux box, so until I move onto testing my scripts on Windows, I don’t realize the discrepancies. I wanted to get the username of the user, and there is this command in the os module called getlogin() which worked perfectly on my Linux box. In Windows it raises an exception. I checked the documentation and it says that getlogin() is only supported in UNIX, go figure… The way it recommends to get the username is from the os environment variable dictionary. The documentation tells you to use the “LOGNAME” key. I’ve found that “LOGNAME” does not work in Windows, but “USERNAME” works in both…

import os
os.getenv("USERNAME")

Anyways, tread lightly and always look at the documentation, even when something seems to be named so appropriately in such a mainstream language/module. One of the things that I really love about Python is that it usually does what you expect it to do. I guess nothing’s perfect though, and it pays to always be on your toes.

I am currently writing a lot of file manipulation code and keep running into snags that’s led to a lot of interesting debugging…

The most recent issues have been with editing files with interesting permissions set. I’ve been messing with the sticky bit, which you trigger so that the files owner is the user that runs the application regardless of who actaully runs the application.

The situation I’m interested in here, is that I set a file to be owned by a user named ‘admin’ and set the sticky bit to active (so that the file would always be run by ‘admin’). Then I tried running that file, and having that script manipulate a file for which it needed to be the user ‘admin’, and that failed, since the file was actually not being run by the ‘admin’ user.

After a bunch of debugging I realized that although the script is set to the ‘admin’ user, the actual thing running the application is the Python interpreter, which is being run by my logged in user (not the ‘admin’ user’).

The following code snippets will take you through the scenario so that you can reproduce and feel the silly complexity of Linux file manipulation on your own.

Let’s say our file is named ‘sticky’ and we are just going to work in our current directory:

> ls -ld sticky
-rw-rw-rw-   1   my_user   group   1K 2009-07-24 12:00 sticky

> sudo chown admin:group sticky
> sudo chmod 750 sticky
> sudo chmod ug+s sticky

> ls -ld sticky
-rwsr-s---   1   admin   group   1K 2009-07-24 12:00 sticky

> ./my_python_script.py

Right there is where my problem happened. I couldn’t run any of the file manipulation stuff on the ‘sticky’ file, even though my script had its sticky bits set and its owner was the same as that of the file I wanted to manipulate. One way around this is to write a wrapper in C++ that then calls your Python script. As such:

my_cpp.cpp:

#include <sstream>
#include <cstdlib>
#include <iostream>
#include <errno.h>
using namespace std;

int main(int argc, char **argv)
{
    string cmd = "./my_python_script.py";
    int return_code = execvp(cmd.c_str(), argv);

    if (-1 == return_code)
        cout<<errno<<": "<<strerror(errno)<<endl;

    return 0;
}

my_python.py:

print "Inside Python file"

You would run your code as such:

> chmod 750 my_python.py my_cpp.cpp
> g++ my_cpp.cpp -o my_runner
> my_runner

OK, so maybe you can see a bit of a pattern here. I’ve been messing with git trying to get to the point where I can do what I want with it. Now that I understand how the branching functions, the next step was figuring out how to merge.

First thing to note is that this is how you can see the file differences in the two branches:
git diff ––name–status master..branch
Other views of the data:
git diff master..branch
git log master..branch
git shortlog master..branch

So that covers what the differences are between the branches before merging. Then there’s the real merging.
git merge branch ––no–commit

Then you can look at the changes with:
git status
git diff

If you see any problems you can turn back the clock with this:
git reset ––hard

When branching in git it is important to understand the meaning of branches. Branching needs to have state, so if any files are not commited, switching contexts between branches does not make sense. If you want to switch branches you should first commit the changes you’ve made to the branch OR stash your changes with the git stash command.

I’m still getting used to git, and what I was doing was modifying the same file in two different branches. I was constantly switching contexts through git checkout and everything was getting very confusing.

So just make sure that when switching contexts between branches that you either commit your changes to the branch you’re switching from or stashing away those changes.

There are easier ways to do this, but if you cannot use git-svn for whatever reason you can follow these directions.

1. Start at the trunk level of your svn checkout and initialize git:
   git init
2. Then exclude the subversion folders, by adding the following line to “.git/info/exclude”:
   .svn
*.pyc

3. Commit all the code:
   git add .
git commit -m "Initial import of trunk"
4. Create branch and move into branch:
   git checkout -b svn_branch
5. Now merge the subversion changes into this git branch (check out this short post). So what’s actually happening is since your in the git “svn_branch” checkout it will add all your merge changes to that git branch.
   svn merge -r <branch copy rev>:HEAD http://<repo_url>/branches/<name of branch>
6. At this point check to see if subversion did any merges. Add and commit the changes to the git branch:
   git add .
git commit -a -m "Import of svn branch"

I was trying to figure out what had changed since I branched. You can diff changes between versions, but you cannot run status on the difference between changes.

A simple workaround to get something that works like status is to run diff and then just print out those lines that start with “Index”. You need two things to do this:

1) The branch revision number: You need the revision number of the revision that created the branch. You can find this by using “svn log –stop-on-copy” from within the branch.
2) The URL of the branch. If you run “svn info” on the branch you can get this address.

svn diff -r <branch num>:HEAD http://<repo_url>/branches/<name of branch> | grep "^Index"

By the way, the most natural way to do this within subversion, is to first merge and then run status. Run the following code from within trunk:
svn merge -r <branch num>:HEAD http://<repo_url>/branches/<name of branch>"
svn st

Which is a bit much if you don’t want to merge yet.

I’ll preface this post by mentioning that the steps for branching are already out there on the web. I’m more concerned here with the complexities in setting up a branch in SVN, and then move onto showing how to do the same thing in git. Branching discussed on other sites:
- Mosuma.com
- svn book
- stackoverflow “How do I create a branch in SVN”

The actual command looks as such:
svn copy http://mysite.com/trunk http://mysite.com/branches/wonderful_branch

As you’ll notice if you look at the subversion command above, the action of branching is not really a staple of subversion. Essentially you’re taking the codebase as of a certain version, copying the entire trunk and placing it in a new home to work with. This process can be quite aggravating, and unless necessary I really try avoiding it.

At a TechTalk on git, Linus Torvalds actually talks at length about this: http://www.youtube.com/watch?v=4XpnKHJAok8.

If you do brave the current in SVN, just keep in mind that the longer you’re branch is disconnected from trunk, the more merging fun you will have to deal with.

Another thing I’d recommend is removing any branches as soon as you work the changes back into trunk. There’s really no reason for branches to just sit around.

As a tool though, the only time I’d really use a branch in SVN world, is when I’m touching lots of different parts of a system and I just need further testing before bringing my changes to trunk. For the casual change, I’d rather just not deal with that. A better idea is to setup a local repo of some sort and just work off that.

Git

I was curious to see what git was all about. Being an avid Linux user, I can’t say that Linus has led me astray in the past… Also, to see if the branching really did make more sense in that world. I must say that it’s unbelievable. And as Linus said, git really does get you to interact differently with your repository. Because branching is done so well and you switch contexts so easily, I can see myself making branches A LOT more often. How it works is you create a branch and that branch kind of happens in the background. To actually view the branch you have to switch contexts, by calling the git checkout command. Once in that context there’s no drilling down to different branch directories. When you checkout the branch, you’re already in the branch and any changes stay in the branch. Then merging and deleting the branch are also very easy to do. It’s really quite impressive in a, yes that’s how it should be kind of way.

Right now, with subversion, I resist branching unless absolutely necessary. In SVN, it costs a lot (space) to branch, it’s very slow, it’s a pain to setup, and it’s tough to merge back.

And here is what it looks like in git (main tutorial)

Initializing a project (run these command within directory of project)
$ git init
$ git add .
$ git commit -m "Initial test"

Editing a file
$ vi modify_some_file
$ git add modify_some_file
$ git commit -m "Test"

Ok, now back to the topic of merging.
Create the branch and check it out. (‘git branch’ shows you which branch is the active branch)
$ git branch
$ git branch new_branch
$ git branch
$ git checkout new_branch
$ git branch

Edit a file
$ vi perl.txt
$ git commit -a -m "Expermintal test"

Change context to main branch (master)
$ git checkout master
$ less perl.txt

Merge new branch
$ git merge new_branch
$ less perl.txt

Delete the branch
$ git branch -d new_branch

Was looking into auto-generated documentation for my python code and came across the following projects: pydoc, epydoc, and sphinx. Pydoc was pretty limited and didn’t output the most useful data. Epydoc is quite easy to use and does a great job at showing all your classes, functions, and variables. And then there was Sphinx. Sphinx is really the end-all and be-all documentation tool. It does a lot and lets you do a lot, BUT you have to put a little time into it. So if the documentation is just for your own purposes, it might not be worth it. If the documentation’s for an open source project, it’s probably worth the fuss, because it’s really not that bad and makes some really good looking docs. My one further point on this, is that the other tools are very limited in scope compared to Sphinx. You use Sphinx as a tool to write documentation outside the documentation in your code. The other tools pretty much strip the doc-strings from your code and make them pretty. Definitely worth looking into Sphinx for any serious open source project.

As an aside, I always wondered what Django used to make their documentation, and now it seems it was Sphinx all along. http://docs.djangoproject.com/en/dev/internals/documentation/

Epydoc: http://epydoc.sourceforge.net/
epydoc --html python_file_1 python_file_2 python_file_3 version -o packrat_docs

Sphinx: http://sphinx.pocoo.org/
By far the nicest of the three tools is Sphinx.
The Python documentation was done using Sphinx: http://docs.python.org/dev/
Sphinx was a little confusing to setup. First off there was a dependancy to jinja2 that really wasn’t spelled out (beyond the errors that came up during install).

So after installing ‘jinja2′ it was as simple as running
sudo easy_install -U Sphinx

pydoc: http://docs.python.org/library/pydoc.html
Command looks something like this:
pydoc -w python_file-1
Seems to want the data to fit a certain setup.