Iraniblog

Apparently documentation is important sometimes… I’m writing this Python script that’s run in both Linux and Windows. My development happens on a Linux box, so until I move onto testing my scripts on Windows, I don’t realize the discrepancies. I wanted to get the username of the user, and there is this command in the os module called getlogin() which worked perfectly on my Linux box. In Windows it raises an exception. I checked the documentation and it says that getlogin() is only supported in UNIX, go figure… The way it recommends to get the username is from the os environment variable dictionary. The documentation tells you to use the “LOGNAME” key. I’ve found that “LOGNAME” does not work in Windows, but “USERNAME” works in both…

import os
os.getenv("USERNAME")

Anyways, tread lightly and always look at the documentation, even when something seems to be named so appropriately in such a mainstream language/module. One of the things that I really love about Python is that it usually does what you expect it to do. I guess nothing’s perfect though, and it pays to always be on your toes.

I am currently writing a lot of file manipulation code and keep running into snags that’s led to a lot of interesting debugging…

The most recent issues have been with editing files with interesting permissions set. I’ve been messing with the sticky bit, which you trigger so that the files owner is the user that runs the application regardless of who actaully runs the application.

The situation I’m interested in here, is that I set a file to be owned by a user named ‘admin’ and set the sticky bit to active (so that the file would always be run by ‘admin’). Then I tried running that file, and having that script manipulate a file for which it needed to be the user ‘admin’, and that failed, since the file was actually not being run by the ‘admin’ user.

After a bunch of debugging I realized that although the script is set to the ‘admin’ user, the actual thing running the application is the Python interpreter, which is being run by my logged in user (not the ‘admin’ user’).

The following code snippets will take you through the scenario so that you can reproduce and feel the silly complexity of Linux file manipulation on your own.

Let’s say our file is named ‘sticky’ and we are just going to work in our current directory:

> ls -ld sticky
-rw-rw-rw-   1   my_user   group   1K 2009-07-24 12:00 sticky

> sudo chown admin:group sticky
> sudo chmod 750 sticky
> sudo chmod ug+s sticky

> ls -ld sticky
-rwsr-s---   1   admin   group   1K 2009-07-24 12:00 sticky

> ./my_python_script.py

Right there is where my problem happened. I couldn’t run any of the file manipulation stuff on the ‘sticky’ file, even though my script had its sticky bits set and its owner was the same as that of the file I wanted to manipulate. One way around this is to write a wrapper in C++ that then calls your Python script. As such:

my_cpp.cpp:

#include <sstream>
#include <cstdlib>
#include <iostream>
#include <errno.h>
using namespace std;

int main(int argc, char **argv)
{
    string cmd = "./my_python_script.py";
    int return_code = execvp(cmd.c_str(), argv);

    if (-1 == return_code)
        cout<<errno<<": "<<strerror(errno)<<endl;

    return 0;
}

my_python.py:

print "Inside Python file"

You would run your code as such:

> chmod 750 my_python.py my_cpp.cpp
> g++ my_cpp.cpp -o my_runner
> my_runner

OK, so maybe you can see a bit of a pattern here. I’ve been messing with git trying to get to the point where I can do what I want with it. Now that I understand how the branching functions, the next step was figuring out how to merge.

First thing to note is that this is how you can see the file differences in the two branches:
git diff ––name–status master..branch
Other views of the data:
git diff master..branch
git log master..branch
git shortlog master..branch

So that covers what the differences are between the branches before merging. Then there’s the real merging.
git merge branch ––no–commit

Then you can look at the changes with:
git status
git diff

If you see any problems you can turn back the clock with this:
git reset ––hard

When branching in git it is important to understand the meaning of branches. Branching needs to have state, so if any files are not commited, switching contexts between branches does not make sense. If you want to switch branches you should first commit the changes you’ve made to the branch OR stash your changes with the git stash command.

I’m still getting used to git, and what I was doing was modifying the same file in two different branches. I was constantly switching contexts through git checkout and everything was getting very confusing.

So just make sure that when switching contexts between branches that you either commit your changes to the branch you’re switching from or stashing away those changes.

There are easier ways to do this, but if you cannot use git-svn for whatever reason you can follow these directions.

1. Start at the trunk level of your svn checkout and initialize git:
   git init
2. Then exclude the subversion folders, by adding the following line to “.git/info/exclude”:
   .svn
*.pyc

3. Commit all the code:
   git add .
git commit -m "Initial import of trunk"
4. Create branch and move into branch:
   git checkout -b svn_branch
5. Now merge the subversion changes into this git branch (check out this short post). So what’s actually happening is since your in the git “svn_branch” checkout it will add all your merge changes to that git branch.
   svn merge -r <branch copy rev>:HEAD http://<repo_url>/branches/<name of branch>
6. At this point check to see if subversion did any merges. Add and commit the changes to the git branch:
   git add .
git commit -a -m "Import of svn branch"

I was trying to figure out what had changed since I branched. You can diff changes between versions, but you cannot run status on the difference between changes.

A simple workaround to get something that works like status is to run diff and then just print out those lines that start with “Index”. You need two things to do this:

1) The branch revision number: You need the revision number of the revision that created the branch. You can find this by using “svn log –stop-on-copy” from within the branch.
2) The URL of the branch. If you run “svn info” on the branch you can get this address.

svn diff -r <branch num>:HEAD http://<repo_url>/branches/<name of branch> | grep "^Index"

By the way, the most natural way to do this within subversion, is to first merge and then run status. Run the following code from within trunk:
svn merge -r <branch num>:HEAD http://<repo_url>/branches/<name of branch>"
svn st

Which is a bit much if you don’t want to merge yet.

I’ll preface this post by mentioning that the steps for branching are already out there on the web. I’m more concerned here with the complexities in setting up a branch in SVN, and then move onto showing how to do the same thing in git. Branching discussed on other sites:
- Mosuma.com
- svn book
- stackoverflow “How do I create a branch in SVN”

The actual command looks as such:
svn copy http://mysite.com/trunk http://mysite.com/branches/wonderful_branch

As you’ll notice if you look at the subversion command above, the action of branching is not really a staple of subversion. Essentially you’re taking the codebase as of a certain version, copying the entire trunk and placing it in a new home to work with. This process can be quite aggravating, and unless necessary I really try avoiding it.

At a TechTalk on git, Linus Torvalds actually talks at length about this: http://www.youtube.com/watch?v=4XpnKHJAok8.

If you do brave the current in SVN, just keep in mind that the longer you’re branch is disconnected from trunk, the more merging fun you will have to deal with.

Another thing I’d recommend is removing any branches as soon as you work the changes back into trunk. There’s really no reason for branches to just sit around.

As a tool though, the only time I’d really use a branch in SVN world, is when I’m touching lots of different parts of a system and I just need further testing before bringing my changes to trunk. For the casual change, I’d rather just not deal with that. A better idea is to setup a local repo of some sort and just work off that.

Git

I was curious to see what git was all about. Being an avid Linux user, I can’t say that Linus has led me astray in the past… Also, to see if the branching really did make more sense in that world. I must say that it’s unbelievable. And as Linus said, git really does get you to interact differently with your repository. Because branching is done so well and you switch contexts so easily, I can see myself making branches A LOT more often. How it works is you create a branch and that branch kind of happens in the background. To actually view the branch you have to switch contexts, by calling the git checkout command. Once in that context there’s no drilling down to different branch directories. When you checkout the branch, you’re already in the branch and any changes stay in the branch. Then merging and deleting the branch are also very easy to do. It’s really quite impressive in a, yes that’s how it should be kind of way.

Right now, with subversion, I resist branching unless absolutely necessary. In SVN, it costs a lot (space) to branch, it’s very slow, it’s a pain to setup, and it’s tough to merge back.

And here is what it looks like in git (main tutorial)

Initializing a project (run these command within directory of project)
$ git init
$ git add .
$ git commit -m "Initial test"

Editing a file
$ vi modify_some_file
$ git add modify_some_file
$ git commit -m "Test"

Ok, now back to the topic of merging.
Create the branch and check it out. (‘git branch’ shows you which branch is the active branch)
$ git branch
$ git branch new_branch
$ git branch
$ git checkout new_branch
$ git branch

Edit a file
$ vi perl.txt
$ git commit -a -m "Expermintal test"

Change context to main branch (master)
$ git checkout master
$ less perl.txt

Merge new branch
$ git merge new_branch
$ less perl.txt

Delete the branch
$ git branch -d new_branch

Was looking into auto-generated documentation for my python code and came across the following projects: pydoc, epydoc, and sphinx. Pydoc was pretty limited and didn’t output the most useful data. Epydoc is quite easy to use and does a great job at showing all your classes, functions, and variables. And then there was Sphinx. Sphinx is really the end-all and be-all documentation tool. It does a lot and lets you do a lot, BUT you have to put a little time into it. So if the documentation is just for your own purposes, it might not be worth it. If the documentation’s for an open source project, it’s probably worth the fuss, because it’s really not that bad and makes some really good looking docs. My one further point on this, is that the other tools are very limited in scope compared to Sphinx. You use Sphinx as a tool to write documentation outside the documentation in your code. The other tools pretty much strip the doc-strings from your code and make them pretty. Definitely worth looking into Sphinx for any serious open source project.

As an aside, I always wondered what Django used to make their documentation, and now it seems it was Sphinx all along. http://docs.djangoproject.com/en/dev/internals/documentation/

Epydoc: http://epydoc.sourceforge.net/
epydoc --html python_file_1 python_file_2 python_file_3 version -o packrat_docs

Sphinx: http://sphinx.pocoo.org/
By far the nicest of the three tools is Sphinx.
The Python documentation was done using Sphinx: http://docs.python.org/dev/
Sphinx was a little confusing to setup. First off there was a dependancy to jinja2 that really wasn’t spelled out (beyond the errors that came up during install).

So after installing ‘jinja2′ it was as simple as running
sudo easy_install -U Sphinx

pydoc: http://docs.python.org/library/pydoc.html
Command looks something like this:
pydoc -w python_file-1
Seems to want the data to fit a certain setup.

When loading up subversion’s status command it is quite annoying to get a list of files with the extension “pyc” in the list of files to look through. What you can do is setup rules within subversion to ignore these files. Subversion calls these settings “properties” and you can add them to different levels of your files (which actually can be confusing). What happened in my situation is that one of the folders I was working on did not have this value set, so that folder’s bytecode files were showing up but not any of the others.

The command is pretty simple:
svn propset svn:ignore "*.pyc" folder_name/

To view those properties that are set to a folder simply do:
svn proplist folder_name/

To delete the “svn:ignore” property do:
svn propdel svn:ignore folder_name/

Just started working on this new project and ran into a filetype — and an issue — I’ve never ran into before. The file’s extension is “qrc” and it’s doctype is “RCC”, but it looks very much like a straight up XML file. I did a little lookup and found out it’s Qt’s “resource system.” Qt is a set of libraries written in C++ that includes GUI, networking, thread, and many other utilities. The resource system lets you embed different types of files into your programs more easily. These “resources” are then “part of” your program… Instead of having to figure out a way to store an image somewhere and then make the connection to your program, the image becomes actually part of your software.

I had to take the “resources.qrc” file that was in the codebase and convert it to python (since I’m using PyQt). The command to convert “resources.qrc” to “resources.py” is as simple as:

pyrcc4 -o resources.py resources.qrc

PyQT 4 Ref