Snow Leopard, wxPython and py2app

As I write this blog post I'm happy to say that I have finally managed to get a new build of the QTIMigration tool on OSX, however this post is not about the migration tool so much as the process of getting the binaries built on a Snow Leopard-based machine.


The QTIMigration app runs in either GUI or command-line mode.  The GUI is based on wxPython which does not run well on 64-bit python builds.  The GUI part was written by Pierre and hasn't been changed in 3 years, in 2008 we had no trouble using py2app to bundle up a binary distribution.


In Snow Leopard, the default python interpreter runs in 64 bit mode.  It takes a bit of fiddling but it is relatively straightforward to check out the migration tool source and run it from the terminal forcing the interpreter into 32bit mode to satisfy wxPython.  I found this stackoverflow thread helpful in understanding the issue and ended up with a little script like this on my path:


#! /bin/bash
arch -i386 /usr/bin/python2.6 "$@"

I called the above script python32 (which seems dumb now that python3.2 is out) and it works well enough.

So to build the binary distribution, in theory, all I need to do is run py2app from the command line...

python32 setup.py py2app

Unfortunately, the resulting app fails when run with an unusual error about a missing attribute:

AttributeError: 'module' object has no attribute 'TickCount'

The error can be found on the system console.  As ever, someone has experienced the problem before (for example, see this post) but the real solution lies in the sage advice that the best way to run py2app is to use a standard python distribution from python.org and to ignore the one that came with the original OSX.  Furthermore, if you want to create applications that will run in 32bit mode you need to install the 32bit architecture version of python.

So I downloaded python-2.7.1-macosx10.3.dmg and installed it.  Fortunately there is no python2.7 on Snow Leopard so there is no problematic name clash to resolve.  I also installed setuptools by downloading setuptools-0.6c11-py2.7.egg (note that this uses whatever python python2.7 points to).  Then I installed wxPython from wxPython2.8-osx-unicode-2.8.12.0-universal-py2.7.dmg.  There doesn't seem to be a way of forcing the wxPython install to use a particular python but again, it seemed to find its way into my new python2.7 install without difficulty.  At last, I was ready to install the other modules needed by the migration script, including py2app.

To avoid confusion I extracted the tars manually and ran each of the add-in modules setup.py script using:

python2.7 setup.py install

This step included installing the new pyslet package I'm working on and its dependencies.

Finally, I was able to re-run the py2app package step using my new python environment:

python2.7 setup.py py2app

The resulting binary worked!  Although I no longer have an older Mac to properly test the compatibility of the new binary I could at least test it worked on a machine that hasn't had the custom python build applied.

I guess all this extra complexity has at least helped to test out the code a bit more thoroughly.  I was pleased that the unit tests for pyslet all ran fine on python2.7.  Unfortunately the migration tool itself has a bug when handling non-ascii file names.  This is because my python2.7 environment is now capable of using unicode strings for file names but at one point in the migration code I'm using the old urllib.pathname2url which chokes on my chinese examples (as they have chinese filenames).  I believe this behaviour is different from the built-in python on Snow Leopard, but either way there is no easy fix and it looks like I'll have to wrap or replace my use of this function before I can post the new OSX binaries.

Watch this space, I feel like I'm getting close to a new binary distribution now.

International alarm rings over UK ICT policy

This is an interesting article drawn to my attention by SALTIS. The position taken by ISO/IEC and its advisors seems at odds with the benefits brought about by open standards from bodies such as the IETF (think 'internet' and 'world-wide-web').

International alarm rings over UK ICT policy - Public Sector IT: "promise"

Of course, failure to deliver on the open standards strategy is likely to reduce competition and drive up costs, but perhaps the strangest part of this is the monopoly governments give to the national standards bodies in the first place.  Perhaps simply opening up the world of standards itself to competition will be enough to bring about the changes required.

There are parallels with academic publishing here as it is not the authors getting rich from the charges levied on the purchase of standards documents: I love this article from the ACM in 1995: Standards, Free or Sold?.  ISO now lumps publications income with 'other services' in their financial statements and there is still no easy why of looking at the publication costs,  but clearly these could be reduced to a minimum by adopting royalty free re-distribution policies.   Either way, most of the funding required to develop the standard is donated by the sponsoring organizations or is supported indirectly by government programmes to help industry.  ISO's FAQ points out that publications income actually helps balance these larger interests but this argument seems obscure.

In my personal opinion, even if the UK government fails to compel standards to be made available openly this time around (as the EU did when it caved in over royalty free distribution earlier this year) the desired changes can still be brought about by encouraging the use of standards from de facto bodies like W3C, IETF  - after all, you are probably reading this article via http!

Standards: so many to choose from!

It's an old joke that gets trotted out at most standards meetings but the truth is that most bits of technology are the result of hundreds, if not thousands of supporting components and systems.  Just image how many technical standards must apply to something like the average family car once you've taken into consideration all the components.

The world of software is no less complex.  I just had a look at the UK government's survey on use of open standards: the number of standards is not only bewildering but when I got to the section on education I realized that it is clearly still only the tip of the iceberg.  And I didn't spot HR-XML, SIF or PESC to name just a few.

Those with an interest in UK public sector procurement might want to hop along to the survey website

My First Steps into the Iron Age

When PyAssess was originally being developed we did some experiments on getting it running in Jython.  Jython is an alternative implementation of the Python interpreter which runs inside a Java virtual machine.  Unfortunately, we'd relied fairly heavily on being able to distinguish regular ASCII strings and Unicode strings and this was not supported in Jython at the time.  I'm sure it has moved on since then but I haven't had a second go - and anyway, for Python 3 I'll need to sort the string/unicode issue out anyway.

C# programmers work in a similar environment to Java.  (As an aside, the sheer cost to the industry of Microsoft and Sun's failure to reach an agreement in those early days must be staggering.)  Not surprisingly, there is a C# equivalent to Jython, the project is called IronPython and I feel that it is about where Jython was when I was involved in my previous PyAssess experiments around 2003.

With my expectations set realistically I set about taking the first steps towards getting my latest python code running in the .Net environment using IronPython.

Installing IronPython and the associated toolset for Visual Studio 2010 went well and there was a useful walkthrough document to help me get started.  However, much of the documentation seems aimed at introducing experienced Windows developers to Python whereas I could have really done with something the other way around.  My first problem was that I'd installed Visual Studio 2010 with some type of Product Management profile and step 1 of the walkthrough involved selecting a menu option I didn't even have!  I couldn't figure out how to automatically reconfigure the menus in Visual Studio (even rerunning the installer) so had to go hunting for the "New Project..." menu item and add it to the File menu manually.  Still, when in Rome...

My simple "Hello World!" script went without a hitch but I ran into the following issue almost immediately: http://ironpython.codeplex.com/workitem/29077  - I ended up writing the following code which has to be used as a prefix to the first loaded python module in the project (and assumes you've installed your IronPython in the default location).

try:
    import string
except ImportError:
    import sys
    IRONPYTHON_PATH_FIX=['.', 'C:\\Windows\\system32', 
        'C:\\Program Files\\IronPython 2.7\\Lib',
        'C:\\Program Files\\IronPython 2.7\\DLLs',
        'C:\\Program Files\\IronPython 2.7',
        'C:\\Program Files\\IronPython 2.7\\lib\\site-packages']
    sys.path=sys.path+IRONPYTHON_PATH_FIX
    import string

As usual, the command window in Windows seems to dissappear before you've had a chance to read the output of your program but I did eventually get the following script working (with the above header of course):

import string, time
print string.join(['Hello','World!'],' ')
time.sleep(10)

In the traditional spirit of starting to run as soon as I'd taught myself to walk I checked out the latest python package code (pyslet) from the QTI migration project and installed it.  I was intrigued that IronPython has byte-compiling disabled but this doesn't seem to prevent the install from completing.

My next task was to check out the unittests and run them against the installed module.  At this point I tripped over my laces and fell flat in the mud: setuptools is not supported on IronPython and, therefore, the pkg_resoures module I use to check dependencies in the unittests is not available.

It is probably too much to expect a complex module like setuptools to work at this stage, I feel somewhat chastened by the realization that it isn't part of the main python distribution yet anyway!  This two-year old blog post suggests that problems getting zlib working are holding it back but the good news is that zlib is reported to be fixed in the latest release of IronPython (2.7) -- this was only released a couple of weeks ago and is one of the reasons why I'm looking at this environment now.

So although progress halted, I think I can work around the lack of pkg_resources.  I now plan to add exception handling to prevent it aborting the tests and have another go, at which point I'll post an update on progress to this blog.

OAuth, Python and Basic LTI

On a recent long flight I was working on a Python script to act as a bridge between an IMS Basic LTI consumer and Questionmark Perception motivated by a rash claim that this was achievable given a suitably long flight away from other distractions.

The first part of the job (undertaken at Heathrow's Terminal 3) was to download the tools I would need.  The moodle on my laptop was still on 1.9.4 so I needed to upgrade before I could install the Basic LTI module for Moodle 1.9 and 2.  Despite the size of the downloads the 3G reception is great at Heathrow.

Basic LTI uses OAuth to establish trust between the Tool Consumer (Moodle in my case) and the Tool Provider (my script) so I needed to get a library to jump start support for OAuth 1.0 in Python.  Consensus on the web seems to be that the best modules are available from the Google Code project called, simply, 'oauth'.  The python module listed there is straightforward to use, even without a copy of the OAuth specification to hand.

Of course, these things never go quite as smoothly as you would like (and I'm not just talking about turbulence over Northern Canada).  I put together my BLTI module and hooked it up to Moodle but there were two critical problems to solve before I could make it work.

Firstly, BLTI uses tokenless authentication and the Python module has no method for verifying the validity of a tokenless request.  As a result, I had to dive in a bit deeper than I'd hoped.  Instead of calling the intended method: oauth_server.verify_request(oauth_request) I'm having to unpick that method and make a low-level call instead: oauth_server._check_signature(oauth_request, consumer, None) - the leading underscore is a hint that I might get into trouble with future updates to the oauth module.

Once I'd overcome that problem, I was disappointed to find that my tool provider still failed with a checksum validation error.  The tool consumer in Moodle was signing a request in a way that my module was unable to reproduce.  The BLTI launch call can take quite a few extra parameters and all of these variables need to put into the hash.  It's not quite a needle in a haystack but I looked nervously at my remaining battery power and wondered if I'd find the culprit in time.

The problem turns out to be a small bug in the server example distributed with the python oauth module.  The problem relates to the way the URL has to be incorporated into the hash.  (Section 9.1.2 of the OAuth spec)  The example server assumes that the path used by the HTTP client will be the full URL.  In other words, they assume an HTTP request like this:

POST http://tool.example.com/bltiprovider/lms.example.com HTTP/1.1
Host: tool.example.com
....other headers follow

In the example code, the oauth request is constructed by a sub-class of BaseHTTPRequestHandler like this:

oauth_request = oauth.OAuthRequest.from_request(self.command,
  self.path, headers=self.headers, query_string=postdata)

When I was testing with Moodle and Chrome my request was looking more like this:

POST /bltiprovider/lms.example.com HTTP/1.1
Host: tool.example.com

This resulted in a URL of "///bltiprovider/lms.example.com" being added to the hash.  Once the problem is identified it is fairly straight forward to use the urlparse module to identify the shorter form of request and recombine the host header and scheme to make the canonical URL.  I guess a real application is unlikely to use BaseHTTPRequestHandler so this probably isn't a big deal but I thought I'd blog the issue anyway because I was pleased that I found and fixed it before I had to sleep my MacBook.

Debugging unsortable problems in Python

Working in Python 2.6.1 on my Mac I noticed the following behaviour recently while debugging the QTI migration code:

>>> 'z'<('a','b')
True
>>> ('a','b')<u'a'
True
>>> u'a'<'z'
True

These three comparisons, between a string, a tuple and a unicode string demonstrate that it is easily possible to create an unsortable list of objects out of basic immutable objects such as might be used as keys in a dictionary.

This might look a bit esoteric but I'm only writing this blog post because I caught a bug which was caused by the incorrect assumption that lists of strings, tuples and unicode strings sort predictably.  I was representing XML attribute names using tuples if an attribute had a defined namespace.  The names were then used as keys into a dictionary.  Note that both 'a' and u'a' can be used interchangeably in Python 2.6 when looking up an entry in a dictionary so it was easy to go one step further and grab the list of keys, sort them and assume that the result would be predictable.  Not so.

The order of the keys returned by the key() method of a dictionary is not defined and the sort method will return different results depending on the initial order of the resulting list.

It took me a while to find someone else struggling with a similar problem but I took great solace in Incomparable Abominations.   This blog post deals with changes from Python version 1 to version 2.

I believe that Python 3 is doing two things to address the problem I'm having.  Firstly, the sloppy lack of distinction between strings and unicode strings is being cleaned up.  The transition will be painful (and mean more work getting the QTI migration tool working on Python 3-based systems) but it will prevent the type of comparison loop above.  Comparisons are also being tightened to prevent different types comparing unpredictably, a (unicode) string and a tuple will not be comparable in future meaning I catch bugs like this one earlier.

So a better future awaits, but why do the comparisons give the results they do in Python 2?  The answer is almost poetic.  Objects of different types usually sort by their class name, the comparison of a string and a unicode string is the exception because, provided the string is 7-bit clean, it is assumed to be ascii and compared as a string of characters.  We can reveal the class names using the interpreter:

>>> 'z'.__class__.__name__
'str'
>>> ('a','b').__class__.__name__
'tuple'
>>> u'a'.__class__.__name__
'unicode'

As you can see, the type names start with the alphabetic sequence 's','t','u'.

Semantic Markup in HTML

A few days ago I spotted an interesting link from the BBC about the use of semantic markup.

This page got me thinking again about something I blogged about on my Questionmark blog too.  One of the problems we experienced during the development of QTI was the issue of 'presentation'.  In QTI, the presentation element refers to the structured material and interaction points that make up the question.  However, to many people the word 'presentation' means the form of markup used at the point the question is delivered.

I always found this confusion difficult.  Clearly people don't present the XML markup to candidates, so the real fear was the QTI would allow Question authors to specify things that should be left open until the method of presentation is known by the delivery system.

For some people, using a language like HTML implies that you have crossed this line.  But the BBC page on using HTML to hold semantic markup is heartening to me because I think that QTI would be better bound directly into HTML too.

HTML has been part of QTI since the early days (when you had to choose between HTML and RTF for marking up material).  With QTI version 2 we made the integration much closer.  However, XHTML was in its infancy and work to make the XHTML schema more flexible through use of XML Schema and modularisation of the data model was only just getting going.  As a result, QTI contains a clumsy profile of XHTML transformed into the QTI namespace itself.

In fact, XHTML and XML Schema have not played so well together and HTML5 takes the format in a new technical direction as far as the binding is concerned.  For QTI, this may become a block to the rapid building of innovative applications that are also standards compliant.

But bindings are much less important than information.  I always thought that QTI xml would be transformed directly into HTML for presentation by server-side scripts or, if required, by pre-processing with XSLT to make HTML-based packages.  That hasn't really happened, so I thought it might be harder than I thought.

However, I did a little research and have had no difficulty transforming the simple QTI XML examples from QTI version 2 into XHTML 5 and back again using data-* attributes to augment the basic HTML markup.  I'll post the transform I used if there is interest.  Please add a comment/reply to this post.

Steve