Using gencodec to make a custom character mapping

One of the problems I face in the QTI migration tool is markup that looks like this:

<mattext>The circumference of a circle diameter 1 is given by the mathematical constant: </mattext>
<mattext charset="greek">p</mattext>

In XML the charset used in a document is detected according to various rules, starting from information available before the XML stream is parsed and culminating in the encoding declaration in the XML declaration at the top of the file:

<?xml version = "1.0" encoding = "UTF-8">

For this reason, the use of the charset parameter in QTI version 1 is of limited value, at best it might provide a hint on an appropriate font to use when rendering the element.  This is not a huge problem these days but when QTI v1 was written it was common for document renderings to be peppered with large squares indicating that the selected font had no glyph for the required character.  These days renderers are smarter about selecting default fonts enabling developers to display arbitrary unicode text.

So you would think that charset is redundant but there is one situation where we do need to take note: the symbol font. The problem is explained well in this article: Symbol font – Unicode alternatives for Greek and special characters in HTML.  The use of 'greek' in the QTI v1 examples is clearly intended to indicate use of the symbol font in a similar way - not the use of the 'greek' codepage in ISO-8859. The Symbol font is used a lot in older mathematical questions, you can play around with the codec on this neat little web page: Symbol font to Unicode converter.

According to the above article the unicode character representing the lower-case letter 'p', when rendered in the symbol font actually appears to the user like this: π - known as Greek small letter pi.

The problem for my Python script is that I need to map these characters to the target unicode forms before writing them out to the QTI version 2 file.   This is where the neat gencodec.py script comes in.  I don't know where this is documented other than in the gencodec source file itself.  But this is a very useful utility!

The synopsis of the tool is:

This script parses Unicode mapping files as available from the Unicode
site (ftp://ftp.unicode.org/Public/MAPPINGS/) and creates Python codecmodules from them.

So I downloaded the following mapping to a directory called 'codecs' on my laptop:


Then I ran the gencodec script:

$ python gencodec.py codecs pyslet
converting SYMBOL.TXT to pysletsymbol.py and pysletsymbol.mapping

And confirmed that the mapping was working using the interpreter:

$ python
Python 2.7.1 (r271:86882M, Nov 30 2010, 09:39:13) 
[GCC 4.0.1 (Apple Inc. build 5494)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> unicode('p','symbol')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: unknown encoding: symbol
>>> import pysletsymbol
>>> reg=pysletsymbol.getregentry()
>>> import codecs
>>> def SymbolSearch(name):
...   if name=='symbol': return reg;
...   else: return None
>>> codecs.register(SymbolSearch)
>>> unicode('p','symbol')
>>> print unicode('p','symbol')

In previous versions of the migration tool I didn't include symbol font mapping because I thought it would be too laborious to create the mapping.  I was wrong, future versions will do this mapping automatically.


In Memoriam Claude Ostyn: Account Suspended

It is strangely saddening to have to mourn a professional colleague twice but today I noticed for the first time that one of my 'go to' sites for information on SCORM has finally also been laid to rest (this is perhaps old news).

Fortunately much of the information is preserved by archive.org but the strength of his voice seems strangely diminished there:


Open Source Routers to the Rescue!

I spent a few happy hours this weekend fiddling with routers and stumbled across a great open source project called dd-wrt.  This one is not for the timid as there is a serious risk of bricking your router but if you are anything like me you have an old box gradually filling up with discarded routers you are keeping just in case you main router fails.

I recently switched tariffs with my broadband supplier, Virgin Media.  As a result, they replaced my cable modem with an all-in-one wireless-router/cable modem.  I can see that this simplifies their business for most customers but when you already had a good router before it is a pain when you realise that you are now tied to something with poorer range and and annoying tendency to make your printer panic.

Anyway, as a result I went and bought an ASUS device (RT-N12) capable of being used as a wireless repeater to improve coverage in those hard-to-reach corners of my small 3-bed semi.  Unfortunately, try as I might I could not get it to work as advertised.  At one point I actually got it to repeat the wifi signal from the Virgin router but the LAN ports were all on their own private network.  My attempts to fix the LAN ports broke wifi connectivity for everyone in the house and almost bricked the device.  I updated to the latest ASUS firmware but still no joy - I was beginning to get bored of the recovery mode procedure for restoring factory defaults.  There has to be a better way, surely?

Part of the problem is that the ASUS device has attempted to simplify the user interface of the router to an extreme making it very hard to fix issues.  The quality of the translation in the UI is very poor and the little help icon almost always brings up a little box with placeholder text explaining what the help function does, rather than actually providing any useful insight into the settings.  The one saving grace is that, with so few settings to control, one can quickly exhaust all possible combinations and conclude that home-networking is not for mere mortals.

But, masochists take note: there is even more fun you can have with these devices and you don't need to spend a penny more (the RT-N12 is available from £28 at 52 stores according to Google).

The software distributed by dd-wrt is a replacement firmware that actually works. Don't be scared off by the use of 'beta' or the frightening stories of bricked devices. I'm sure there is a reason why you are supposed to use the special ASUS firmware updating tool on Windows but I just flashed the firmware through Safari from my Mac and within a few minutes I was looking at a whole new user interface. This software is a breath of fresh air. It may be free but there is more documentation than the supplied firmware and being German the quality of the English is better than I could write myself.

The ASUS device clearly has software shortcomings but it isn't clear when linkingWifi devices which party is to blame. The market always seems to be getting ahead of the standards and even with the faster 802.11n mode it seems that devices can't agree which frequency to talk on and that in practice most networks are operating at b/g speeds for legacy reasons.

Routers like the Virgin hub are understandably configured for compatibility so most people seem to recommend turning off the wireless completely and then using one of the four LAN ports to plug in a completely separate wireless infrastructure to split new 'n' devices from the old b/g - now where is that box I keep all my old routers in...