sphinx: ignoring sys.path?

I'm a big fan of using Sphinx for documentation and I use it for my Pyslet Python module. For some time now I've had an infuriating problem with generating documentation that I've been putting off solving on the assumption that it is caused by some weird and hard to discover configuration issue.

The problem is that whenever I run sphinx to build my package documentation it uses the currently installed version of the package and not the local copy I'm working on. This is particularly annoying because I have various versions of Python installed on my Mac (going back some years) and sphinx always seems to run with the native Python that came with XCode (Python 2.7.10 at the time of writing) whereas typing "python" at the terminal uses one of my older custom builds. I guess I'd always thought there was something weird going on and just relied on the workaround which was to run:

$/usr/bin/python setup.py install

...every time I wanted to rebuild the docs. That forced the latest version of the package to be installed into the Python interpreter that was going to be run by sphinx.

I had already edited my sphinx conf.py file as follows:

sys.path.insert(0, os.path.abspath('..'))

That should have been enough to put the correct path to the working copy in the search order before anything that was installed in the site packages but, alas, it just didn't seem to work. I checked that sys.path was correct, I even started the interpreter on the command line to ensure the working copies were getting loaded after this modification. They were, it seemed inexplicable!

I finally solved the mystery today, and thought I'd blog the answer in case anyone else does the same stupid thing I did.

Earlier today I added a new module to my package and running the docs gave me an import error. Previously I'd thought that some strange site-specific import hook was causing sys.path to be ignored but an ImportError suggested that sys.path was being completely ignored. I got so frustrated that I edited the autodoc.py package to check that sys.path was set correctly just before it calls __import__ and then I set a breakpoint to see if I could see what was happening. I tried setting sys.path to a single directory, the one with the working version of my package in it. Still the import brought up the version installed in site packages. How is this possible?

When you import a module with a name like "package.modA" you're actually (sort of) importing the package first and not the module. As a result, if the package has already imported and you execute import package.modB Python will ignore sys.path because it already knows where the package is. This was exactly what was going wrong for me...

The Solution

My package contains a trivial module called 'info' which contains a few strings that I use to describe the package both within the package itself and also in setup.py. At some point I must have gone in to the conf.py I use for Sphinx and optimised away some of the redundant text by importing the strings directly from my package...

#
# conf.py
# 
import pyslet.info as info

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('..'))

# ... [snip]

# General information about the project.
project = info.title_name
copyright = info.copyright

Notice that conf.py imports my 'info' module before I change sys.path, as a result it gets imported from site packages and every future import from my pyslet package will come from site packages too. The solution is simple, and sanity has been restored:

#
# conf.py
# 

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('..'))
import pyslet.info as info

# ... [snip]

# General information about the project.
project = info.title_name
copyright = info.copyright

And now my documentation builds correctly from the working copy. I cannot believe that I'm the first person to trip over this issue. A bit of Googling (after the fact) reveals this module, for example: CodeChat conf.py which uses a similar technique to mine.

Happy New Year!