GSoC: week 5: SymPy now runs in Python 3!

As the title says, you can now run SymPy in Python 3! No, it's not perfect, there are test failures (or rather, exceptions, some 200 of them), but ~90% tests pass and that's getting somewhere. I haven't tried actually using it, but at least for some definitions of "works", SymPy definitely works in Python 3. I'll document some of the issues I solved below, and I'll also update the porting tips and tricks page when I get the chance.

The cmp issue

While the new unicode/string features are probably the major Python 3 incompatibility for most programs, SymPy's foremost issue was the use of cmp(), which has been discontinued in Python 3 in favor of __lt__ and other rich comparisons. SymPy uses cmp() in various areas of code (sympy.core.Basic for one, and almost everything derives from that). I've tried to approach this from various angles, but most required a deeper understanding of SymPy than I currently posses. In the end, I opted for the most ingenious solution of all - sidestepping the issue. To see how I did this, first lets see the two different ways the removal of cmp() affects us:

a) builtin.sorted() and list.sort() no longer accept the cmp argument providing a comparison function. Use the key argument instead.

Now, this is a good thing! The cmp() function is slow in this case, as it will compare every two elements (or so). In comparison, the key function takes some defined "key" value and sorts based on; this means it's called exactly once on each object. However, if you rely deeply on customized __cmp__ functions, it can be difficult to simply use a key instead of cmp. The following recipe, which provides a cmp_to_key function, allows us to avoid this by doing just what the name says. It's not an elegant solution, but it allows us to move on with porting at least. The recipe has been included in the functools module in Python's 2.7 and 3.2, but we implement it in core.compatibility as we need to support more Python versions.

b) The cmp() function should be treated as gone, and the __cmp__() special method is no longer supported.

This is the root cause of the above point. Again, SymPy uses this quite extensively and fixing everything is beyond me at the moment. However, it is possible to naively define cmp as (a>b) - (a<b) if the builtin is not available, and I did exactly that in core.compatibility. That this works means that we aren't doing anything too complex with out __cmp__ implementations (it does fail in one case, though), which means that porting is possible.


Neither of the above changes is efficient or very nice at all (in fact, I described them as "evil" in a commit message). Still, going over these issues allows me to see the other problems that will arise and work on fixing them. It will allow me to finally have some visible progress on my project. This is important because my goal for the mid-term evaluation is to have a working SymPy in Python 3. Once SymPy is working completely, I will go back and implement a "correct" solution for this.

The other issues I solved are comparatively minor: I fixed some warnings produced by "-3" (the backquote is no longer a shortcut to repr(), tuple parameter unpacking has been removed) and some of the exceptions raised when running with Python3 (file() has been removed and we were encoding to ascii needlessly. I plan to continue working on these exceptions; as a lot of the tests all fail at the same points, I except to rapidly have almost all tests running in Python 3 (if not passing). I've also added a "FIXME-py3k" comment in a couple of files where I identified the issue but couldn't solve it; these are all things to discuss with my mentor. If someone wants to test it out, I've written some instructions on the relevant issue in our bug tracker. Merging will probably wait some time, but I will push for it to happen as soon as possible.


Also during this week, I worked on setting up our Jenkins server, you can see it here. Anonymous users can see just the build history; developers should contact me if they want an account set up. It is currently down, however.This was mostly a test run - we decided to reinstall our server from an ancient Debian to a current Ubuntu and only then run the "live" Jenkins server. This will happen when Ondřej, who runs the server we use, has more time (around the middle of July). I don't expect further issues to arise.

I also have a test branch with a port to distribute (instead of the currently distutils we use). We need to use Distribute to support automatic running of 2to3 when installing. The porting itself was trivial, but I need to read a lot of documentation before I figured out what the differences are (I even asked a question on StackOverflow about it). To say the documentation is confusing would be an understatement. The major problem is how to solve the mpmath issue.

As I wrote in my week 3 report, mpmath is already Py3 compatible and 2to3 shouldn't be run on it. I've since checked it out and I don't see an easy method to skip the mpmath directory. It might be possible, but I don't see an easy way without going around Distribute entirely (which is, again, not the point). I even asked about in on the distutils-SIG mailing list, the answer is a "don't do it, unbudle instead". While I've been in the unbudle camp since the start, the discussion hasn't been so one-sided (I won't summarize it here, you can read the linked issue). In the end, though, if I don't find a way to support it in Python 3, we will have to unbundle (Ronan has a branch doing just this, and it is fairly painless).

Setting up and using Tox in SymPy

In my previous post on the subject I mentioned we decided on using Tox. In the weeks since, we've established a workflow and successfully used Tox to find various bugs (see my post from last week for a particularly interesting one). As I've found the Tox documentation confusing in places, I've decided to write a "how-to" for setting up Tox and then talk some more about how we use it.

Setting up Tox

A simple "pip install tox" will handle the installation. Then you'll want to create a tox.ini file where your setup.py is located. The .ini can be very simple, just listing the wanted environments and the commands to run in each.

[tox]
envlist = py25, py26, py27
[testenv]
commands=python bin/test []
         python bin/doctest []

Here we are just using the default environments (py24-py32, Jython, PyPy) and then running our test suite in each. The square brackets are important - our bin/test is programmed in such a way to accept a testname as an argument, it will then run just the specified test(s). The brackets allow us to replicate this behaviour with Tox, instead of running the whole test suite every time. However, Tox makes it very easy to define a custom environment. One such that we could use is:

[testenv:py27-32-gmpy]
basepython=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
deps = http://gmpy.googlecode.com/files/gmpy-1.14.zip
commands = python bin/test []

SymPy can optionally use gmpy to provide better ground types. This test environment tests it with a 32-bit version of Python 2.7; the "basepython = " allows us to specify the path to a specific python interpreter to use. The "deps" command specifies dependencies (multiple dependencies should be in multiple lines, like for "commands" above). It can download them automatically using pip, or you can specify a zip/tarball like above. You can also see the full Tox specification, but these are the most important commands and the ones most likely to be used. You can also see the tox.ini.sample file we use, it has some more examples. (Note: this is how configuration files should be handled in DCVS': a "canonical" .sample file, which should then be copied to .ini and modified to needs. The .ini file itself should be ignored by your DCVS)

Using Tox

Tox works by creating a virtualenv for each environment specified, by default in a .tox directory, so there is no way it can mess up your system. It will reuse the existing environments, but it's possible to force a rebuild with "tox --recreate". You normally run Tox with a simple "tox", which runs all the environments listed in "envlist". You can optionally specify particular environments with -e (so, "tox -e py25,py26" will run just py25 and 26). A neat feature here is that you don't need to have an environment in your default envlist to run it with "-e".

Now obviously, the basic usage is a simple "tox". That will build all the default environments and run the whole test suite in each. Unfortunately, our test suite takes around 10 mins to run on average hardware and we currently support 3 Python versions (to go up to 4 or 5 when we finally get Python 3 support). If we add in {gmpy / no-gmpy} and even {32bit, 64bit}, we quickly get to an unrealistic amount of combinations to run. Yes, it is important to run these tests occasionally, but it isn't realistic to expect a developer to run all of them for every change they make (that will be the job of our CI server, after all). Still, it's important to test at least some basic testing for every change. As such, I advocate running py25-py27 by default, every time. In my opinion, this is a good compromise between time and testing coverage. If a bug occurs in some version, one can then research it further by running more variations (with gmpy, on 32bit etc) or preferably just fix it if possible.

Another useful feature is that our test suite supports running just specific tests (as mentioned above). Depending on the code changes, it might be enough to run just the appropriate test suite. This will drastically reduce the time required (and will allow the developer to run in all versions, rather than the reduced list above). Alternatively, if just one test fails after a change, it's easier to debug it if you can run just the specific test.

That's basically it! Tox is very simple to use and quite effective. The biggest challenge is probably persuading developers to use it, but the value is clear. It's also invaluable for testing before a release, per the words of the maintainer of SymPy, Aaron.

GSoC: week 3

This week I've worked on adapting my work into commit-worthy chunks. The results are pushed to my fork on Github, you can see them here. I've tried to make the commits fairly self explanatory - the first several are just fixing the warnings "-3" produces in a matter that's Python 2.5+ compatible. Unfortunately, Googling the error didn't produce much information in most cases; because of this, I've created a page where I've collected the solutions I used to solve each warning. You can find it on the right-hand sidebar. I've since also updated the version of mpmath bundled in SymPy to 0.17 (which supports Python 3 but drops support for Python 2.4, thus representing the first commit not compatible with Python 2.4).

Unfortunately, here I've hit a problem. Namely, mpmath is written with a single code-base that can run both Python 2 and Python 3 unmodified (this is not the recommended course of action usually, but mpmath didn't need to change much so I guess it makes sense). Normally, this is good, but SymPy will need to use 2to3 and this creates errors in SymPy. Why? One banal example is something like:

try:
    from itertools import izip
 except ImportError:
     izip = zip

Now, when 2to3 runs on this it removes the "from itertools..." line and thus we get an error. This is just one simple case, but there are plenty of others. It's obvious that running 2to3 on compatible code is not going to produce the desired results. And this is where I'm stuck currently. 2to3 doesn't support skipping some directories, so I'll probably have to develop our "internal" script that will use lib2to3 directly. To do this, I've decided to first try to integrate automatic 2to3 into our setup.py, so that I'd have an idea on how it works. [As a side note, thanks to Lennart Regebro and Benjamin Peterson, who've helped me in this thread on the Python-porting mailing list when I asked how to speed up 2to3 -- using the latest version of it was particularly good advice as it provides ~40% speedup in my case]

Once mpmath is integrated properly, I'll be able to continue on porting SymPy itself. I feel like I'm on track with the timeline in my proposal, which would be a working SymPy by the midterm evaluations.


I'd also like to take a moment here to comment on the value of Tox. While porting yesterday, I encountered an error in a polys test. As mpmath is used heavily in this code, I thought it was something to do with the new version. Later, I realized it occurs in master too and quickly bisected it down. And this is where the trouble started, as no one could reproduce the error, with any combination of Python version, architecture, ground-types and cache used. Digging down deeper, we found that the hashes of some functions really were different for everyone, though it was just me getting the error. In the end, someone else confirmed it - Python 2.6, 64-bit, with python ground types (and only on Linux, I think).

The error itself was nasty - int(1) and long(1) do not have the same hash [EDIT: See Aaron's comment below for the details] (I was told similar problems occurred before as well). As the underlying hash was hashing lists of lists (possibly deeply nested), the fix was to convert it to tuples of tuples and then hash it. The patch was written by asmeurer, and the whole discussion can be found on the issue page (and the IRC logs from yesterday). The moral of the story? Use Tox! How else would we find something that only appears in such specific circumstances.

GSoC: week 2

This has been a particularly uninteresting week of work, as I've expected. I have done some work towards making SymPy code Python 3 compatible, but there are no visible results - it just doesn't run in Python 3. As soon as it's possible to at least run SymPy in Python 3, I will begin publishing my branch. A hurdle is that SymPy should still be supporting Python 2.4 until the release of 0.7.0. Unfortunately, this release has been "imminent" for a while now. Initially, I was hoping to start integrating my work in small chunks as soon as possible. However, as this is the second week already, I've had to start some work in parallel. I only hope merging later will not be a problem.

It should also be noted that constantly re-running 2to3 is not a pleasant task. While this has some implications for testing, it's more importantly slowing down my work.