GSoC: Final Report

Introduction

My project, officially named just "Porting to Python 3", is actually divided into two separate (albeit mutually complimentary) parts: my first goal was to set up a testing framework, to ensure continuous testing of SymPy across different versions of Python. SymPy used to have a server that ran buildbot, but it stopped working some time ago. This was to provide a solid base for working on my main project, making the code Python 3 compatible. As this was to be done with a single code-base, I estimated (correctly) that these could lead to subtle incompatibilities between various Python versions. Time permitting, I also intended to leverage this framework and my knowledge to get SymPy compatible with PyPy, too. The original application can be seen at the SymPy Wiki, here. More details about my progress can also be found in my blog.

Testing framework

As mentioned above, SymPy used to use buildbot, so this was my first choice. I also read about Tox, a tool that is written for the sole purpose of testing Python programs under various conditions (different interpreters, presence or lack of certain dependencies), which also provides good integration with Jenkins, a well known server for continuous integration. My initial thoughts on this are in one of my first blog posts, where I had decided to use Tox and then later try to integrate it with Jenkins, to form a fully functioning CI server. While Tox was immediately useful (here's a post on setting up and using it), the integration with Jenkins proved to be more arduous than my initial tests showed. In retrospect, perhaps I should have given buildbot a more thorough look later, rather than eliminate it so early. Furthermore, while I saw Tox as a great tool, the uptake among other developers has been.. less than stellar (other than Aaron, I'm not aware of anyone using it regularly).

Fortunately, parallel to me setting up Tox/Jenkins, work was progressing on sympy-bot. The main need for continuous integration came from a desire to review all pull requests and test them for errors - while bigger companies and projects might need real CI, all of SymPy's code gets in through the GitHub pull request system, so theoretically it should be enough to just thoroughly test every pull request; sympy-bot was developed with this purpose in mind. Designed to be ran manually, it still has the basic functionality which I couldn't manage to replicate in Jenkins: run the test suite and post the results back. Work on it has also quickened somewhat in the last couple of months, and I now consider further development of sympy-bot a better idea than working more with Jenkins.

Python 3 porting

Even with the relative failure of setting up a robust testing framework, my main project was also progressing. Due to the nature of the issue, progress was somewhat sporadic and didn't proceed at a steady pace. This was particularly apparent during the start - I was simply stumped by some of the errors I was getting and couldn't get around them; once I made a key breakthrough, I was quickly able to get SymPy importable under Python 3, though this only happened by week five. The rest of my summer was spent hunting down the remaining errors, which was interesting at first but got very tiresome by the end. In fact, at the end Mateusz had to step in and fix the remaining few failures as I simply couldn't bring myself to look at them yet again. Thanks Mateusz! [Mateusz also did a lot of work on improving PyPy support, something for which I simply didn't find the time, so double thanks to Mateusz!]

One issue that arose early during the porting process was the (un)bundling of libraries with SymPy. SymPy bundled Pyglet and mpmath. Bundling the first was probably a bad idea at the start, and it was finally removed by Stefan Krastanov sometime early in the summer to unanimous approval. Unbundling mpmath was a more contentious issue, it sparked a very lively discussion on the issue tracker. I won't rephrase it here, but in the end it was decided not to unbudle it. This meant that I had to write a custom tool to handle calling 2to3: we needed to avoid calling it on the mpmath/ directories, because mpmath is already py3k compatible (and running 2to3 on such code produces bad code).

It was ultimately decided that this tool will live in bin/use2to3 and work by creating a Python 3-compatible version of the source code in a py3k-sympy/ subdirectory (originally sympy-py3k/ but that interfered too much with tab-completion!), from which SymPy could then be ran normally under Python 3. While I initially had misgivings about the script, I now think it's quite powerful. It's not the most ideal solution, but it does work and was the last missing link in seamless Python 3 support (eg. it also corrects shebangs and fixes some whitespace issues caused by 2to3).

Conclusion

Officially, my project was a success, but I really couldn't have done it without the help of other developers working on SymPy, in particular Ronan, Aaron and Mateusz. Beyond the GSoC period, I've got every intention to continue working with SymPy, as I think I've already shown with the few pull requests I've submitted since; I have also decided to take a more active role in helping with the Google Code-In project (assuming SymPy is accepted). As for my project, I intend to focus more on the infrastructure needed to support SymPy, rather than the math issues. Still, as my knowledge of math and SymPy internals increases, I'm sure I'll find other places to contribute as well.

To future GSoC students, I suggest maintaining good communication links and trying to be involved with the project as much as possible. Good communication with the core developers and general awareness of the current state of SymPy helped me a lot. While this was arguably more important for my project than others, at least Sean Vig has also expressed regret at not being more involved. The second most important bit of advice is to try and split your work into multiple pull requests and try to get them merged as fast as possible. SymPy has a very rapid pace of development, and as such it is always better to integrate sooner rather than later. This ties in to making good, atomic commits, but means more than that: your work should be clearly separated into small, logical chunks (<= 20 commits is my suggestion). A lot of the work done this summer has still to be integrated, or there were many troubles getting it finally in (eg. the physics.mechanics module). Finally, try to budget a lot of extra time in your project application - most of us are not experienced developers and cannot estimate the amount of work needed for something correctly. Plus, when some additional problems arise (and they will), it's always better to have time set aside to deal with them.

GSoC: final week

Well, I obviously haven't kept up with my blogging. Partly this is because there wasn't too much to report - I've been stuck at the few remaining bugs for a couple of weeks now and just couldn't make myself go at them for real. Still, I've now tentatively submitted a "final" pull request, which has some misc fixes and more importantly adds the "use2to3" script that will form the basis of our Python 3 support. The idea is to run the script, which will create a "sympy-py3k" directory, which is a copy of the SymPy directory structure but with 2to3 ran where required. It should then be possible to use SymPy normally under Python 3 from that folder (even installing it). Now, this script could do with some improvements (looking at "git ls-files" instead of traversing the whole dir structure for one), but most importantly it is not really well tested. I've only tested it on my computer and while it should be sound there have already been some problems reported (in the pull request). Still, the script should hopefully be robust and so, if you have the time and/or care about Python 3 support at all: please try out the script and see if it breaks. You can get it in my "porting4" branch.

Now, there are some issues remaining (Ronan is working on the LambertW issue and the other two, test_priority error and tensor doctest failures are on the issue tracker), but I still consider my project as successful. Python 3 support is a moving target anyway (all new code that gets in is another potential error, at least until people start actively testing it) and I also expect issues to crop up that just aren't covered by the test suite. For the moment, Python 3 support can be considered an experimental, dev feature and I'd like to keep it that way for a few weeks before considering a release. Again, if you are interested in Python 3 support, please try it out, start using it for your normal work and note if something unexpected comes up.

For the foreseeable future, I definitely intend to stay with SymPy. I found the hacking genuinely interesting and would like to finish my stated goals of achieving both Python 3 and PyPy compatibility. Part of my reasoning is completely selfish - I expect these skills to be quite valuable in the near future and hacking one a code base as large as SymPy is bound to teach me a few tricks. Mostly, though, I'd just like to make SymPy better - my line of thought being, if I can do the mundane infrastructure work, then that will enable the mathematicians and physicists among us to implement more cool algorithms.

I'd also like to extend my thanks to Ronan Lamy, my mentor, who was kind enough to solve a few issues for me and for generally always being around to point me in the right direction; to Aaron Meurer, the maintainer of SymPy, who always had the time for a detailed review of my code, even if our views didn't always coincide :) (mpmath, anyone?); and finally, to all the other devs in SymPy who've helped me by reviewing my code and for making such a great program in the first place!

GSoC: week 9: Almost there

I've just now seen I managed to completely miss my blog post last week. Bah! So, for the past two weeks, I've worked on two things in parallel. Lets take it from the top:

Python 3 support

This is going as well as could be hoped. The latest pull request is awaiting merging. With it, there are three exceptions and one failure in the main test suite remaining and this hasn't changed in about a week. I've asked Ronan for help on these, and he's opened an issue for one of the exceptions. It's something deep in the assumptions code; as Ronan knows his way best around this code, I hope he'll be able to solve it. I haven't looked in detail at the other problems, but hopefully they won't be too hard.

What I have been doing is working on the doctests and the failures they show - at least three real issues that aren't covered in normal tests have beeen uncovered. The others have been mostly problems with the doctests themselves - doctests rely on comparing the exact output and some minor things have changed in Python 3 (eg. is now ). Most of these fixes are in the pull request referenced above. There are 2 exceptions and 2 failures remaining to fix. One of the failures (while trivial to hack-fix) actually exposed issue 2590: "jn_zeros in functions/special/bessel.py should return SymPy Floats, not Python floats". The rest shouldn't be too bad (though I wish there was someone around to help me with tensor/index_methods.py).

The real problem was getting the doctests to actually run. Now, to start with, doctests also need to be converted by 2to3. This isn't done automatically but by passing a "-d" flag to 2to3 (which converts only doctests). Unfortunately, this crashed in ntheory/factor_.py because of the use of reduce(). I couldn't work around this so I've opened an issue upstream (I've also opened two more issues for other 2to3 deficiencies that I've managed to work around). This is a problem because even if there's a fix upstream it will only apply to new Pythons. For the moment I've just deleted the relevant bit of doctest, but this warrants further discussion. [EDIT: In fact, I've just opened issue 2605 for this] There's one more fix required (unicode-related) but that can just be wrapped with a version check.

The second problem was running the .txt doctests. First I learned that I need to pass the .txt files to 2to3 explicitly, they aren't caught otherwise (this from Lennart Regebro's excellent Porting to Python 3 book, which is now available for free; thanks Lennart!). Then it turned out our doctest runner depends on an internal procedure (_load_testfile) which changed from Python 2 to Python 3 to have an extra argument for an encoding. Finally, the doctests had a few errors in them but nothing major (one polys issue, though). The real peculiar thing is that some lines seem to print an extra "None" after what they're supposed to print; this happens in tutorial.txt and matrices.txt but nowhere else. As I haven't been able to reproduce this, I must conclude it's related to our testrunner somehow. Furthermore, this only happens with .txt doctests which further narrows it down. It's not clear to me how to solve this yet. Our doctest runner is a bit of a mess, using slightly different methods for .py and .txt doctests and copy-pasting some upstream methods but not all. One solution might be copying the rest of the code over, another could be porting us to py.test. This is something to raise at the next (IRC) meeting with my mentor. It's a minor issue, not impacting any real functionality, but as it could hide other issues it should be fixed.

We've also reached a solution - of sorts - on the unbundling of mpmath. As Aaron and Ondřej were adamantly against unbundling it, and as no clear benefit was then seen to porting to Distribute (not that it'd be easy without unbundling), I've decided to "concede" that argument. At the moment, the plan is to write a script which would copy the code to a sympy-py3k directory and run 2to3 as appropriate on it. This will probably require changes to various files (setup.py at least should then be compatible with both Python versions without the need for 2to3) and it's not yet clear how all details will be handled but it's definitely possible. I feel such a solution is going to be fragile by definition and could lead to problems down the line, but it's also basically the only way to have my work visible to users in the short run so I will go along it.

Nevertheless, I feel I'm very close to completing my goal (and the Python 3 part of my project) of porting SymPy to Python 3. Once the above script is decided upon and implemented, I hope to get wider testing from other developers and users. It is almost certain that there will be issues not caught by the tests and as I don't use SymPy myself having others test them is the only way for me to see them. There's an imminent minor release of SymPy, 0.7.1., which unfortunately doesn't include much of my work. I hope, however, that once it's fully complete and merged it will quickly be followed by a 0.7.2-alpha release to allow testing beyond the developer community.

Jenkins for SymPy

Last week Ondřej was free to work on reinstalling the current server we use and I've since setup a Jenkins CI server there; you can find it at the old address. As before, anonymous users can see the build history while developers should drop me a note if they'd like an account. Currently the builds still have to be triggered manually, but I hope to get the Github-integration complete and have tests run on every change pushed. The crucial issue here was that the tests were taking up too much memory, causing Jenkins to swap and take hours to run the test suite. This was fixed with pull 507, which clears the cache after running each file. This more than halved the memory usage (from ~450 MB to less than 200) which allows Jenkins to complete the 6 test runs (py25-27 x {python, gmpy} ground types) in just under an hour. The change should otherwise be harmless enough, but issues could arise (which is why it isn't included in the upcoming release).

Increasing the number of tests ran (scipy and numpy integration should also be tested, but in separate jobs probably) and having a working framework for automatically testing pull requests are the remaining tasks, but I consider the Jenkins server running and this part of my project a success. It will also be invaluable when Python 3 support is finally in (as it's unlikely many, if any, developers will test on Python 3 when developing). I've also added another project to continually test a single file to help another GSoC student, Tom Bachmann (ness), iron out some issues with random numbers in tests. It tests his branch every 5 minutes, which is many hundreads of runs daily and a lot more coverage than a single developer could reasonably provide. I consider it an experiment, to see how feasible it is to offer a level of extra support for critical changes. So far seems to be working fine. There's also a Jenkins plugin worth investigating, which implements priority for projects, which could allow us to have many such "side" experiments without impacting the main testing process whatsoever. The plugin is currently under development, but it is something to look into.


Phew! That was one long post! Hopefully, it makes up for the missing last week post at least a bit.

GSoC: week 7

Ack! This has been a really bad week for me! I'd like to blame it on the intermittent internet access, but the reality is that I've just come home after about 6 months and it's hard to avoid catching up "just a bit". I did manage to fix some of the unicode issues bringing us down to just two test failures and 9 exceptions, but I should've had it all working by now. In other news, Renato has worked some more on running in PyPy, bringing it down to one exception and 9 failures - it's quite possible that some of those are actually SymPy errors where we rely on some CPython-specific detail, those are going to be particularly interesting to track down.

GSoC: week 6: Further Python 3 porting work

Continuing on last week's progress, I've worked on the other exceptions and, with the help of Mateusz and Ronan, we are now down to some 33 test failures and 9 exceptions. I've sent a pull request with my current work, you can see it here; I've agreed with my mentor that sending frequent pull requests is the right path to take. I've been traveling for a few days, but my immediate goal is to bring the above numbers down to zero and get the doctest suite to run (which should point out further errors). The pull request also contains a commit marking all errors with a "FIXME-py3k" in the code, which should hopefully encourage other developers to take a look.

My other goal for next week is to port us to Distribute finally, which shouldn't be too hard. Like last week, the biggest hurdle is the decision on mpmath. Once that is taken care of, I'll see about using py.test again (SymPy currently uses an older fork we made) - this will make testing easier (as there have been quite some improvements since it was forked) and will also help when we finally start using Jenkins as it can export data to JUnitXML (which is handled well by Jenkins). This has the downside of disabling our benchmarking suite, but I've agreed with Ronan that we can keep the old code around until it too can be ported. It's hardly used, so it's not that big of a priority.

Another piece of good news is that the core SymPy tests now run under PyPy, which is the second part of my GSoC project. You can see the relevant issue and star it to automatically get any updates. Thanks to Renato for trying this out and reporting back!

GSoC: week 5: SymPy now runs in Python 3!

As the title says, you can now run SymPy in Python 3! No, it's not perfect, there are test failures (or rather, exceptions, some 200 of them), but ~90% tests pass and that's getting somewhere. I haven't tried actually using it, but at least for some definitions of "works", SymPy definitely works in Python 3. I'll document some of the issues I solved below, and I'll also update the porting tips and tricks page when I get the chance.

The cmp issue

While the new unicode/string features are probably the major Python 3 incompatibility for most programs, SymPy's foremost issue was the use of cmp(), which has been discontinued in Python 3 in favor of __lt__ and other rich comparisons. SymPy uses cmp() in various areas of code (sympy.core.Basic for one, and almost everything derives from that). I've tried to approach this from various angles, but most required a deeper understanding of SymPy than I currently posses. In the end, I opted for the most ingenious solution of all - sidestepping the issue. To see how I did this, first lets see the two different ways the removal of cmp() affects us:

a) builtin.sorted() and list.sort() no longer accept the cmp argument providing a comparison function. Use the key argument instead.

Now, this is a good thing! The cmp() function is slow in this case, as it will compare every two elements (or so). In comparison, the key function takes some defined "key" value and sorts based on; this means it's called exactly once on each object. However, if you rely deeply on customized __cmp__ functions, it can be difficult to simply use a key instead of cmp. The following recipe, which provides a cmp_to_key function, allows us to avoid this by doing just what the name says. It's not an elegant solution, but it allows us to move on with porting at least. The recipe has been included in the functools module in Python's 2.7 and 3.2, but we implement it in core.compatibility as we need to support more Python versions.

b) The cmp() function should be treated as gone, and the __cmp__() special method is no longer supported.

This is the root cause of the above point. Again, SymPy uses this quite extensively and fixing everything is beyond me at the moment. However, it is possible to naively define cmp as (a>b) - (a<b) if the builtin is not available, and I did exactly that in core.compatibility. That this works means that we aren't doing anything too complex with out __cmp__ implementations (it does fail in one case, though), which means that porting is possible.


Neither of the above changes is efficient or very nice at all (in fact, I described them as "evil" in a commit message). Still, going over these issues allows me to see the other problems that will arise and work on fixing them. It will allow me to finally have some visible progress on my project. This is important because my goal for the mid-term evaluation is to have a working SymPy in Python 3. Once SymPy is working completely, I will go back and implement a "correct" solution for this.

The other issues I solved are comparatively minor: I fixed some warnings produced by "-3" (the backquote is no longer a shortcut to repr(), tuple parameter unpacking has been removed) and some of the exceptions raised when running with Python3 (file() has been removed and we were encoding to ascii needlessly. I plan to continue working on these exceptions; as a lot of the tests all fail at the same points, I except to rapidly have almost all tests running in Python 3 (if not passing). I've also added a "FIXME-py3k" comment in a couple of files where I identified the issue but couldn't solve it; these are all things to discuss with my mentor. If someone wants to test it out, I've written some instructions on the relevant issue in our bug tracker. Merging will probably wait some time, but I will push for it to happen as soon as possible.


Also during this week, I worked on setting up our Jenkins server, you can see it here. Anonymous users can see just the build history; developers should contact me if they want an account set up. It is currently down, however.This was mostly a test run - we decided to reinstall our server from an ancient Debian to a current Ubuntu and only then run the "live" Jenkins server. This will happen when Ondřej, who runs the server we use, has more time (around the middle of July). I don't expect further issues to arise.

I also have a test branch with a port to distribute (instead of the currently distutils we use). We need to use Distribute to support automatic running of 2to3 when installing. The porting itself was trivial, but I need to read a lot of documentation before I figured out what the differences are (I even asked a question on StackOverflow about it). To say the documentation is confusing would be an understatement. The major problem is how to solve the mpmath issue.

As I wrote in my week 3 report, mpmath is already Py3 compatible and 2to3 shouldn't be run on it. I've since checked it out and I don't see an easy method to skip the mpmath directory. It might be possible, but I don't see an easy way without going around Distribute entirely (which is, again, not the point). I even asked about in on the distutils-SIG mailing list, the answer is a "don't do it, unbudle instead". While I've been in the unbudle camp since the start, the discussion hasn't been so one-sided (I won't summarize it here, you can read the linked issue). In the end, though, if I don't find a way to support it in Python 3, we will have to unbundle (Ronan has a branch doing just this, and it is fairly painless).

Setting up and using Tox in SymPy

In my previous post on the subject I mentioned we decided on using Tox. In the weeks since, we've established a workflow and successfully used Tox to find various bugs (see my post from last week for a particularly interesting one). As I've found the Tox documentation confusing in places, I've decided to write a "how-to" for setting up Tox and then talk some more about how we use it.

Setting up Tox

A simple "pip install tox" will handle the installation. Then you'll want to create a tox.ini file where your setup.py is located. The .ini can be very simple, just listing the wanted environments and the commands to run in each.

[tox]
envlist = py25, py26, py27
[testenv]
commands=python bin/test []
         python bin/doctest []

Here we are just using the default environments (py24-py32, Jython, PyPy) and then running our test suite in each. The square brackets are important - our bin/test is programmed in such a way to accept a testname as an argument, it will then run just the specified test(s). The brackets allow us to replicate this behaviour with Tox, instead of running the whole test suite every time. However, Tox makes it very easy to define a custom environment. One such that we could use is:

[testenv:py27-32-gmpy]
basepython=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
deps = http://gmpy.googlecode.com/files/gmpy-1.14.zip
commands = python bin/test []

SymPy can optionally use gmpy to provide better ground types. This test environment tests it with a 32-bit version of Python 2.7; the "basepython = " allows us to specify the path to a specific python interpreter to use. The "deps" command specifies dependencies (multiple dependencies should be in multiple lines, like for "commands" above). It can download them automatically using pip, or you can specify a zip/tarball like above. You can also see the full Tox specification, but these are the most important commands and the ones most likely to be used. You can also see the tox.ini.sample file we use, it has some more examples. (Note: this is how configuration files should be handled in DCVS': a "canonical" .sample file, which should then be copied to .ini and modified to needs. The .ini file itself should be ignored by your DCVS)

Using Tox

Tox works by creating a virtualenv for each environment specified, by default in a .tox directory, so there is no way it can mess up your system. It will reuse the existing environments, but it's possible to force a rebuild with "tox --recreate". You normally run Tox with a simple "tox", which runs all the environments listed in "envlist". You can optionally specify particular environments with -e (so, "tox -e py25,py26" will run just py25 and 26). A neat feature here is that you don't need to have an environment in your default envlist to run it with "-e".

Now obviously, the basic usage is a simple "tox". That will build all the default environments and run the whole test suite in each. Unfortunately, our test suite takes around 10 mins to run on average hardware and we currently support 3 Python versions (to go up to 4 or 5 when we finally get Python 3 support). If we add in {gmpy / no-gmpy} and even {32bit, 64bit}, we quickly get to an unrealistic amount of combinations to run. Yes, it is important to run these tests occasionally, but it isn't realistic to expect a developer to run all of them for every change they make (that will be the job of our CI server, after all). Still, it's important to test at least some basic testing for every change. As such, I advocate running py25-py27 by default, every time. In my opinion, this is a good compromise between time and testing coverage. If a bug occurs in some version, one can then research it further by running more variations (with gmpy, on 32bit etc) or preferably just fix it if possible.

Another useful feature is that our test suite supports running just specific tests (as mentioned above). Depending on the code changes, it might be enough to run just the appropriate test suite. This will drastically reduce the time required (and will allow the developer to run in all versions, rather than the reduced list above). Alternatively, if just one test fails after a change, it's easier to debug it if you can run just the specific test.

That's basically it! Tox is very simple to use and quite effective. The biggest challenge is probably persuading developers to use it, but the value is clear. It's also invaluable for testing before a release, per the words of the maintainer of SymPy, Aaron.

GSoC: week 3

This week I've worked on adapting my work into commit-worthy chunks. The results are pushed to my fork on Github, you can see them here. I've tried to make the commits fairly self explanatory - the first several are just fixing the warnings "-3" produces in a matter that's Python 2.5+ compatible. Unfortunately, Googling the error didn't produce much information in most cases; because of this, I've created a page where I've collected the solutions I used to solve each warning. You can find it on the right-hand sidebar. I've since also updated the version of mpmath bundled in SymPy to 0.17 (which supports Python 3 but drops support for Python 2.4, thus representing the first commit not compatible with Python 2.4).

Unfortunately, here I've hit a problem. Namely, mpmath is written with a single code-base that can run both Python 2 and Python 3 unmodified (this is not the recommended course of action usually, but mpmath didn't need to change much so I guess it makes sense). Normally, this is good, but SymPy will need to use 2to3 and this creates errors in SymPy. Why? One banal example is something like:

try:
    from itertools import izip
 except ImportError:
     izip = zip

Now, when 2to3 runs on this it removes the "from itertools..." line and thus we get an error. This is just one simple case, but there are plenty of others. It's obvious that running 2to3 on compatible code is not going to produce the desired results. And this is where I'm stuck currently. 2to3 doesn't support skipping some directories, so I'll probably have to develop our "internal" script that will use lib2to3 directly. To do this, I've decided to first try to integrate automatic 2to3 into our setup.py, so that I'd have an idea on how it works. [As a side note, thanks to Lennart Regebro and Benjamin Peterson, who've helped me in this thread on the Python-porting mailing list when I asked how to speed up 2to3 -- using the latest version of it was particularly good advice as it provides ~40% speedup in my case]

Once mpmath is integrated properly, I'll be able to continue on porting SymPy itself. I feel like I'm on track with the timeline in my proposal, which would be a working SymPy by the midterm evaluations.


I'd also like to take a moment here to comment on the value of Tox. While porting yesterday, I encountered an error in a polys test. As mpmath is used heavily in this code, I thought it was something to do with the new version. Later, I realized it occurs in master too and quickly bisected it down. And this is where the trouble started, as no one could reproduce the error, with any combination of Python version, architecture, ground-types and cache used. Digging down deeper, we found that the hashes of some functions really were different for everyone, though it was just me getting the error. In the end, someone else confirmed it - Python 2.6, 64-bit, with python ground types (and only on Linux, I think).

The error itself was nasty - int(1) and long(1) do not have the same hash [EDIT: See Aaron's comment below for the details] (I was told similar problems occurred before as well). As the underlying hash was hashing lists of lists (possibly deeply nested), the fix was to convert it to tuples of tuples and then hash it. The patch was written by asmeurer, and the whole discussion can be found on the issue page (and the IRC logs from yesterday). The moral of the story? Use Tox! How else would we find something that only appears in such specific circumstances.

GSoC: week 2

This has been a particularly uninteresting week of work, as I've expected. I have done some work towards making SymPy code Python 3 compatible, but there are no visible results - it just doesn't run in Python 3. As soon as it's possible to at least run SymPy in Python 3, I will begin publishing my branch. A hurdle is that SymPy should still be supporting Python 2.4 until the release of 0.7.0. Unfortunately, this release has been "imminent" for a while now. Initially, I was hoping to start integrating my work in small chunks as soon as possible. However, as this is the second week already, I've had to start some work in parallel. I only hope merging later will not be a problem.

It should also be noted that constantly re-running 2to3 is not a pleasant task. While this has some implications for testing, it's more importantly slowing down my work.

Continuous integration and SymPy: Buildbot or Jenkins (with Tox)

In software engineering, continuous integration (CI) implements continuous processes of applying quality control — small pieces of effort, applied frequently. Continuous integration aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development. --Wikipedia, Continuous integration

The above definition might seem a bit strange and archaic - isn't most open-source software developed in small chunks and merged in as soon as possible? That's why we have git and other distributed version control systems, as they make this work much more manageable. Continuous integration also means testing, which is something that is especially important for a library, and that's where we hit the first snag. SymPy has a policy that all tests must pass before making a change, which is a valid policy, but the simple fact of life is that SymPy is supposed to work on various platforms across multiple Python versions (and different ground types!). This means that a single developer cannot reasonably check all possible combinations and in the long run introduces subtle bugs in the code, especially in eg. older Python versions. This is where a continuous integration server comes in.

The goal of a continuous integration server is to, well, continually integrate. It controls some slaves, gives them tasks and collates the results. Usually, this means building the project and running the test suite, but it can be anything. This process is then repeated nightly, or after every commit, or started manually (eg. with specific parameters); in general, CI servers are very powerful and extendable, to fit the needs of specific projects. Part of my GSoC project was to investigate which CI server could be used with SymPy. I have considered buildbot, which was used in SymPy previously, and Jenkins.

Current SymPy workflow is for another developer to run the test suite on a given pull request before (thinking of) merging. To automate this process, a helper tool called SymPy-bot has been developed. It is a simple Python script which can list all the pull requests and test a particular one. The results are presented in a table on pastehtml and a comment is automatically made in the pull request. In essence, a poor man's continuous integration. :)

Buildbot

Buildbot was used by SymPy before, so it was a natural first choice (the fact that it's widely used, notably by Mozilla and Chrome, is another big plus). Buildbot has a classic master/slave structure, which means that each slave has to be setup separately, with the appropriate environment prepared in advance. As a test, I have created a local buildbot and some slaves and played around with them. Buildbot presents the information in a table format by default (like this), but can easily send it by mail or to an IRC channel or any combination of the above. It is in general a very robust project. A big advantage is the existence of a "TryScheduler", which applies a given patch and runs tests. This closely approximates the current SymPy workflow and I consider it a very desireable feature.

Unfortunately, the robustness of Buildbot comes at the cost of complexity. Setting up a build slave is a non-trivial task and in the end I decided to take a look at Jenkins first, before continuing to work with Buildbot.

Jenkins

Jenkins (formerly Hudson) is a CI server written in Java, providing much of the same functionality as Buildbot. Unfortunately, it does not support Python natively. This is where Tox comes in. Tox is an "automation project" for Python programs, which uses virtualenv to create different environments where the program can be tested in. It is extremely easy to setup, the following tox.ini file is all I needed to have (and appropriate Python versions installed, of course): 

[tox]
envlist = py25, py27, docs
[testenv]
changedir=bin
commands=python test []
[testenv:docs]
commands=python doctest []

It is then run with "tox", which automatically creates the necessary virtualenv's (reusing them if they already exist, of course) and then executes the given commands. The [] brackets allow us to replicate the behaviour of "./bin/test hydrogen" (runs just hydrogen tests) with "tox hydrogen". I like Tox a lot, as it makes it easy for a single developer to test many Python versions (as long as they are installed). Alternatively, it is easy to modify the tox.ini file to test with just the available Python interpreters.

Moving on, Tox also provides seamless integration with Jenkins (through the use of a "multi-configuration project") which then provides all the features expected from a CI server, including automatic builds, nice presenting of data and so on (speaking of presentation, it would be a good idea to have our test tool support JUnitXML, which shouldn't be too hard). One disadvantage of the Tox/Jenkins combination is the lack of a TryScheduler like the one Buildbot has. I have spoken with the developers, and they will add it to their Github plugin (eventually). In the mean time, it should be possible to manually program the functionality we need (it is possible to request a build with a parameter, which could be the name of a pull request) accessing Github directly. As this is something SymPy-bot does, I wanted to speak with Ondrej how to do this exactly.

In conclusion, Tox/Jenkins impressed me enough not to go back to Buildbot. Tox is very simple to configure and can be run locally. Jenkins appears simpler than Buildbot to setup and maintain, while offering almost the same functionality. The lack of a try scheduler is unfortunate, but as we've basically already engineered a solution to the same problem, I'm confident a solution will be found. In general, though, I feel that using Tox/Jenkins is an excellent choice for any Python project concerned with compatibility. [Update: I've written a short guide to using Tox with SymPy on the Wiki, you can read it here]

./setup.py GSoC/blog

[title shamelessly stolen from my mentor, Ronan Lamy]

As hinted to by the title, I've been accepted for the Google Summer of Code 2011. My work will be on SymPy, which is a Python library for symbolic mathematics (think Maple or Mathematica), and my goal is to convert it to Python 3. Namely, I will try to support all Python versions starting from 2.5, and this should be accomplished with the use of a single code-base (and automatic running of the 2to3 tool in the Python 3 use case). You can read my full application here.

As this is a complex issue, the first order of business would be to establish an automated testing system that would be able to spot any regressions (currently, SymPy supports Python 2.4 - 2.7, and there are already various small bugs and differences in behavior based on Python version); the current contenders are Jenkins (via Tox) and Buildbot. I will write-up a more detailed comparison, as related to the specific use-case of SymPy, when I've tried both. Of course, any comments, suggestions or new ideas are very welcome.

My second major goal for the community bonding period (ending on the 23rd) is to work on the warnings running with "Python -3" produces. This is an important first step to Python 3 porting, as it will solve some real issues and help me understand the code better. If possible, I will also try to improve code coverage of the test suite (or, alternatively, bother the developers responsible for that piece of code), which is also important for preventing subtle bugs from appearing.

Here's to running SymPy on Python 3 (and PyPy!) by the end of the summer!