Monday, July 25, 2011

GSoC: week 9: Almost there

I've just now seen I managed to completely miss my blog post last week. Bah! So, for the past two weeks, I've worked on two things in parallel. Lets take it from the top:

Python 3 support

This is going as well as could be hoped. The latest pull request is awaiting merging. With it, there are three exceptions and one failure in the main test suite remaining and this hasn't changed in about a week. I've asked Ronan for help on these, and he's opened an issue for one of the exceptions. It's something deep in the assumptions code; as Ronan knows his way best around this code, I hope he'll be able to solve it. I haven't looked in detail at the other problems, but hopefully they won't be too hard.

What I have been doing is working on the doctests and the failures they show - at least three real issues that aren't covered in normal tests have beeen uncovered. The others have been mostly problems with the doctests themselves - doctests rely on comparing the exact output and some minor things have changed in Python 3 (eg. is now ). Most of these fixes are in the pull request referenced above. There are 2 exceptions and 2 failures remaining to fix. One of the failures (while trivial to hack-fix) actually exposed issue 2590: "jn_zeros in functions/special/bessel.py should return SymPy Floats, not Python floats". The rest shouldn't be too bad (though I wish there was someone around to help me with tensor/index_methods.py).

The real problem was getting the doctests to actually run. Now, to start with, doctests also need to be converted by 2to3. This isn't done automatically but by passing a "-d" flag to 2to3 (which converts only doctests). Unfortunately, this crashed in ntheory/factor_.py because of the use of reduce(). I couldn't work around this so I've opened an issue upstream (I've also opened two more issues for other 2to3 deficiencies that I've managed to work around). This is a problem because even if there's a fix upstream it will only apply to new Pythons. For the moment I've just deleted the relevant bit of doctest, but this warrants further discussion. [EDIT: In fact, I've just opened issue 2605 for this] There's one more fix required (unicode-related) but that can just be wrapped with a version check.

The second problem was running the .txt doctests. First I learned that I need to pass the .txt files to 2to3 explicitly, they aren't caught otherwise (this from Lennart Regebro's excellent Porting to Python 3 book, which is now available for free; thanks Lennart!). Then it turned out our doctest runner depends on an internal procedure (_load_testfile) which changed from Python 2 to Python 3 to have an extra argument for an encoding. Finally, the doctests had a few errors in them but nothing major (one polys issue, though). The real peculiar thing is that some lines seem to print an extra "None" after what they're supposed to print; this happens in tutorial.txt and matrices.txt but nowhere else. As I haven't been able to reproduce this, I must conclude it's related to our testrunner somehow. Furthermore, this only happens with .txt doctests which further narrows it down. It's not clear to me how to solve this yet. Our doctest runner is a bit of a mess, using slightly different methods for .py and .txt doctests and copy-pasting some upstream methods but not all. One solution might be copying the rest of the code over, another could be porting us to py.test. This is something to raise at the next (IRC) meeting with my mentor. It's a minor issue, not impacting any real functionality, but as it could hide other issues it should be fixed.

We've also reached a solution - of sorts - on the unbundling of mpmath. As Aaron and Ondřej were adamantly against unbundling it, and as no clear benefit was then seen to porting to Distribute (not that it'd be easy without unbundling), I've decided to "concede" that argument. At the moment, the plan is to write a script which would copy the code to a sympy-py3k directory and run 2to3 as appropriate on it. This will probably require changes to various files (setup.py at least should then be compatible with both Python versions without the need for 2to3) and it's not yet clear how all details will be handled but it's definitely possible. I feel such a solution is going to be fragile by definition and could lead to problems down the line, but it's also basically the only way to have my work visible to users in the short run so I will go along it.

Nevertheless, I feel I'm very close to completing my goal (and the Python 3 part of my project) of porting SymPy to Python 3. Once the above script is decided upon and implemented, I hope to get wider testing from other developers and users. It is almost certain that there will be issues not caught by the tests and as I don't use SymPy myself having others test them is the only way for me to see them. There's an imminent minor release of SymPy, 0.7.1., which unfortunately doesn't include much of my work. I hope, however, that once it's fully complete and merged it will quickly be followed by a 0.7.2-alpha release to allow testing beyond the developer community.

Jenkins for SymPy

Last week Ondřej was free to work on reinstalling the current server we use and I've since setup a Jenkins CI server there; you can find it at the old address. As before, anonymous users can see the build history while developers should drop me a note if they'd like an account. Currently the builds still have to be triggered manually, but I hope to get the Github-integration complete and have tests run on every change pushed. The crucial issue here was that the tests were taking up too much memory, causing Jenkins to swap and take hours to run the test suite. This was fixed with pull 507, which clears the cache after running each file. This more than halved the memory usage (from ~450 MB to less than 200) which allows Jenkins to complete the 6 test runs (py25-27 x {python, gmpy} ground types) in just under an hour. The change should otherwise be harmless enough, but issues could arise (which is why it isn't included in the upcoming release).

Increasing the number of tests ran (scipy and numpy integration should also be tested, but in separate jobs probably) and having a working framework for automatically testing pull requests are the remaining tasks, but I consider the Jenkins server running and this part of my project a success. It will also be invaluable when Python 3 support is finally in (as it's unlikely many, if any, developers will test on Python 3 when developing). I've also added another project to continually test a single file to help another GSoC student, Tom Bachmann (ness), iron out some issues with random numbers in tests. It tests his branch every 5 minutes, which is many hundreads of runs daily and a lot more coverage than a single developer could reasonably provide. I consider it an experiment, to see how feasible it is to offer a level of extra support for critical changes. So far seems to be working fine. There's also a Jenkins plugin worth investigating, which implements priority for projects, which could allow us to have many such "side" experiments without impacting the main testing process whatsoever. The plugin is currently under development, but it is something to look into.


Phew! That was one long post! Hopefully, it makes up for the missing last week post at least a bit.

7 comments:

  1. > and some minor things have changed in Python 3 (eg. is now ).

    What did you mean to put here?

    ReplyDelete
  2. > The real peculiar thing is that some lines seem to print an extra "None" after what they're supposed to print; this happens in tutorial.txt and matrices.txt but nowhere else.

    This is probably related to the printer and the displayhook.

    ReplyDelete
  3. If you haven't been already, can you run the 0.7.1 branch through Jenkins? I can't tell if you are or not because of it's horrible interface.

    ReplyDelete
  4. Aaron,

    1) Yeah, Google ate it because it thought it was html. It used to be "type 'int'" but now it's "class 'int'" (with brackets instead of quotes). Some of this is fixed in my commit "Make some doctests Python 3 compatible" in the latest pull request, and some other will be in the next. As I said, minor.

    2) Yeah, I guess, but I'd have expected it to happen for all printing done, not just in some cases. I'm sure I'll track it down, it's just completely baffling at this point.

    3) I've done it now, in a separate job. The easiest way to do this is to say "New job", then select "Copy existing job" and type out SymPy (don't forget to name the new job either). The configuration page will then open and you just have to change the branch to be used. Once the release is done you could just delete the branch or disable the project (and probably also delete the files it made on the server manually, as they can get quite big and the hard disk it small).

    ReplyDelete
  5. > 2) Yeah, I guess, but I'd have expected it to happen for all printing done, not just in some cases. I'm sure I'll track it down, it's just completely baffling at this point.

    Well, I know tutorial at least is one of the few files that manually sets the displayhook in the doctests. See http://docs.sympy.org/0.7.0/tutorial.html#symbols. The matrices file also sets it (so it can get 2-d printing of matrices.

    By the way, do you have problems with the wester file (http://docs.sympy.org/0.7.0/modules/polys/wester.html)? If not, I think it's because that file uses init_printing instead of setting the displayhook. I think you should change the other files to do that, and hopefully it will fix your problems.

    > Once the release is done you could just delete the branch

    I do plan to delete the 0.7.1 branch once it's merged with master. Will that automatically stop the job?

    ReplyDelete
  6. Ah, I'll check out the displayhook stuff, that's probably it. Thanks for the hint, I'm not sure I'd have figured it out on my own.


    Yeah, deleting the branch will work fine. Actually, since nothing is ran automatically right now, there's no need to do anything, this is just for future reference. The manual deleting of files is the important thing, as each job can take ~600-700 MB (sadly, Jenkins+Tox duplicates the git repo for every TOXENV) and the whole hard disk is just 15GB or so.

    ReplyDelete
  7. Well, I've had to fix problems with the displayhook in the txt docs two or three times in the past, so I know that it's a common cause of issues.

    Regarding file duplication, can you make it do local clones, which should use hard links?

    ReplyDelete