Continuous integration and SymPy: Buildbot or Jenkins (with Tox)

In software engineering, continuous integration (CI) implements continuous processes of applying quality control — small pieces of effort, applied frequently. Continuous integration aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development. --Wikipedia, Continuous integration

The above definition might seem a bit strange and archaic - isn't most open-source software developed in small chunks and merged in as soon as possible? That's why we have git and other distributed version control systems, as they make this work much more manageable. Continuous integration also means testing, which is something that is especially important for a library, and that's where we hit the first snag. SymPy has a policy that all tests must pass before making a change, which is a valid policy, but the simple fact of life is that SymPy is supposed to work on various platforms across multiple Python versions (and different ground types!). This means that a single developer cannot reasonably check all possible combinations and in the long run introduces subtle bugs in the code, especially in eg. older Python versions. This is where a continuous integration server comes in.

The goal of a continuous integration server is to, well, continually integrate. It controls some slaves, gives them tasks and collates the results. Usually, this means building the project and running the test suite, but it can be anything. This process is then repeated nightly, or after every commit, or started manually (eg. with specific parameters); in general, CI servers are very powerful and extendable, to fit the needs of specific projects. Part of my GSoC project was to investigate which CI server could be used with SymPy. I have considered buildbot, which was used in SymPy previously, and Jenkins.

Current SymPy workflow is for another developer to run the test suite on a given pull request before (thinking of) merging. To automate this process, a helper tool called SymPy-bot has been developed. It is a simple Python script which can list all the pull requests and test a particular one. The results are presented in a table on pastehtml and a comment is automatically made in the pull request. In essence, a poor man's continuous integration. :)

Buildbot

Buildbot was used by SymPy before, so it was a natural first choice (the fact that it's widely used, notably by Mozilla and Chrome, is another big plus). Buildbot has a classic master/slave structure, which means that each slave has to be setup separately, with the appropriate environment prepared in advance. As a test, I have created a local buildbot and some slaves and played around with them. Buildbot presents the information in a table format by default (like this), but can easily send it by mail or to an IRC channel or any combination of the above. It is in general a very robust project. A big advantage is the existence of a "TryScheduler", which applies a given patch and runs tests. This closely approximates the current SymPy workflow and I consider it a very desireable feature.

Unfortunately, the robustness of Buildbot comes at the cost of complexity. Setting up a build slave is a non-trivial task and in the end I decided to take a look at Jenkins first, before continuing to work with Buildbot.

Jenkins

Jenkins (formerly Hudson) is a CI server written in Java, providing much of the same functionality as Buildbot. Unfortunately, it does not support Python natively. This is where Tox comes in. Tox is an "automation project" for Python programs, which uses virtualenv to create different environments where the program can be tested in. It is extremely easy to setup, the following tox.ini file is all I needed to have (and appropriate Python versions installed, of course): 

[tox]
envlist = py25, py27, docs
[testenv]
changedir=bin
commands=python test []
[testenv:docs]
commands=python doctest []

It is then run with "tox", which automatically creates the necessary virtualenv's (reusing them if they already exist, of course) and then executes the given commands. The [] brackets allow us to replicate the behaviour of "./bin/test hydrogen" (runs just hydrogen tests) with "tox hydrogen". I like Tox a lot, as it makes it easy for a single developer to test many Python versions (as long as they are installed). Alternatively, it is easy to modify the tox.ini file to test with just the available Python interpreters.

Moving on, Tox also provides seamless integration with Jenkins (through the use of a "multi-configuration project") which then provides all the features expected from a CI server, including automatic builds, nice presenting of data and so on (speaking of presentation, it would be a good idea to have our test tool support JUnitXML, which shouldn't be too hard). One disadvantage of the Tox/Jenkins combination is the lack of a TryScheduler like the one Buildbot has. I have spoken with the developers, and they will add it to their Github plugin (eventually). In the mean time, it should be possible to manually program the functionality we need (it is possible to request a build with a parameter, which could be the name of a pull request) accessing Github directly. As this is something SymPy-bot does, I wanted to speak with Ondrej how to do this exactly.

In conclusion, Tox/Jenkins impressed me enough not to go back to Buildbot. Tox is very simple to configure and can be run locally. Jenkins appears simpler than Buildbot to setup and maintain, while offering almost the same functionality. The lack of a try scheduler is unfortunate, but as we've basically already engineered a solution to the same problem, I'm confident a solution will be found. In general, though, I feel that using Tox/Jenkins is an excellent choice for any Python project concerned with compatibility. [Update: I've written a short guide to using Tox with SymPy on the Wiki, you can read it here]