In my previous post on the subject I mentioned we decided on using Tox. In the weeks since, we've established a workflow and successfully used Tox to find various bugs (see my post from last week for a particularly interesting one). As I've found the Tox documentation confusing in places, I've decided to write a "how-to" for setting up Tox and then talk some more about how we use it.
Setting up Tox
A simple "pip install tox" will handle the installation. Then you'll want to create a tox.ini file where your setup.py is located. The .ini can be very simple, just listing the wanted environments and the commands to run in each.
Using Tox
Tox works by creating a virtualenv for each environment specified, by default in a .tox directory, so there is no way it can mess up your system. It will reuse the existing environments, but it's possible to force a rebuild with "tox --recreate". You normally run Tox with a simple "tox", which runs all the environments listed in "envlist". You can optionally specify particular environments with -e (so, "tox -e py25,py26" will run just py25 and 26). A neat feature here is that you don't need to have an environment in your default envlist to run it with "-e".
Now obviously, the basic usage is a simple "tox". That will build all the default environments and run the whole test suite in each. Unfortunately, our test suite takes around 10 mins to run on average hardware and we currently support 3 Python versions (to go up to 4 or 5 when we finally get Python 3 support). If we add in {gmpy / no-gmpy} and even {32bit, 64bit}, we quickly get to an unrealistic amount of combinations to run. Yes, it is important to run these tests occasionally, but it isn't realistic to expect a developer to run all of them for every change they make (that will be the job of our CI server, after all). Still, it's important to test at least some basic testing for every change. As such, I advocate running py25-py27 by default, every time. In my opinion, this is a good compromise between time and testing coverage. If a bug occurs in some version, one can then research it further by running more variations (with gmpy, on 32bit etc) or preferably just fix it if possible.
Another useful feature is that our test suite supports running just specific tests (as mentioned above). Depending on the code changes, it might be enough to run just the appropriate test suite. This will drastically reduce the time required (and will allow the developer to run in all versions, rather than the reduced list above). Alternatively, if just one test fails after a change, it's easier to debug it if you can run just the specific test.
That's basically it! Tox is very simple to use and quite effective. The biggest challenge is probably persuading developers to use it, but the value is clear. It's also invaluable for testing before a release, per the words of the maintainer of SymPy, Aaron.
Setting up Tox
A simple "pip install tox" will handle the installation. Then you'll want to create a tox.ini file where your setup.py is located. The .ini can be very simple, just listing the wanted environments and the commands to run in each.
Here we are just using the default environments (py24-py32, Jython, PyPy) and then running our test suite in each. The square brackets are important - our bin/test is programmed in such a way to accept a testname as an argument, it will then run just the specified test(s). The brackets allow us to replicate this behaviour with Tox, instead of running the whole test suite every time. However, Tox makes it very easy to define a custom environment. One such that we could use is:[tox] envlist = py25, py26, py27 [testenv] commands=python bin/test [] python bin/doctest []
SymPy can optionally use gmpy to provide better ground types. This test environment tests it with a 32-bit version of Python 2.7; the "basepython = " allows us to specify the path to a specific python interpreter to use. The "deps" command specifies dependencies (multiple dependencies should be in multiple lines, like for "commands" above). It can download them automatically using pip, or you can specify a zip/tarball like above. You can also see the full Tox specification, but these are the most important commands and the ones most likely to be used. You can also see the tox.ini.sample file we use, it has some more examples. (Note: this is how configuration files should be handled in DCVS': a "canonical" .sample file, which should then be copied to .ini and modified to needs. The .ini file itself should be ignored by your DCVS)[testenv:py27-32-gmpy] basepython=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 deps = http://gmpy.googlecode.com/files/gmpy-1.14.zip commands = python bin/test []
Using Tox
Tox works by creating a virtualenv for each environment specified, by default in a .tox directory, so there is no way it can mess up your system. It will reuse the existing environments, but it's possible to force a rebuild with "tox --recreate". You normally run Tox with a simple "tox", which runs all the environments listed in "envlist". You can optionally specify particular environments with -e (so, "tox -e py25,py26" will run just py25 and 26). A neat feature here is that you don't need to have an environment in your default envlist to run it with "-e".
Now obviously, the basic usage is a simple "tox". That will build all the default environments and run the whole test suite in each. Unfortunately, our test suite takes around 10 mins to run on average hardware and we currently support 3 Python versions (to go up to 4 or 5 when we finally get Python 3 support). If we add in {gmpy / no-gmpy} and even {32bit, 64bit}, we quickly get to an unrealistic amount of combinations to run. Yes, it is important to run these tests occasionally, but it isn't realistic to expect a developer to run all of them for every change they make (that will be the job of our CI server, after all). Still, it's important to test at least some basic testing for every change. As such, I advocate running py25-py27 by default, every time. In my opinion, this is a good compromise between time and testing coverage. If a bug occurs in some version, one can then research it further by running more variations (with gmpy, on 32bit etc) or preferably just fix it if possible.
Another useful feature is that our test suite supports running just specific tests (as mentioned above). Depending on the code changes, it might be enough to run just the appropriate test suite. This will drastically reduce the time required (and will allow the developer to run in all versions, rather than the reduced list above). Alternatively, if just one test fails after a change, it's easier to debug it if you can run just the specific test.
That's basically it! Tox is very simple to use and quite effective. The biggest challenge is probably persuading developers to use it, but the value is clear. It's also invaluable for testing before a release, per the words of the maintainer of SymPy, Aaron.
The tox documentation is indeed a little confusing. For example, I had to figure out on my own that to use multiple dependencies, you have to put them on multiple lines and indent each additional line (all the docs say is "deps=MULTI-LINE-LIST").
ReplyDeleteBy the way, if you only run the tests in on environment, I would recommend it being Python 2.5. Almost any bug that only shows up in one environment and not another would show up in that one, because there are still a few Python features that were not implemented until Python 2.6 (like the f(*args, namedarg=1) syntax, and the use of the with statement without importing it from __future__), though it isn't nearly as bad as it was with Python 2.4.
Of course, if you're only going to run one or two test files (like just the core, for example), it's worth it to run "tox core" instead of "./bin/test core" in my opinion.