Announcing: mmockito 0.1.0 - MATLAB Mock framework based on Java Mockito

Well, the thesis got finished, and successfully defended, but the promised open-sourcing somehow took a back seat to celebrating and travel. Nevertheless, it's never too late to rectify the situation! Here is a copy of the announcement mail; more content will hopefully follow on the blog shortly (and probably doubling as documentation for the project).

I would like to announce the first release of mmockito, which is an open-source Mock framework for MATLAB, inspired by Java mockito. The work is a product of my bachelor's thesis and to my knowledge no other mocking frameworks exist for MATLAB. The project can be freely downloaded from its Github page:

https://github.com/vperic/mmockito

which can also be used to submit pull requests and report issues. A mailing list is also available for the project at:

mmockito@googlegroups.com

Documentation is (surprisingly) sparse at the moment; some is available through MATLAB and more information and a detailed comparison to Mockito and mockito-python can be found in my thesis, which can be freely downloaded. If you are interested in following the project, my personal blog will be updated occasionally; general discussion should be directed to the mailing list.

mmockito: How stubbing works

The previous post discussed some MATLAB-imposed limitations on the syntax we can offer. In the end, we've opted for a mock.when.aFunc(args).thenReturn(res) syntax. In the meanwhile, stubbing support has slowly matured and is essentially complete now, so it might be a good idea to showcase some of the functionality supported. While the syntax and the features provided were mostly inspired by (Java) mockito and python-mockito, we've diverged when we felt it made more sense for MATLAB code.

Basic usage

>> m = Mock();
>> m.when.aFunc(1,2).thenReturn(42);
>> m.aFunc(1,2)
ans =
    42
>> m.when.aFunc(1,'str').thenThrow(MException('a:b:c', 'exception!'));
>> m.aFunc(1,'str')
Error using Mock/subsref (line 57)
exception!

In the simplest use case, we can define an arbitrary function, provide the exact arguments it should take and the result it should give. "thenThrow" is offered as a convenience method to make code easier to read, the error would also be thrown if we just pass a MException object to "thenReturn". Also available is "thenPass", which is short for "thenReturn(true)".

Matchers

Because defining the exact arguments can be tedious, we've implemented Matchers. Matchers allow us to accept any argument satisfying a certain condition, instead of an exact value. Combining matchers and exact values is of course, also possible. For example, the Any matcher, if constructed with a class (either as a string or a meta.class object) only accept arguments of that class. If called with no arguments, the Any() matcher will accept anything at all.

>> m.when.a(Any('char')).thenReturn('ok');
>> m.when.a(Any(?double)).thenReturn('bad');
>> m.a('asdf')
ans =
ok
>> m.a(15)
ans =
bad
>> m.when.b(NumberBetween(1,5)).thenReturn(true);
>> m.b(3)
ans =
     1

It is very easy to define a custom matcher of arbitrary complexity, a couple of simple examples are included as a demonstration (and there might be a future blog post on this topic). A special case is the "ArgThat" matcher, which takes a "Constraint" in it's constructor (from the matlab.unittest.Constraints package). Since Constraints from the new unittest module serve basically the same purpose, they can be reused as Matchers. Our initial solution had us using Constraints directly, but they have a somewhat unfortunate naming scheme which doesn't "read" well. Nevertheless, with the "ArgThat" matcher users can reuse existing and custom Constraints, if needed.

>> import matlab.unittest.constraints.*;
>> m.when.a(ArgThat(HasNaN)).thenReturn('a Nan');
>> m.when.a(ArgThat(IsFinite)).thenReturn('no NaN');
>> m.a([5 6 14])
ans =
no NaN
>> m.a([1 2 NaN])
ans =
a Nan

Consecutive calls

Sometimes, we want to only stub a certain amount of calls. To do this, we can just chain the then* statements in any order. A "times" keyword is also offered for ease of use. Usually, the last stubbed return value will remain valid, but if we end the chain with the "times" keyword, it also will only apply the given number of times.

>> m.when.stub(1).thenReturn(1).times(2).thenReturn(2).thenReturn(3);
>> m.stub(1)
ans =
     1
>> m.stub(1)
ans =
     1
>> m.stub(1)
ans =
     2
>> m.stub(1)
ans =
     3
>> m.stub(1)
ans =
     3

It is important to note that the first stubbing is the one that is valid, and if it is setup for infinite stubbing it will not be possible to override it (short of creating a new mock). While in interactive use this might be a slight inconvenience, we get the added flexibility when making tests. For example, using Matchers, we can stub more complex behavior sanely.

>> m.when.stub(5).thenReturn('ok');
>> m.when.stub(Any(?double)).thenReturn('not implemented');
>> m.when.stub(Any()).thenReturn('bad!');
>> m.stub(5)
ans =
ok
>> m.stub(6)
ans =
not implemented
>> m.stub('asdf')
ans =
bad!

Strict vs. tolerant mocks; mocking real objects

By default, mocks are 'strict' -- methods which are not explicitly mocked will throw an error, the same as when calling a non-existent method. However, we can also create 'tolerant' mocks, which instead just return an empty list (MATLAB's equivalent of None or null in other languages).

It is also possible to pass a real object to the constructor of a Mock. In that case, if we are not stubbing a given method, we will pass it on to the object used in the constructor. Stubbed methods are always preferred, even if they shadow an existing method. Note: this feature isn't strictly complete as of right now and might change, but probably won't.


I hope that the presented features are compelling enough to interested some of you in using mmockito. If there's something more you'd want to see, don't hesitate to let me know. And, if you're interested in testing this code, I would be very glad for the feedback - please contact me! Coming up next, verification!

Mocking syntax and MATLAB's method handling shenanigans

One of the first decisions made when starting my thesis was to emulate the python-mockito syntax and provide similar features (namely, stubbing and verification). Why Python and not the original and more popular Java version, I'm not sure. Perhaps simply because when I approached my mentor, I was inquiring about any possible Python projects. In any case, the stubbing syntax boils down to:

when(mock).aFunction('arg').thenReturn('res')

Which actually looks quite clear and intuitive. Now, it was obvious from the start that to support such arbitrary calls to objects, I would have to overload the subsref method in MATLAB, and my initial prototype code did just that. I also knew that obj.method and method(obj) produce the same result in MATLAB, so my code handled only the first variant (it was a feasibility prototype, after all). This turned out to be a mistake - as far as I can see (and others agreed) - calling method(obj) does not, in fact, pass through subsref. As such, the above snippet is just not possible in MATLAB, erroring with: "Undefined variable "when" or class "when"."

But the error did give me an idea? What if I created a class called "when"? It could catch such calls and pass them on to the mock object, whose overloaded subsref could then handle them appropriately. In fact, it could even simplify it and that can only be a good thing. Sure, there would be the extra overhead of creating another class (no, a function wouldn't work), but these are supposed to be used in tests, which are hardly performance-critical (or rather, the added overhead can safely be ignored). Several lines of code later, everything worked great, the above line was correctly caught and I was just writing a script to make my results reproducible, when MATLAB backhanded me with another error:

Static method or constructor invocations cannot be indexed.
Do not follow the call to the static method or constructor with
any additional indexing or dot references.

The error message is clear enough - no indexing after constructors - but the code worked fine from the command prompt. Of course, there's already a StackOverflow question on this: it works in the command prompt and functions, but not in scripts. It even works in cell-mode, just not regular scripts! The answerer on SO seems to think this shouldn't work at all, but I disagree, and another blog gives a decent reason: using Java code in MATLAB. Someone suggested using "eval" to evaluate the code, but that is just not an option in this scenario, as it vastly decreases readability. As such, I've reached an impasse: directly working with the mock object doesn't seem to be a good idea, but creating a "fake" class for some reason doesn't work in scripts.

One line of reasoning is that mocks probably shouldn't be used in scripts (and cell-mode works, anyway) but mostly functions, so using the "when" class isn't a big problem. The issue, of course, is that sooner or later you're going to hit that very cryptic error message. A cleaner solution would be to offer a more verbose syntax for the mock object, perhaps something along the lines of:

mock.when('aFunction').with('arg').thenReturn('res')

Verbose, doesn't flow so well ("doesn't read like a sentance"), not to mention that it's a departure from other frameworks and mockito itself; also not an ideal solution. The third variant is to offer both: mock objects would expose the above API, and the "when" class would just call the mock with the appropriate syntax. But two ways to get the same thing is not good design. At this point I turn to you, dear readers, as I know I've managed to accrue at least a few interested followers. What would you like to see?

Announcing my Bachelor's thesis: Mock library for MATLAB

After positive experiences blogging about my GSoC projects, I've decided to do the same for my Bachelor's thesis. I've always felt that writing helped me process and refine my thoughts, and I'm hoping to leverage this for my thesis. Therefore, I plan on writing regularly, even about failed approaches.

As the title suggests, my thesis will be about creating a library for Mock objects for MATLAB. During talks with my mentor, we concluded that, while a unit-test library for MATLAB exists (xUnit) there is no support for mock objects. Mocks are very useful as they allow developers to write robust, correct tests faster, so their lack in MATLAB can be felt. As MATLAB supports object-orientated programming, we are reasonably certain that Mocks can also be implemented. Thus, the first part of my thesis is to conduct a feasibility study and develop a simple prototype as a proof-of-concept. To do that, I would need to define exactly which features I am looking for.

Coming from a Python background, my first instinct was to check existing Python Mock libraries (from this nice list). Most of them are ports of existing mocking frameworks to Python, though there are varieties. What was more interesting to me is that there is actually quite a bit of confusion about terminology (in particular, the term "mock" is used to refer to objects of vastly varying complexity). Investigating this, I came across an excellent article by Martin Fowler, Mocks Aren't Stubs. The article is a great read throughout, as it goes into quite a bit of detail on "mocking" and general design issues; it also made me realize that most approaches are quite complicated. Taking a step back, it's fair to assume that the "average" user of MATLAB isn't a dedicated CS specialist, but rather a researcher. As such, I believe ease of use should be valued the highest, otherwise no one would take the time to learn these tools.

This approach led me to mockito (nice presentation on differences between mockito and other frameworks, slides 38-40 provide direct comparisons of code). mockito defines itself as a "Test Spy" framework, where a "spy" is in essence a stub which remembers which interactions were invoked on it. This is a very natural, light-weight approach which is easy to pick up and almost as powerful as more verbose frameworks. After discussing it with my mentor, we both agreed that basing the new framework on mockito is the right approach. The next course of action is to create a basic set of use-cases, based on existing mockito examples, and start implementing them. This first code will be strictly proof-of-concept.

Finally, one of the first decisions we made was to make the final, resulting code public; development will also probably happen on Github. Philosophically speaking, I would prefer that any work I do be actually useful, and the idea behind creating a library is for many people to use it, and that cannot be accomplished in the confines of a single faculty or university. It would also be useful to have some projects which would like to use Mocks, so that the library can be tested (and designed!) with real code in mind. Here I would like to turn to the community - if someone has felt a need for a similar library, please send me your desired use-cases and any other comments and ideas you might have. They would be a tremendous help!

Bootstrapping Trial in Python 3

Initially, I had tried an "extensive" approach to porting Twisted - picking a certain error and fixing it in every module. Unfortunately, as I've found out, this isn't very practical: not only is Twisted a large code base, it's also very old. While updating this crufty code might have been doable, Twisted also has a requirement that all changes to code need to be tested (and I think this is very nice!*). This has been enforced quite strictly in the last few years, but of course, the code using the really old, Python3-incompatible idioms, is the same code which has no tests. As such, to make any sort of substantial change I would also need to write tests. This proved to be a little too much, and Itamar suggested I consider a more "intensive" approach - fixing Twisted a module at a time, starting with the core.

In this I also meandered slightly, but after discussing it with exarkun on IRC, we concluded it would be best to pick a file with tests, run it under Python 3 and fix the failures which arise. This is in line with Twisted's programming paradigm, test-driven development, and is a very comfortable way of working. The idea, of course, was to start with modules which have no dependencies on the rest of Twisted, and then work "down" the dependency tree as individual modules are ported. While this sounds ideal, I've encountered two problems: the minor one is that Twisted depends on itself a lot, and it's hard (although not impossible) to identify modules which do not use any others; the major issue is the test runner itself, Trial.

Trial is Twisted's (quite versatile) framework for testing, based on the standard unittest module. In time, the TestCase class was completely rewritten (though in a compatible way) to support various features which make testing easier. Now, when importing a file in Python 3, it needs to be syntax-compatible with Python 3, but all of it's imports need to be compatible too. So now, each test subclasses from twisted.trial.unittest.TestCase and the twisted.trial.unittest module is very large and unfortunately imports a large chunk of Twisted itself (notably, twisted.internet.reactor, but also half the twisted.python package). Therefore, it's impossible for me to actually run the tests, as I need Trial and Trial needs other things and none of this is compatible with Python 3. I had tried writing a large patch to at least make Trial importable, but it was rejected (and for good reason, I now think). Obviously, the huge patchset would need to be broken into smaller tickets, but preferably in a logical way.

Luckily, the solution came via the official unittest module - if I only change the test case to import from the official library, rather than from Trial, it will work! Then a simple ``python3.2 -m unittest twisted.test.test_whatever`` runs the tests. I have successfully used this method for several simpler files but I fear the low-hanging fruit are gone - as was to be expected, many test files do use functionality provided only by Trial's TestCase. I am still trying to "pick around" here and there, and have also submitted tickets which do not fix a specific module, but just a single issue (eg. removing __cmp__ in t.p.versions, removing uses of UserDict). It is clear, however, that this approach will not lead me to my immediate goal - running Trial itself under Python 3.

And this is where I currently am: my goal is to bootstrap Trial, to make it runnable in Python 3, which will make running tests (and, by extension, fixing relevant failures) much easier. The "pick a test_file and fix it" method cannot bring me there and I've been unable to think of a better alternative. One idea was to use an alternative TestCase implementation (where I tried testtools, which unfortunately isn't as-is compatible with Twisted's tests); using a different runner wouldn't help, as the modules would still need to be imported. Another idea is to provide some sort of temporary class, which would extend unittest from the official library with the specific methods I'm lacking; this class would then be deleted as soon as it's possible to run Trial itself. This doesn't strike me as a very clean approach, but it might be the only plausible one, unless someone has a different suggestion...

In the meantime, I'm focusing on fixing what I can (even if it doesn't directly lead to supporting Trial) and making more "general" changes, to lower the size of further patches (but there will be at least a couple of big ones, there's no avoiding it). In fact, I've been focusing on making tickets as small as possible, to ease review burden, though I've still got plenty awaiting review: any help on this front would also be very appreciated. I've also tried reviewing other tickets, to ease the general burden, though the one case where I actually "passed" a review I had to revert the change, so I'm trying to be more careful about it now.

*While I do find it very nice, I do have some issues with this policy and I feel that a few carefully thought-out exceptions would have been very helpful in my project. More thoughts on this in a future blog post.

Another year, another GSoC!

Well, this blog post took long enough, but I'm happy to announce that I've been accepted once again for the Google Summer of Code, this year for Twisted ("an event-driven networking engine written in Python"). What's more, my project is essentially the same as last year - porting Twisted to Python 3, or at least getting as close as possible (my actual proposal is available on Google Docs). Unfortunately, compared to last year, my school load was much higher this time around, so I've done much less work than I would have liked.

At the start, I've mostly focused on fixing the warnings thrown when running the test suite with "-3", taking care of most of the trivial ones (eg. has_key, apply, classical division). Currently, I'm looking into replacing "buffer()", which was a built-in but is removed in Python 3. While the work is similar, the workflow for getting changes in is quite different from SymPy. Twisted uses a svn repository, and trac for issue tracking; each change must be done in a separate branch and have a corresponding ticket; SymPy uses the classical Github + git workflow, with pull requests and reviews in the online interface. Now, I got too used to git to just give it up easily (especially as the Twisted workflow almost requires additional tools on top of vanilla svn), and this guide was very useful in setting up git svn. Although I'm getting used to the review process (eg. changesets are reviewed in total, not per-commit), I still find the Github (and git) model more productive - it streamlines review and allows small, atomic commits to be made (although I've been trying to keep my changes as small as possible, each such small change requires the opening of another ticket and creation of another SVN branch, so there's a point at which it's too much effort to do it all). Still, Twisted is unlikely to change so I will have to accommodate - at least I can follow my own practices in my own repo.

The next steps are deciding which porting strategy to ultimately pursue - some Twisted developers suggest a single code-base strategy (py2 and py3 compatible), I personally favor a single code base which relies on 2to3 for py3k compatibility and there was even an attempt at a dual code-base, by Antoine Pitrou. While I feel that approach is the least likely to succeed, as it introduces a high burden on the maintainer (and the effort has indeed stalled), the code already there will be helpful in my own work. Often, the changes made in py3k code can be reused in the "main", Python 2 code with little or no changes. Still, all approaches deserve investigation and my mind is still open to other ideas. Twisted currently supports Python 2.6+, which makes my job easier. The final piece of good news is that I may be able to get some sort of help (or at least support) from Canonical, as part of their plan to install only Python 3 in the next desktop Ubuntu release.

Thoughts on Google Code-in 2011

Google Code-in is the high-school equivalent of the Google Summer of Code. The program ran from Nov 21st to Jan 16th, though we've only now gotten around to sending a "summary" mail to the list about it. As Aaron noted, we've had some translation work, some work on SymPy Live and a bevy of documentation and code improvements. With 176 tasks completed, I'd say the whole project was a success for SymPy. I was involved as a mentor, so here are some general thoughts and observations about the process.

E-mail spam. In SymPy we didn't have a clean separation of mentor duties (eg. KDE only allowed tasks for which someone volunteered to mentor), so the initial idea was to add all (most) mentors to all tasks. This meant a lot of mails, an effect worsened by the fact that each commenter to the issue starts another "conversation" when viewed from Gmail (which I even reported to Melange as a feature request/bug). At the height of activity, I could get upwards of 30-40 mails ("conversations") daily, which by far dwarfed my other mail traffic. Then, because each comment is basically a separate mail, I wasted a lot of time looking at issue that someone already addressed (again, most mentors could handle most tasks). For the second round of tasks I didn't add myself to each task, otherwise I'm sure I'd have gotten even more spam. The bug I reported in Melange was fixed, so hopefully this will be less of an issue next year.

Being a mentor takes a lot of time. Partly a consequence of above, partly due to all the work being done, but being a mentor took a lot of time. Many students were unfamiliar with git (and didn't want to read the instructions on development workflow on our excellently-written (in my opinion) GCI Landing Page) and solving issues with them was a constant topic on IRC. Students also lacked follow-through with comments (or, occasionally, expected the work handed down to them) which didn't help. Finally, many students were very anxious, and didn't appreciate that we are all volunteers and cannot be around 24/7. All of this resulted in a process that was frustrating at times and stressful for mentors.

Regardless of all of the above, a lot of work was done for SymPy. While I didn't look at the stats, my feeling is that the biggest improvement could be seen in our SymPy Live interface (and our webpage) and our documentation. Yes, we also saw some code improvements, but they were probably a smaller part of the overall contribution (though by no means less important). Interestingly, I think this exposes the two types of tasks the GCI contest is well-suited to: tasks where there is no "in-house" expertise (anything web related in our case) and uninteresting tasks/chores (writing documentation, in our case and probably for most projects). In the first case, we managed to attract experienced developers who could improve our webpage much faster and better than any of the core developers. Writing documentation is also an important task, but one that is shunned by most developers. Still, it is mostly simple work and (more importantly) doesn't usually require in-depth understanding of the code. This made it ideally suited for new contributors. The financial award (100$ for every 3 completed tasks, up to 500$) was enough of a motivation for students. The all-around improvements to our documentation are probably the single biggest advantage of our participation in GCI.

Translations. In GCI, tasks were divided into categories and we needed to have at least 5 tasks in every category. While we managed to "fill-up" most categories, Translation was probably the biggest problem. As a, basically, command-line library, it does not make a lot of sense for SymPy to be translated in other languages. In the end, we created tasks for translating our webpage and tutorial to the languages covered by the development team and some of these were done, but I consider this a waste of time. Though this issue is "near and dear" to me (I'm not a native speaker of English), I'm of the opinion that it would be impossible for someone without at least a basic knowledge of English to program with SymPy. Simply, however much effort we put into translating, the class and method names will remain in English and there's no helping that. I very much doubt the newly translated documents will be even used and they're bound to fall behind as the original document changes. We also had to start using gettext to manage the translations, which is a non-trivial amount of work (and there are still some issues). In my opinion, it adds another layer of complexity (however small) for very little gain.

In conclusion: did we get stuff done? Yes, without a doubt. Would we have gotten more if the mentors used their mentoring time for coding? Perhaps, but not necessarily. Are some of the students going to keep contributing? Most likely not. Still, I would consider the whole program, and our participation in it, a success. Ideas for next year could be focusing more on stuff none of the core developers can do (eg. the website work), but we can't really say how far along will SymPy development progress during this year or which tasks might be available to students. Hopefully, more people will volunteer to mentor next year, which would help with most issues I raised here. It is interesting, though, that even with our normally very fast development process we couldn't handle the influx of student work. It'd be interesting to see how other organizations coped.

Here's to another GCI this year!