Testing with Respect

As developers, one of our most important skills is how quickly we can produce finished code that works. This is not the same as simply producing finished code that works - anyone can do that given enough time - but what marks the top developers out from the crowd is how quickly they can produce production-ready code. When many are still just trying to get something to work properly, these guys are shipping their code to production.

Developers tend to be optimists who are motivated by the prospect of spinning something up from literally nothing and then seeing people use it and like it. So the focus is often on the thing which people are going to use - the app or gem that is the product of the developer’s hard work. It is also often the only view other people get of this developer. I, for instance, have never met Yehuda Katz, but I have a pretty good impression of his abilities and intellect from looking at what he has produced.

Some of this is down to natural skill, but an awful lot more is down to the habits and processes they use to develop code. One of the key elements is their testing.

Time for a confession - I have just fallen prey to a standard sin of poor developers. I have not been giving my tests the respect they deserve.

Over the past 3 months I have developed an API and published a gem called easy_json_matcher which is designed to help you test the JSON responses from your JSON API. Funnily enough the two are related. The API itself has a rather torrid history. It is a personal project that has never been launched and is now on its third attempt, representing my entry to web development with Java EE, my transition to Ruby on Rails, and having reached an intermediate level with the framework a total rewrite using some of the tricks I had learned.

I took a break from the API to work on easy_json_matcher, and it was here that really started to think about tests.

Initially, being an ADDer I just went at it and tried to mash all my ideas into the MiniTest framework, which is very good but not one I am very familiar with. I usually use RSpec, but I had created the gem’s scaffold using the rails g plugin command which uses MiniTest. So here’s the first lesson: Think! Your choice of test framework should be like your choice of language - use the one you are most skilled in, and is most appropriate to the job in hand. Just because Rails provides a generator doesn’t mean you should use it because it’s convenient, there are better options if you’re not developing for Rails (like bundle gem <name>).

So I pushed on and wrote a set of tests that asserted what I needed them to and then got on with writing the gem itself. The tests were not good. They were rough-cast with multiple assertions in single tests, long and confusing names, and practically no unit tests. Everything was a test from the user’s perspective. The gem itself was similarly muddled with responsibilities focussed around an inheritance hierarchy that quickly became bloated and confused. Here’s one monstrosity:

  test "As a user, given that I have specified that a boolean should map to a given key and that the actual value is not a boolean, I want to now that the value is not a boolean" do

     test_schema = EasyJSONMatcher::SchemaGenerator.new { |s|
       s.has_boolean(key: :bool)
     }.generate_schema

     no_bool = {
       bool: "false"
     }.to_json

     assert_not(test_schema.valid? no_bool)

     assert_match(/.* is not a Boolean/, test_schema.get_errors[:bool][0])
  end

The problem with tests like these is that they represent fuzzy thinking on the developers part. They do not allow the developer to think clearly and focus on one thing at a time. It is important to realise that your tests relate to your view of the app. A test which clearly demonstrates a single aspect of functionality demonstrates to you as well as anyone else, that you have thought the problem through and broken it down into its component parts. By preparing your tests in this manner you force yourself to consider the abstractions and patterns upon which that piece of functionality relies and in doing so are more likely to come up with components in your app which have clear and well-defined responsibilities. It was only once I started to think about cleaning up the tests that I began to see where I had gone wrong in my thinking with the architecture of the gem itself and so was able to produce a far more robust and changeable solution - which I implemented in half the time.

So the key message here is: keep your tests focussed and simple. Break the problem down using tools like RSpec’s context method separate them into relevant groups and your will find it a lot easier to think about the problem you are trying to solve in a concise and logical manner.

Back to the API. Having taken a couple of months off from it to work on my gem and deliver some client work, I then got back to my API and got back into the tests. Almost immediately, my enthusiasm started to ebb away, and it was down to how I had written the tests. The app itself is OK, but the tests were written almost like an afterthought, without sufficient attention being paid to how they will work. I had used RSpec so I had done a better job of using the test framework properly and breaking down the problems into their individual components. However, what I had not done was to actually develop the tests. Once again I had treated the tests like an afterthought and in this case had simply filled in the blanks of the generators - with the exception of the controller tests; I had the presence of mind to just delete those. However the request tests were (are) a major source of frustration. I have made them pretty exhaustive, but in the process generated a huge amount of bloat and boilerplate. One test file is over 300 lines long. Yuck.

The thing is, your test framework is an app. It’s just not the app. You can organise it in whichever way you like, but you should treat it like it is an app in its own right. Consider the abstractions. Refactor patterns into independent classes that can be reused, and diverge from the recommended approach delivered by the generators if that is appropriate. You don’t have to take all of their advice! It is your test suite after all. I have refactored the main testing patterns from the request tests into a series of helper macros that will greatly reduce the time taken to write the remaining tests and the amount of boilerplate, leaving me with more time and energy to focus on the different facets of what I want to test. If I had realised this in January I would probably be finished by now. The same attitude also applies to support services such as FactoryGirl. If you have a complex data graph, then refactor its generation into classes composed of FactoryGirl factories and keep that complexity safely locked away where you can manage it without allowing it bleed into the application.

All these things seem obvious, but they will only happen if you apply the same level of energy and concentration to your tests that you give to your app.

In summary:

• Treat your tests like they are an app of their own
• Get to know a set of testing tools well, and use them
• Don’t just follow the structure recommended by a generator. Use your skills to produce a test suite suitable to your application.
• Refactor your tests as aggressively as you would your own code.

Further Reading

Metz: Practical Object-Oriented Design in Ruby. Pearson, 2013 - Designing Cost-Effective Tests

Written with StackEdit.