Unit Testing
CSC 207 Algorithms and Object Oriented Design

Summary: As you surely know by now, you have a responsibility to make sure that the classes you develop work correctly. One mechanism for checking your procedures is a comprehensive suite of tests. In this reading, we review the design and use of tests and point you to the Java testing environment we will be using.

Introduction

Most computer programmers strive to write clear, efficient, and correct code. It is (usually) easy to determine whether code is clear. With some practice and knowledge of the correct tools, one can determine how efficient code is. Your experience may tell you that it is often difficult to determine whether code is correct.

Formal proof is a gold-standard of correctness, but it typically requires a rich mathematical toolkit and significant effort. For many programs, it is often more than can be reasonably expected. There is also a disadvantage of formal proof: Code often changes and the proof must therefore also change.

Hence, we need other ways to have some confidence that our code is correct. A typical mechanism is a test suite, a collection of tests that are unlikely to all succeed if the code being tested is erroneous. One nice aspect of a test suite is that when you make changes, you can simply re-run the test suite and see if all the tests succeed. To many, test suites encourage programmers to experiment with improving their code, since good suites will tell them immediately whether or not the changes they have made are successful.

But when and how do you develop tests? These questions are the subject of this reading.

What is a Test?

As the introduction suggested, you should write tests when you write code. But what is a test? Put simply, a test is a bit of code that reveals something about the correctness of a procedure or a set of procedures. Most typically, we express tests in terms of expressions and their expected values.

We could express those expectations in a variety of ways. The simplest strategy is to execute each expression, in turn, and see if the result is what we expected. You have probably been using this form of testing regularly in your coding.

Of course, one disadvantage of this kind of testing is that you have to manually look at the results to make sure that they are correct. You also have to know what the correct answers should be. But reading isn't always a good strategy. There's some evidence that you don't always catch errors when you have to do this comparison, particularly when you have a lot of tests (which you should have for more complicated classes). I know that I've certainly missed a number of errors this way. An appendix presents an interesting historical anecdote about the dangers of writing a test suite in which you must manually read all of the results.

Since reading the results is tedious and perhaps even dangerous, it is often useful to have the computer do the comparison for you. For example, we might write a method, check, that checks to make sure that two expressions are equal. Of course, one must still be sure that the test itself is correct.

Now, confirming that our code is correct would simply be a matter of scanning through the results and seeing if any failed. Just as importantly, we can specify the expected result along with each expression, so we don't need to look it up manually.

Of course, there are still some disadvantages with this strategy. For example, if we put the tests in a file to execute one by one, it may be difficult to tell which ones failed. Also, for a large set of tests, it seems a bit excessive to print OK every time. What happens if there is an error in the middle of a series of tests? The whole thing may come to a screeching halt, which is a disadvantage if we want to see all the results.

When to Write Tests

To many programmers, testing is much like documentation. That is, it's something you add after you've written the majority of the code. However, testing, like documentation, can make it much easier to write the code in the first place.

As has been suggested before, by writing documentation first, you develop a clearer sense of what you want your classes and methods to accomplish. Taking the time to write documentation can also help you think through the special cases. For some programmers, writing formal postconditions can give them an idea of how to solve the problem.

If you design your tests first, you can accomplish similar goals. For example, if you think carefully about what tests might fail, you make sure the special cases are handled. Also, a good set of tests of the form this expression should have this value can serve as a form of documentation for the reader, explaining through example what the procedure is to do. There is even a popular style of software engineering, called test-driven development, in which you always write the tests first. (Test-driven development is a key part of Extreme Programming and a variety of so-called agile software development strategies.)

Testing Environments

To handle all of these additional issues, many program development environments now include some form of testing environment, in which you specify a sequence of tests and receive summaries of the results of the tests.

Different testing environments are designed differently. There is a rather extensive environment called JUnit that is used by many professional Java programmers. However, it has a steep learning curve and many features that are more rich than we will need.

To help promote your use of unit testing (which you will be expected to do on every subsequent assignment), we will use a somewhat simpler testing environment designed for students called tester Prima.

Please begin to familiarize yourself with tester by closely studying the overview (linked below), and then reviewing the following list of specialized test types. Read the first three now; the additional links to some more advanced features are included for completeness and future reference.

Tester Lib

Writing Tests for This Class

Here are some frequently asked questions about writing tests for OOP and this class.
What do I need to test?

You should strive to test every non-obvious method. You do not usually need to test get/set methods, as they are extraordinarily simple.

The variety of cases you need to test for should be appropriate to the complexity of the method. Don't write a gaggle of tests for a very simple function.

What sort of test should you use in the tester for void methods?

The typical circumstance here is that such a method call will modify the state of the owner object. In those cases, you should test that the state changed correctly (i.e., often with a get method). In the context of the tester library, this likely means initializing your "dummy" test objects within the test method, rather than at the class level.

Another related case to consider: If your method takes in an object as a parameter and modifies the parameter, then your test should check whether the object parameter is correctly modified.

What about private members and methods?

It turns out there is a lot of debate as to unit testing such relatively "private parts." Some say unit testing is simply for the code's "contract" (i.e., public interface). Others say testing should be at the smallest level possible to help isolate errors' locations. Still others say that when private methods are sufficiently complicated as to warrant unit testing, the code should probably be redesigned.

There is no single, simple answer for this class. In general, I lean toward writing tests to find errors, but not at the cost of excessive interface complexity. When in doubt, ask.

Most of the time it will not be strictly necessary. Inner classes are perhaps the stickiest case.

What about testing uses of Java's collections, such as ArrayList?

While the Tester library's checkExpect method for objects is nicely specified (it recursively checks all member variables that are objects until getting primitive values to test for equality), as of version 1.3.5, it seems not to work for the Java collections.

The tester documentation does specify that three categories of variables that we haven't talked about are not checked (transient, static, and volatile. Because we can't peer into the collections, it is hard to say exactly why it doesn't work for the Java collections.

Using checkExpect for objects would absolutely be the preferred method of use in general, and you should use it for any user-defined classes if/when the time becomse appropriate.

However, at this stage you can use tests incorporating the equals method of the various Java implementations (so long as you justify why they are sufficient).

What if we need sample external data files to test our homework assignments?

You should include them in the tar archive you submit and also "cat" them into the transcript you submit so they are easily visible in hard-form.

How do I do unit testing for an abstract class?

You can only test concrete methods. So you'll have to test only those methods that have concrete implementations. Because you can't instantiate an object of an abstract class, only its derivatives, if you have non-static methods implemented in the abstract class, you'd have to test them via the derived classes.

Appendix: Historical Anecdote

Many of us are reminded of the need for unit testing by the following story by Doug McIlroy, posted to The Risks Digest: Forum on Risks to the Public in Computers and Related Systems:

Sometime around 1961, a customer of the Bell Labs computing center questioned a value returned by the sine routine. The cause was simple: a card had dropped out of the assembly language source. Bob Morris pinned down the exact date by checking the dutifully filed reversions tests for system builds. Each time test values of the sine routine (and the rest of the library) had been printed out. Essentially the acceptance criterion was that the printout was the right thickness; the important point was that the tests ran to conclusion, not that they gave right answers. The trouble persisted through several deployed generations of the system.
McIlroy, Doug (2006). Trig routine risk: An Oldie. Risks Digest 24(49), December 2006.

If, instead of a thick printout, Bell Labs had arranged for a count of successes and a list of failures, they (and their customers) would have have been in much better shape.

Jerod Weinman
Created: 21 January 2010
Modified: 20 January 2011
Portions adapted from "Testing Procedures" by Janet Davis, Matt Kluber, Sam Rebelsky, and Jerod Weinman.
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License .