Testing is a fundamental component of the software development process -- or at least it should be. As a developer, I know firsthand how difficult it can be to discipline oneself to test as you develop. Honestly, testing can be downright boring. To me, there's nothing more tedious than writing test cases, defining the expected outputs, debugging the test cases themselves, and then actually running the tests.
When I look for a testing tool, I want one that melds with the nature of testing. Anything less will fail like a New Year's resolution; you'll stick with it for a while, but inevitably you'll revert to your old bad habits.
As a developer, I want a tool that supports unit and regression testing. The tool must provide feedback quickly, easily, and reliably. Honestly, when evaluating a testing tool, I look for instant gratification. ParaSoft's Jtest for Linux fulfills these requirements -- and then adds some surprises of its own.
When deadlines loom, it is often tempting to put off testing as long as possible since it seems to take time away from development. Unfortunately, the more you put off testing, the more trouble you'll get yourself into later. Luckily, when done right, testing can actually speed up the development process.
Testing doesn't only help you find bugs. In fact, once you get into the habit of testing as you code, you will find that testing can actually speed up development, not hinder it. There are three main ways that testing helps you program:
First, testing decreases development time by decreasing the amount of time that you must spend debugging code. Testing as you develop catches bugs earlier. If you catch a bug early on you won't give yourself a chance to add another! You will find it easier to debug code that contains only one or a small number of problems. However, the longer you wait to test the longer you give yourself to introduce more errors into your code. Code that has multiple bugs takes much longer to fix than code in which each bug is fixed as soon as it is detected.
Second, testing decreases development time by encouraging safe reuse. Unit testing allows you to test and validate each of your components individually. Once you validate a component you can reuse it with the assurance that the component will work properly. So, when new bugs do crop up, you can be fairly certain that the bug lies in the code that uses the component, not the component itself. Unit testing allows you to build a solid base of reliable software components. You can use these components safely since they won't introduce problems in your new programs.
Finally, testing decreases development time by allowing you to quickly test the impact of your changes. Regression testing allows you to refactor your code with confidence. When you set out to make changes to your code, you can do so without worrying about breaking it. If you have test cases ready, you can simply run them when you're finished with the change and compare the output to previous test runs to see if you broke anything. Without these tests, you might be reluctant to make changes because you would lack a quick answer to the question, "Did I break something?" When you do make changes, regression testing gives you immediate feedback on your changes.
Installation of Jtest is fairly straightforward. There are two installation files on the CD: jtest.linux.tar.Z and jtest.linux.tar.gz. Pick one of the files and copy it to the directory where you'd like to install it. Once copied you need to extract two files from the archive: install-jtest and README. install-jtest is the installation script. To install Jtest, simply execute the script and Jtest does the rest. Once the installer completes, you should add jtest/bin to your path.
Jtest doesn't require the installation of a JDK since it ships with a JRE. However, to use all of Jtest's features you will need a JDK. You can also configure Jtest to use the JRE or JDK of your choice.
Testing out of the box
Out of the box, Jtest allows you to begin testing your code. To test, you select a class by pressing the Browse button and choosing a class. Once you have picked a class, you begin testing by pressing the green arrow button. Jtest does the rest. When testing is complete, the results are displayed in a tree view.
As you can see in Figure 1, the class that I tested has six static analysis errors and one uncaught runtime exception. So without doing anything but picking a class and telling Jtest to test it, I found some problems. Now that's what I call instant gratification.
Jtest allows you to run three types of tests on your classes: static analysis testing, white box testing, and black box testing. If you just pick a class and test, Jtest will perform static analysis and white box testing automatically. Black box testing requires a bit more interaction.
Static analysis allows you to set up a number of rules and apply them to your code. For example, if your organization has a coding standard that says all private variables must begin with "_", you can set up a rule to test that all privates are prefixed with "_". When Jtest runs, it marks all variables that violate the rules as errors. Rules can be much more complicated, of course. Rules can test code for anything from simple conformance to naming conventions to the proper use of inner classes and method declaration rules. In all, Jtest ships with 174 built-in rules.
The rules aren't arbitrary. They're based on well-documented Java test practices. I was thrilled to see that many of the rules were based on the idioms presented in Java in Practice, by Nigel Warren and Philip Bishop, one of my all-time favorite Java books. Naturally, Jtest allows you to create new rules through a graphical rules builder.
Adding a rule is fairly straightforward, and the tutorials supplied with Jtest are invaluable. However, the rule builder is a powerful tool and it takes time to get the hang of formulating more complex rules.
Static analysis is a great tool for organizations with well-defined coding standards. Without an automated process, these standards must be enforced during code reviews. Jtest makes code reviews more efficient by freeing the code review from the task of convention policing.
White box testing:
When Jtest performs automatic white box testing, it attempts to break your code by feeding it unexpected inputs. So white box testing does not test functionality per se. Instead it tests an object's ability to accept and properly respond to unexpected inputs.
For example, when I tested a Queue class, Jtest attempted to enqueue a null parameter to see if the Queue's enqueue method could handle it. While a Queue can easily handle null enqueues, other objects might not react so graciously unless they have been specifically prepared for the input. Jtest helps you identify those weak points in your code.
White box testing is valuable because you can't always think of every possible scenario. In my testing, Jtest did a fairly good job coming up with cases I hadn't thought of. I was humbled by a few surprises and the subsequent code fixes.
A nice feature of Jtest is the ability to suppress individual tests. For example, if Jtest cites an error during a white box test, and you don't agree with the assessment, you can turn that specific test off.
Black box testing
While Jtest automatically performs static analysis and white box testing, black box testing requires user intervention since it tests for functionality. Functional testing verifies that an object does what it's supposed to do. For example, if you have a method that accepts two arguments and adds them together, it should return the proper sum.
Jtest provides two main ways to perform black box tests. The simplest, method manipulation, allows you to test an object by calling it with the arguments you specify. The second testing approach allows you to write test-case classes that perform more complex tests.
Method manipulation allows you to call a method with the arguments of your choice. Jtest provides a number of ways to define those argument values. The simplest method allows you to specify method arguments through a tree view. This tree view lists each method and its arguments.
To add an argument value, simply right click on the argument name and edit the value.
Often you'll find that you use the same constants over and over again. You may also want to use nonprimitives. Jtest provides a repository where you can define constants and methods for retrieving nonprimitive arguments.
Currently, Jtest has deprecated the use of the repository. Instead, Jtest allows you to directly write classes that will construct the argument values you wish to use. To define values through a class, you define a static method that returns the proper value, import it, and insert the method name as the parameter argument. Looking at Figure 4 you can see that I pass the get method values created by Inputs.numbersTable, Inputs.emptyTable, and Inputs.colorsTable. By allowing you to write your own classes for constructing the parameter values, Jtest gives you more flexibility than by just adding a simple method to the repository.
Test case classes
Simple method manipulation is an important way to test a class. However, there are times when your tests need to be more complex. For example, when testing a stack class, you'll want to see whether elements pop off in the proper order. To test the pop functionality, you will need to load some data onto the stack using the push method. Simple method manipulation won't allow you to perform such a complex test since you can only call one method. When you need to call multiple methods to properly test a class, you'll need to write a test class.
Jtest makes the testing process as easy as possible by providing a robust testing framework. You can write your tests once, set up Jtest to use them, and press a key whenever you want to run the test. Jtest allows you to save your settings to a file so that you can perform the same tests without having to set them up each time.
When you save your tests you not only save your settings, you save the results of your previous test runs, allowing you to keep a regression history. Using this history, Jtest informs you of changes that occur over time.
Jtest is a feature-rich product and this review wouldn't be complete without mention of some of the more advanced features: project view, batch mode, and stub generation.
In a project view, Jtest allows you to test multiple classes at a time through the project view.
When you test through the project view, Jtest searches for and tests multiple classes at a time. You can still set up and save your testing environment so you can easily retest later and perform regression testing.
GUIs are nice, but most organizations automatically build and test their projects offline late at night without human intervention. To facilitate such practices, Jtest provides a batch-testing mode. The batch mode is also useful for large projects containing a thousand or more classes.
Finally, Jtest automatically generates stubs for external resources such as CORBA, EJB, databases, and files. This is one more way that Jtest provides instant gratification and immediate feedback. You can begin testing early since you don't have to wait for the availability of external resources.
Like everything else in Jtest, the use of stubs is configurable. You can shut stubs off, have Jtest generate its own stubs, or use user-defined stubs. Jtest will even stub out CORBA and EJB services.
Comparison with free
In my day-to-day development, I tend to use JUnit, a free, bare-bones framework for writing black box unit tests. It's hard to compare JUnit with Jtest, though. While both do black box testing, Jtest automates everything and provides a complete testing environment. Jtest will even save your environment settings so that you can come back later and perform the same tests without having to set everything up all over again. As a bonus it'll even do regression analysis.
JUnit doesn't provide any of these features, but if you don't care about the extras and only wish to do black box unit testing JUnit will do nicely.
There's one more aspect of Jtest I haven't mentioned yet: price. Jtest starts at $3,495 per seat. Floating licenses and site licenses are available. Besides running on Linux, Jtest is available from ParaSoft in versions that run on Windows NT/2000, Windows 98, and Solaris.
A few problems
Of course, no program is perfect and there are two issues that stand out in my mind. First, Jtest's power comes at a price.
Second, all configuration is performed within one dialog. This dialog separates configuration options by placing them in a large tree view. Through this view you set classpaths, choose tests, and configure rules, among other tasks. You also use this view to start the rules builder. There are so many settings that it can be difficult to find the one you're looking for. From looking at the interface, it's not always apparent what can be done to one of the tree elements. You'll find yourself doing a lot of right clicking as you grope around for what you're looking for.
Jtest doesn't take advantage of menus as it should. Instead of having a menu option like Set Class Path, View Test Cases, View Rules, etc., Jtest makes you dig through the tree view. Sometimes what you're looking for won't appear until you right click on just the right element. This is just not intuitive.
The rule builder is another source of frustration. While powerful, it can be difficult to get used to. It is not a tool that you can use, or even easily find, without reading the documentation and going through the tutorial first. Luckily, Jtest provides an excellent set of documentation, tutorials, and examples.
A testing tool must provide feedback quickly, easily, and reliably. If a testing tool doesn't provide instant gratification in this way, no one will use it for long since it works against the grain of programming.
Jtest fulfills each of these requirements admirably. Right out of the box Jtest allows you to begin testing by providing automatic support for static analysis and white box testing. As development progresses, you can add black box tests as you need them (as well as suppress tests that you don't agree with).
While Jtest suffers from a few usability problems in the GUI, I would definitely encourage taking a look at Jtest for Linux as a Java testing solution.