SAN MATEO (06/05/2000) - Those of us who have been in the software development game awhile know that inspection of our designs and source code is imperative to reducing defect rates. In four years as a software developer at IBM Corp., I learned firsthand the importance of code inspection early in the life cycle of a project, long before the testing or deployment phases begin.
Unfortunately, even when companies devote adequate time to design, development, and testing, software quality rarely reaches the level required for today's business environment.
A number of vendors offer testing tools capable of assisting developers with tasks such as style adherence and system load generation, but Reasoning Inc., an ASP (application service provider), is putting an interesting twist on software quality testing: service-based code inspection. Reasoning's InstantQA detects critical defects early in the development cycle.
We wanted to evaluate InstantQA using a real, enterprise-class C/C++ application, so we decided to have Reasoning test the open-source code for the latest version of the Apache Web server, which weighs in at just under 60,000 lines of code. To begin the inspection process, customers send the source code to one of Reasoning's inspection centers via a secure FTP link, which provides encryption, or via magnetic media.
Three days after I submitted the code, a 25-page report -- in the form of a Microsoft Word document -- arrived in my e-mail inbox. The time it takes to receive your report is an issue that customers negotiate with Reasoning before a service contract is signed. The time will vary based on the size and complexity of the application. Also, customers have the choice of receiving their reports in Microsoft Word, Access, or Excel format.
As I expected, the report came with an executive summary, but it was nicely detailed: It stated the project background; the number of lines of code inspected; the number of defects found, which were broken up into categories; and even a defect density ratio, which represents the number of defects per 100,000 lines of code. The report for the Apache code we sent categorized the problems into groups such as memory leaks, out-of-bounds array errors, and null pointer dereferences -- statements that may access a null pointer.
Following the executive summary were individual detail pages for each error found. The detail pages included a useful top-of-page summary -- including type of defect, identification of its location, explanation of the coding anomaly, and the possible negative outcome of the error -- along with the offending lines of code printed just below for easy reference.
Getting down to business
InstantQA employs three different processes during its analysis of your application. It combines automated (machine) code inspection technology, defect database analysis, and independent (human) verification by Reasoning staff. By using a combination of three methods on each project, Reasoning greatly increases the likelihood that its service will perform far better than most tools or other quality assurance methods in unearthing hard-to-find defects that will surely decrease the reliability of a software application.
In addition to defect inspection, Reasoning offers two optional services at added cost that check for violations of coding standards or to measure the fragility of software (the likelihood of new defects being introduced during software maintenance).
InstantQA costs 10 cents per line of code, with volume discounts available for very large applications with a multiple-inspection commitment. Applications of more than 100,000 lines of code but fewer than 1 million lines cost 8 cents per line with a three-times-annual inspection commitment. Applications of 1 million lines or more cost 8 cents per line for one or two annual inspections and 7 cents per line with a commitment to three or more annual inspections.
InstantQA currently supports C, C++, and COBOL. Java support will arrive soon.
Reasoning executives told me that average turnaround is four to five days on an application of approximately 200,000 lines. Larger applications would naturally require a longer turnaround, but as we said earlier, the company negotiates the delivery time on an individual basis. Reasoning officials stated that by year-end they hope to improve on this turnaround time and provide results within 48 hours.
As with any service, InstantQA brings the delicate issue of asset protection to bear. By inspecting for defects at the application level rather than the module level, InstantQA enables up to 100 percent code path coverage, meaning all parts of the applications are inspected in their entirety. This is wonderful for highlighting crash causes and data-corrupting defects, but it does so at the risk of compromising the security of the application and therefore the customer's competitive advantage. For many organizations, this will not be a risk worth taking. Commodity-type applications such as accounting or order entry should cause little worry. For applications that are more closely tied to company financials and customer advantage, each organization will have to weigh the benefits and risks carefully before making a decision to employ InstantQA.
Overall, I found InstantQA to have great value, but this nascent company has left more than a few stones unturned: Most importantly, the company's service agreement fails to address some essential issues that will concern many customers. For example, there is no wording in the contract that addresses the customer's rights in the event that Reasoning does not meet the agreed-on deadline. In many cases, even a one-day delay could upset a company's entire plan for public release of a product, and Reasoning should provide some way of making itself accountable for such an unforeseen occurrence.
Furthermore, the service agreement that Reasoning uses makes no mention of the company's policy -- or the customer's rights -- in regard to protection of a customer's copyright or intellectual property rights. Given today's ultracompetitive software market and the desire of most every organization to protect all intellectual property with hypervigilance, we view this as a major oversight. Although a Reasoning official told us that the company's policy is to destroy all traces of a customer's application once the inspection has been delivered, this fact is stated nowhere in the company's service agreement. We also think that Reasoning should consider providing a more bulletproof security method for transmission of the ensuing reports.
These oversights -- and the lack of Java support -- made our evaluation of InstantQA far less enthusiastic than it otherwise might have been. InstantQA's capability of increasing the quality of your software products and business applications, lighten the burden on your development staff, and hasten time to market will make it welcome and overwhelmingly useful to many companies. But until Reasoning works out some of the lingering business issues, we are forced to limit our overall score to Fair.
Tim Fielden is a senior analyst for the InfoWorld Test Center. Contact him at firstname.lastname@example.org.
THE BOTTOM LINE: FAIR
Business Case: By allowing companies to outsource much of the labor-intensive quality assurance cycle, InstantQA saves money and time over in-house testing in the full life cycle of a software project and increases application quality.
Technology Case: InstantQA provides up to 100 percent code path coverage and source inspection. It uses a combination of defect databases, automated source verification, and inspection criteria to ensure a complete analysis of C, C++, and COBOL code.
+ Quick turnaround time
+ Requires minimal staff intervention
+ Cost-effective over hand-testing
- Releasing software code to a third party raises security concerns- Java currently not supported- Service agreement lacks essential customer assurancesCost: 10 cents per line of code; volume discounts availableQuality of Support: A more comprehensive service agreement is neededPlatform(s): AnyReasoning Inc., Burlington, Massachusetts; (888) 429-4111; www.reasoning.com.