AUCKLAND (06/02/2000) - Core New Zealand Government and other public-sector IT projects are on much the same level as private sector projects in New Zealand, according to the results of a benchmarking study published last week.

Performance figures are also on a par with those revealed by a 1998 survey of U.S. private and public organizations by the Standish Group consultancy.

However, the authors of the benchmarking report -- the New Zealand Institute of Economic Research and the Simpl Group - point to potential problems with the methodology. They used a voluntary survey -- with follow-up interviews of those who responded -- and therefore the sample was to some extent self-selected.

A key question may have been poorly worded. Also two of the most contentious abandoned Government efforts -- Inslaw and the police Incis project -- were not included in the survey because of other investigations in progress on them.

Other differences in the measurements between the local and U.S. survey had to be smoothed out by additional manipulation. The authors acknowledge success or failure of a project is a difficult concept to define, and there are even ambiguities in pinning down a project's precise start and end.

The voluntary survey format was chosen to permit valid comparison with the Standish survey. The authors further defend the methodology by pointing out that the mix of replies received was similar to the distribution of organizations consulted. "Respondents and non-respondents alike included a range of policy ministries and operational departments."

Furthermore "responses were received from organizations that have experienced high-profile problems" -- again supposedly an indicator that the sample is adequate.

Two sets of criteria for success were evaluated, the "tight" criteria -- adopted by Standish -- consider strictly "was the project on time, within budget and to scope?"

A set of broader criteria takes account of the positive outcomes from the development, whether strictly on-time and in-budget or not. This set of criteria is summarised as: "Did the project achieve organizational goals in an acceptable timeframe at an acceptable cost?"

On the tight criteria, a 38 percent success rate was achieved for all IT projects in the New Zealand core government sector. This compares with 38 percent in the non-core public sector and 31 percent in the private sector.

However, the Standish report "seemed to apply only to development projects -- that is (those) that involved a significant degree of (individual) coding", the report says. The report therefore matches up only development projects on the New Zealand side to the Standish figures using two different approaches to adapting the latter.

The New Zealand success rate for all projects ranks at 32 percent as compared with either 36 percent or 26 percent for the U.S., depending on the approach used to reconcile the figures.

On core government projects alone, the figures are 30 percent success for our Government projects as against either 25 percent or 18 percent for the U.S. government projects.

The first approach to analysis divides the sample purely into "successful" projects and projects with "problems". The second approach counts completely failed or cancelled projects as a third category.

This category encompasses 28 percent or all projects in the U.S. study and a surprisingly low 2 percent in the New Zealand study. This is clearly reduced by the non-counting of the Incis and Inslaw projects -- two more failures in a total sample size of 25 Government projects had they been included -- but the report's authors suggest the low figure could also have been influenced by the wording of a question, which asked for reports on the "three most recently completed projects". This may have meant respondents did not report on abandoned projects.

On the broader set of criteria, taking account of the value of what was eventually produced, the survey reckons 88 percent of core Government department IT projects were successful, 9 percent presented problems and the remaining 3 percent failed totally. This compares with an 83 percent success rate in non-core public sector developments and 82 percent in the private sector. No comparison on broad criteria was performed in the Standish survey.