Can process assessments be automated?

Researchers at the University of Southern Queensland (USQ) have built a world-first automated process assessment tool based on the international standards for IT Service Management (ITSM) and process assessment

Researchers at the University of Southern Queensland (USQ) have built a world-first automated process assessment tool based on the international standards for IT Service Management (ITSM) and process assessment.

ITSM promotes service-oriented thinking in IT management. Process assessment is a methodical approach to determine process improvements. Organisations normally engage consulting firms to perform process assessments and to recommend improvements in ITSM processes. However, qualified and experienced ITSM consultants with process assessment expertise can be expensive and scarce.

One alternative to relying on consultants who use their proprietary process assessments is for organisations to carry out process assessment themselves. This involves appointment of an internal team of experts to undertake assessment.

However internal self-assessments have not been effective in practice due to a myriad of challenges such as the lack of objectivity, limited acceptance of findings, internal politics, limited knowledge or skills, and distraction from the regular work.

To overcome these challenges, automated process assessments could provide a serious competitive advantage for organisations.

The opportunity to offer a more efficient process assessment mechanism was the motivation for the team of researchers at USQ to develop a novel method for ITSM process assessment. As a result, a method called Software-mediated Process Assessment (SMPA) was developed in partnership with a leading cloud-based assessment platform provider Assessment Portal Pty Ltd.

The project team led by Professor Aileen Cater-Steel included USQ’s Professor Mark Toleman , Dr Wui-Gee Tan and Dr Anup Shrestha, along with IT standards expert Professor Terry Rout of Griffith University and Assessment Portal Directors Mr Paul Collins and Mr Graham Kennedy.

The SMPA approach aims to improve management processes of IT services in a more transparent and efficient way than either an external consulting firm or an internal team of assessors.

The SMPA software was trialled and evaluated with positive results at two case study organisations, Queensland Govegnment’s primary IT service provider, CITEC and the IT service department of an Australian local government authority, Toowoomba Regional Council (TRC ICT).

But can IT process assessments truly be automated?

About the SMPA approach

The SMPA approach allows IT service organisations to self-assess the capability of their ITSM processes using a cloud-based survey and decision support system. One of the major strengths of the SMPA approach is the consistency and repeatability of the approach for little to no additional costs and effort.

Consequently, iterations of self-assessments of processes facilitated by the SMPA approach can effectively and efficiently identify process improvements and contribute to continual service improvement (CSI).

The SMPA approach has four main phases: a process selection method; online survey for assessment questions; calculation of process capability scores; and generation of process improvement recommendations from a knowledge base.

Based on the collected assessment data, the SMPA approach calculates the mean value score to determine process capability score.

In addition to guidance from the international standard for process assessment ISO/IEC33000 series, the SMPA approach provides additional features such as the granular knowledge items for each assessment question and the use of coefficient of variation metrics to determine reliability of the assessment results. The SMPA approach can also assist the assessors by providing a dataset of testimony evidence for formal assessments.

Online survey

There are a number of inherent strengths of online surveys in comparison with other assessment data collection methods such as interviews and document reviews. Firstly, surveys are capable to measure a wide variety of data.

Process assessment data comprise information about process inputs, process outputs, perceptions of business value of the process, process activities undertaken, process knowledge and process documentation among other things.

Therefore, observing process activities, reviewing process documents and asking people about their work during a face-to-face interview may not reveal real and honest responses. An online survey can solicit data with limited interference to the respondent’s day-to-day operations.

ITSM process assessment collects data about the behaviour of peoples’ work (processes). Survey questions begin with the phrase “Do you know …?” and all questions relate to finding the respondent’s knowledge about the question at hand (as shown in the example below).

Online surveys can gather credible data, especially from the introverts in an organisation who respond best in quiet environments. Online surveys are also ideally suited for remote data collection from a geographically dispersed IT workforce as compared to document reviews or interviews.

The prevalent growth of outsourcing of IT service functions and the use of virtual IT teams across the globe means that online surveys can be a very useful assessment data collection tool to perform ITSM process assessments.


During SMPA evaluation, the participants reported that they found the online survey trustworthy, comfortable, effective and efficient. However further discussions led to a conclusion that a fully automated online survey that is strictly standards-based is not entirely useful.

It was discussed that human input is critical for the facilitation and support of online assessment surveys in order to clarify survey questions with relevant examples when needed and provide assessment support through expert assessment facilitators, online discussion forums and/ or help screens.

Benefits of the SMPA approach

A significant benefit of using the SMPA approach is that practitioners can gain a better understanding of the workflow to comprehend ITSM process capabilities. The implication for practitioners is that the SMPA approach provides a comprehensive set of design knowledge for ITSM process assessments.

The software helps an organisation avoid wasting scarce resources on elaborate and complex assessment techniques. Similarly, when organisations evaluate new or existing ITSM processes, they can regularly use the SMPA approach to assess how well the capabilities of their processes enable service improvement.

Another benefit of using the SMPA approach is that data collection can occur effortlessly. Such assessment data can truly capture the organisation’s viewpoint and the results of the automated assessment are immediately available.

During manual assessments, due to the length of the assessment exercise and consequently staff turnover, different staff participate in assessments due to the time lag between assessments and significant organisational changes during this time.

With SMPA, organisations are able to leverage evidence-based decision support on process improvements. It is well known that process improvement projects are disruptive and management buy-in is crucial early in such projects.

The SMPA approach provides informed choices to assure top management that a structured and transparent method is followed to assess the capability of ITSM processes. Additionally, when practitioners use the process improvement recommendations from the SMPA report for service improvement, they are making informed decisions.

The ITSM industry laments the lack of consistent measurement of CSI activities. SMPA might provide an answer to this challenge when carried out on a regular and consistent basis.

Four reasons why the SMPA approach is superior to a manual assessment

Firstly, during a manual assessment, a competent lead assessor makes the final decision on process capability levels and process improvement recommendations to be included in the assessment report.

The influence of the lead assessor in the manual assessment may introduce bias based on previous experience, a set of underlying assumptions, and perceptions and interpretations while determining the process capability scores. Such bias is absent in the SMPA approach since the software tool uses a consistent approach to calculate the process scores.

Second, the manual assessment is conducted in a group discussion environment including stakeholders from all roles for a particular process. Peer group discussions may be biased since senior managers and extroverts may dominate the discussion and assert their opinions. This behaviour may lead to a lack of insightful contribution from other process stakeholders due to inactive participation.

This limitation is removed in the SMPA approach as everyone has an anonymous and equal say about the processes in a more democratic manner through online surveys, therefore improving accuracy in depicting the true organisation voice.

Third, assessment questions are more granular in the SMPA approach. The manual assessment focuses on high level discussions and the assessors’ judgment of specific assessment indicators based on those discussions.The SMPA approach focuses on the ISO/IEC standard asking very specific questions for every indicator to determine the process capability.

A more granular approach improves the authenticity of the SMPA approach. However, this can also result in a significant time imposition for survey respondents by examining specific aspects of a process in detail.

Finally, process recommendation items are greater in number and more detailed in the SMPA report in comparison with the manual report. This is again due to the granular architecture of the SMPA approach where recommendation items were derived from the ITIL® framework stored in a knowledge base for each assessment question.

For every instance of process area risk, a recommendation item is triggered from the knowledge base and compiled in the SMPA report. In contrast, the manual assessment reported a limited set of action items that highlighted only the most important areas for improvement.

What can be automated?

From a practical standpoint, the SMPA approach collects assessment data, measures process capability and provides process improvement recommendations in an automated manner.

However, the completely automated approach may not be effective and a tiered approach is recommended.The SMPA approach is used to obtain an overall understanding of process capabilities. Building on the overall understanding, human judgment is necessary for assessment validation and improvement and engagement in process improvement.

Professors Aileen Cater-Steel and Mark Toleman congratulate Dr Anup Shrestha (centre) on being awarded the annual medal by the Australian Council of Professors and Heads of Information Systems (ACPHIS) for the best PhD thesis.
Professors Aileen Cater-Steel and Mark Toleman congratulate Dr Anup Shrestha (centre) on being awarded the annual medal by the Australian Council of Professors and Heads of Information Systems (ACPHIS) for the best PhD thesis.

This research, conducted at the University of Southern Queensland (USQ) was funded by an Australian Research Council (ARC) linkage grant in partnership with Assessment Portal Pty Ltd. The learnings from the project have been incorporated across Assessment Portal’s other business assessments, including its Skill Framework for the Information Age (SFIA) skills assessment.

Aileen Cater-Steel is a Professor of Information Systems at the University of Southern Queensland where she teaches and conducts research in the area of IT Service Management.

Join the Computerworld newsletter!

Error: Please check your email address.

Tags service managementitsmIT Service Management (ITSM)IT Service ManagementUniversity of Southern Queensland

More about ARCAustralian Research CouncilGriffith UniversityISOToowoomba Regional CouncilUniversity of Southern Queensland

Show Comments