Investing in a flexible infrastructure monitoring platform has helped the IT team at the Lions Eye Institute slash service desk calls and free up resources to take a more proactive approach to technology, according to the organisation’s chief information officer, Rod Bacon.
The not-for-profit Lions Eye Institute combines world-class research into eye disease with a clinical practice that treats up to 70,000 patients every year.
This combination of research and clinical practice presents Bacon and his team with a range of challenges, the CIO said.
“There are a number of challenges that most research organisations will face — things like growing data storage, big data, the need to collaborate with people around the world,” he explained.
“But we’ve also got very real operational demands: We’ve got patients walking in the door and equipment needs to be functioning then and there. Systems need to have 100 per cent uptime, which most research-type organisations wouldn’t have to worry too much about.”
In the past, IT had a break/fix approach and “desktop support mentality,” Bacon said. “We've worked hard over the last couple of years to drastically change that.”
Since the CIO came on board in September 2015, the IT team has undertaken a rebuild of the institute’s backend infrastructure from the network up, Bacon said.
“We started with network topology, and with all the underlying storage and compute environments and virtualization, and went through the whole gamut of things to build a robust infrastructure on which to redeploy the applications we use,” the CIO said.
At last count, the organisation relies on some 90 applications, he said.
“We were able to achieve a level of stability that enabled us to stop focusing on break/fix and start spending time working out how to better leverage investment in all this technology to deliver real outcomes for the organisation.”
From an IT perspective, one of the key problems in the healthcare sector has been encouraging clinicians to use the new technology available to them, the CIO said.
“People still rely on fax machines to send stuff around; a lot of clinicians like to have everything printed and sitting in a paper file in front of them,” he explained.
“Even though we've worked hard to build technology to replace all those things, getting [clinicians] to actually have confidence in the system has probably been our biggest challenge,” Bacon said.
“The only way we’ve been able to do that is by drastically improving the availability and the stability of the underlying systems. It’s a gradual process for us, but certainly we’re making some significant inroads.”
One ways of addressing the problem has been better delivering visibility across the institute’s infrastructure for the IT team, he said.
The clinical practice relies on a range of expensive equipment such as microscopes and a variety of high-end ophthalmic imaging systems typically connected to “legacy PCs running horribly outdated versions of Windows”.
“Support from the organisations that supply these things is ad hoc at best,” Bacon said.
“We found that a key challenge was making sure that the clinical equipment was always available and was managed correctly,” the CIO said.
Previously the clinical side of the institute had “largely gone and done their own thing with regards to this equipment,” he explained.
“We took the opposite approach, bringing it in and managing it just like any other piece of IT equipment.”
Bacon said that by rolling out Paessler’s PRTG platform, IT has been able to extend monitoring across the clinical equipment on the institute’s network.
The platform has also given IT the ability to monitor end-to-end data flows, the CIO added: “When a patient walks in and has a particular test or scan done, we can make sure that information ends up where it’s supposed to, in the right system, in the correct database, and against the correct patient record, which has created new levels of efficiency.”
Bacon came to the CIO role at the institute after a two-decade stint in managed services. “I was keenly aware of the benefits that were to be had by automating and having visibility and things that could repair themselves, rather than throwing people at it,” he said.
The institute previously had a networking monitoring system in place, but it was cumbersome and IT staff were not using it, Bacon said.
“My requirements [for a new system] were obviously not only something that’s incredibly flexible that can allow us to write our own sensors, develop our own scripts and monitor what we need to — I wasn't expecting to get everything I needed off-the-shelf — but I didn't want something that had to be driven by backend databases or obscure text files and those sorts of things,” the CIO said.
“I didn't want my staff to have to have a massive ramp up time in order to use a tool.”
“We kicked around probably four or five tools,” Bacon said. “We already had some vendors that we used for different other toolsets —our service test tools and those sort of things. They had offerings as well, but I guess after probably about three months of playing with a few different options, PRTG was obvious to me in the end.
“It was priced sensibly [and] it was evident after we did the initial pilot that we were going to be able to set this thing up and have it working inside a couple of days, and that it wasn't going to take me a bucket-load of training to get my staff to be able to benefit from it.
“More to the point, although we don't suffer from high staff turnover, I wanted to make sure that anything I bought was going to be useful beyond the time that myself or other people may be in the organisation.”
The initial rollout of PRTG took less than a week, though since then it has been subject to frequent minor enhancements by the IT team.
“We'll tweak some sensors every couple of days and we might sort of go, ‘Oh, there’s something else we can be looking at,” or ‘We really don't need to monitor that now,’” the CIO explained.
The institute has a couple of hundred users across three sites and around a hundred virtual servers.
“It’s not so much the size of the whole organisation — it’s the inherent complexity because of the clinical equipment,” Bacon said.
“A lot of that stuff is not SNMP compliant. Some of it runs Windows machines and we're able to do basic Windows metrics — obviously we can look at CPU and RAM and we can look at page file usage and that sort of stuff — but I guess where we've taken [PRTG] is having a look at some of the pieces of unique software that exist on these things and monitoring them...
“If something's not happening when it should, based on a particular piece of software, we need to know about it then and there.”
The backend rebuild and a proactive approach to infrastructure management has had a significant impact on the institute’s relatively small service desk team, Bacon said.
“If a patient comes in for a particular test, if that machine isn’t working, then obviously the phone starts ringing,” the CIO explained.
“If someone’s not available then and there [to fix the problem], not only do you delay that person who’s standing there, but the 15 appointments that are behind that person as well.
“Taking a proactive approach to looking at those quirky systems has really largely eliminated those outages.”
Service desk calls have been cut by 80 to 90 per cent.
“When I walked in the door here two years ago and logged into the service desk tool, at one point there were 57 tickets waiting to be addressed,” Bacon said.
“I logged in this morning, and there’s now 15; 14 of those are on hold pending something else that we can't actually action — a large chunk of that break/fix work has just disappeared.”
That has freed IT resources and allowed the CIO to cut down on bringing in external technology partners to work on infrastructure projects
“The only thing we’ve used someone for in the last 12 months was some advice around some specific things in networking,” Bacon said.
“Everything else we do in-house now. We’ve been able to refocus our staff to look at improving the business and improving our use of the technology, rather than just fixing it.”