Garvan Institute eyes OpenStack to help it crunch big data
- 22 December, 2017 06:30
The Garvan Institute of Medical Research is preparing to use an OpenStack-based private cloud to aid the efforts of its researchers.
HPC senior engineer Manuel Ballesteros said the institute is eyeing OpenStack for running data-intensive non-HPC workloads.
“We run HPC and it’s quite successful for us because it gives us the opportunity to compute very efficiently, but one of the challenges we face with HPC is that it is not flexible.”
“With HPC, everything is tied into a single environment,” he said. “All your software needs to fit a specific version of the operating system and needs to talk to all the tools that sit on top of the hardware — between the user application and the hardware underneath.”
Demand on computing services is soaring thanks to genomics research, in particular, undergoing a “disruptive moment,” Ballesteros said.
“The price to sequence a genome has been reduced quite dramatically. The first genome was sequenced for US$100 million; nowadays you can sequence it for [a base cost of] about US$1000. That plus the power this information can give us makes genetic information a very valuable asset.”
“Because of this disruptive moment we are living right now, we have quite a lot of requests from people who want to run different things that in many cases don’t fit our HPC environment,” Ballesteros said.
Meeting those needs has led the institute to assessing the potential of OpenStack-orchestrated infrastructure and virtualized environments that can suit the needs of researchers.
Because the institute is a non-profit, the cost-effectiveness of open source software is a key drawcard, Ballesteros said.
Although the use of OpenStack had long been contemplated, last year the need to support tools that didn’t fit the institute’s HPC environment led to Ballesteros’ team in November standing up an OpenStack “playground”.
“We are not running production workloads yet because we are trying to control OpenStack and to make OpenStack fit our infrastructure; so we want OpenStack to talk nicely with our networking especially, and that is the part we are working on right now. Once we have that part ready then we can start deploying to production and add new OpenStack services.”
The institute is currently upgrading its server room hardware and following a hardware refresh Ballesteros said he expects to have the OpenStack-based cloud dealing with production workloads in mid-2018.