Scientists to share open source data management app
- 29 January, 2004 07:08
In an effort to break away from proprietary computer systems, the Joint Australian Facility for Ocean Observing Systems (Jafoos) has started rewriting its Quality Evaluation of Subsurface Temperatures (Quest) data management application in Java and will release its source code to the public domain.
Dr Ann Thresher, the head of Jafoos, said legacy SGI systems are hampering the Quest data management application.
"Quest was originally coded in Fortran 77 for IRIX on SGI hardware," Thresher said. "Our SGIs are now very old and slow which limits what we can do with data management. It also limits who can use the software to those with older SGI computers. So we decided that all our quality control software should be platform independent."
Jafoos, a joint venture between the Bureau of Meteorology's research centre and CSIRO, chose Java language over Matlab for the new version of Quest, development of which starts this month.
"Quest is pretty versatile and lets us edit the data if we need to," Thresher said. "It measures many parameters, such as temperature and salinity, and can be dimensioned by depth or time. The real power of Quest is its ability to put isolated data in context where they can be compared to other data in the same area. Faults in the data then become obvious. It can also deal with data in isolation by looking for glitches, which for us are often temperature spikes, interpolating across it and then flagging them."
Historically, ocean temperature profiles have been collected since the 1800s with more than seven million gathered.
"This data is generally reliable but as much as 20 percent of it can contain bad values which distort results if it is used without rigorous checks. This led to the development of Quest by Neil Smith at BMRC (the Australian Bureau of Meteorology Research Centre) in collaboration with scientists at CSIRO as a quality control and analysis application," Thresher said. "For example, if a temperature reading is out by more than 0.1 of a degree in some areas of the oceans, the probability is high that there is an error in the data. Statistical screening of data can then be done and these outliers examined."
Thresher expects the bulk of the development work to be completed over the next two months and will allow a further six months for the "bells and whistles" before it is released.
"Quest is good at picking up outliers and the process could be applied to business analytics with some customization," she said. "There is international interest in this system by others involved in the quality control of data.”
A platform independent Quest will allow Jafoos to migrate to commodity hardware running Linux for its data analysis work.
"We are working towards the netCDF binary data format but Quest can also pull in other data files and analyze them," Thresher said.