Growing pains

As pressure mounts on many industries to remain operational 24 hours a day, seven days a week, scalability is becoming a key issue. It can loosely be defined as the ease with which a system or component can be modified to fit a problem area, but while defining the problem may be relatively easy, solving it is another story.

Scalability should not be confused with future-proofing. Scalability is generally about increasing the work or storage capacity of an existing system, while future-proofing reflects changes, such as next-generation technology, rather than load.

While the ability to grow is important to any business, scalability varies in importance from critical -- for areas such as banking and finance and Internet related industries -- to something that is ‘nice to have' for other industries.

For example, when industries need customised solutions to meet the demands of their increasing specialisation, executives find these solutions may come at a cost that is more than just the capital outlay.

While industry-specific applications may be crucial to a business, they often reduce scalability and adversely affect system availability because they are not flexible enough to cater to different sizes of operations. The market for the applications is often small and there is usually little choice. This combination of small market and lack of competition makes it uneconomical for developers to spend the millions of dollars, to test them over a wide range of systems, which are available to developers of major shrink-wrap products. As a result, users are faced with the dilemma of an application that runs fine at a set level of operation but when the operation is scaled up by adding additional servers it collapses.

The only choice the user is left with is to give up any idea of increasing capacity, or a costly duplication of the original system.

The hospitality industry is one where industry specific applications are essential but have not always done as good a job as many systems managers would have liked.

Ben Wrigley, IS manager for Sydney's Inter Continental Hotel, believes scalability has been adversely affected by the custom applications. While he has no problems when there has been a need to scale up server capacity or telecommunications systems, the custom software has prevented him from upgrading the booking system to allow hotel staff to book and confirm online rooms at other hotels within the group.

The application is old and has reliability problems - a common occurence with some industry-specific solutions, where updates and upgrades are not always as frequent as they are needed - but Wrigley says that because of its age most of the reliability problems are now known, so down time can be reduced.

However, his troubles may soon be over, a new system is on the way that he is confident will address most of the reliability issues. It be customisable to the Inter Continental's requirements and should be available early next year.

Antonio Padihla faces a much larger scalability issue. Padihla is Queensland Health's data centre technologies manager and has the task of consolidating hundreds of computer systems across the state in an exercise that will take three years.

"We define both storage and processor capacity as a commodity, and scalability to me is being able to add more capacity on demand without having to change the architecture of the base framework of what I have already.

"We are attacking the whole IT infrastructure in Queensland Health so it can be scalable on demand, every time, all of the time," he says.

"We need to be able to grow on demand without interrupting the production environment and without having to reconfigure just because we have added capacity."

That's a massive task considering Padihla is looking after 269 different computer rooms and there is no uniformity among them.

"We are implementing a project called the data centre consolidation program (DCCP) under which we are reviewing the architecture of all of the sites; the aim is to reduce the number and ensure that they are all operating on the same architecture in such a way that we can have the same operations right across it.

"That way capacity will become a commodity and I will be able to increase it without disrupting the operations. It spells a complete change in the way we have done things in the past."

Padihla wants all the sites to become one large virtual data centre that can be managed in a domain that can be relocated by configuring the management layer in such a way that entire areas can look after themselves.

"We will be able to vary the sizes of these areas to encompass more or fewer sites as required. They will be uniform so that one site can support another. The aim is to get a few sites together where each has one or two people and create one virtual site that has 10 or 12 technical people who can be rostered to provide 24x7 support.

In parallel with the DCCP there is another project called SDN (service delivery network) that will work with Padihla's team to define the major sites and provide reliable and sufficient bandwidth for each of them.

Later this year the concept will be put to the test with the implementation of six sites across the state to test the architecture.

Queensland Health is working on formulating internal service level agreements and developing the tools and methodology needed to measure capacity and create performance indicators for all the capacity the system will have.

"Downtime is a critical issue but it varies from system to system. We will have some sites that operate normal business hours while others must be available 24x7 at 99.999 per cent availability. They all share the same infrastructure, but just have different levels of infrastructure according to their availability requirements," Padihla says.

While Queensland Health is still in the early stages of consolidation and developing a state-wide scalable architecture, the banking and finance industry has been tackling the problem for decades.

Scalability is a critical issue for banks that cope with extremely high volumes of transactions, and during peak periods can be handling hundreds per second.

Alok Chakravarti, KPMG banking and finance analyst, says scalability in the banking industry has changed.

"In the old days there used to be fairly big jumps when you went from one machine and system to another. Nowadays you have a much smoother passage because banks can implement increased capacity in smaller increments as their volume increases.

"Banking and finance has traditionally been at the forefront of a lot of technology, much more so than most other industries in general. They spend up to 20 per cent of their net operating expenditure on IT compared to other industries, which might average five to six per cent. As a result the banks have built large and robust systems over the years.

The banks no longer have to worry so much about developing their own applications in the areas of enterprise resource planning, customer relationship management and human resources applications as many of these shrink-wrapped products are now cross-industry applications and have high availability.

"One of the big challenges for banks as they reduce the number of physical branches and migrate customers towards electronic banking," Chakravarti says, "is to assure them that there will be no drop in the level of service."

He says security and public reassurance are key factors and that makes scalability very important. One of the significant things that has been found with electronic banking is that when customers rely solely on physical branches they might visit the bank once a week to do all their banking. But once they have access to electronic channels the level of transactions increases by five to six times.

Chakravarti says banks are outsourcing their IT requirements and those that have done so are demanding stringent levels of service agreements as part and parcel of outsourcing agreements.

"These can be quite complex, not only because of the level of service required but because they are finding there are a lot of previously unidentified services that are being provided."

Adelaide-based Hostworks is an outsourcing organisation of a different type, but has similar scalability issues. The company hosts some Australia's busiest Web sites including MSN and Ticketek.

Few senior executives know more about scalability issues than Hostworks managing director Marty Gauvin.

"Scalability for us means a given application can continue to receive additional load and perform effectively with no worse than a linear increase in cost," he says. "Future-proofing is a bit different because it reflects changes in technology whereas scalability reflects only an increase in load. If more people hit a site we are hosting that is a scalability issue, but if there is an enhancing of the technology and the hardware no longer copes, that becomes a future-proofing problem."

Gauvin says the three things people most often fail to get right in their implementation on the Internet are availability, scalability and content distribution or publishing.

"In terms of scalability there are two areas that are particularly big problems. One is the different elements of a platform which might be a database from Microsoft or an operating system from another vendor, that don't allow us to scale in that linear fashion. We might have to put in three servers to get double the performance.

"The other problem is that [the elements] might let us scale but in doing so the work load increases. For example going from four servers to five might increase the workload by 50 per cent so you have to shuffle things around manually a lot more.

"But the biggest problem is that people who are developing custom applications are still very much in a cottage industry environment. This means when a developer puts a custom solution together it might be a wonderful application, but their understanding about how it behaves is typically restricted to how it behaves on their laptop. When you are getting thousands or tens of thousands of people accessing it at the same time, it behaves differently.

"We have had one particularly high-profile application where the customer said just add another server, but when we did it fell over. It would only run on one server, so that was the ultimate failure to scale."

According to Gauvin there is often a lot to be said in favour of shrink-wrap applications.

Developers like SAP or Great Plains have spent a lot of money establishing an application's effectiveness to a particular range of businesses.

"There is a clearly understood process for those kind of applications to end up in the sweet spot, which does not necessarily mean that they are scalable but it does mean they work in the environment they are placed in."

Because Hostworks and its customers are unable to aggregate the volume of traffic their customers are going to do, it over-builds its systems.

"We over-build to very high levels," Gauvin says. "For example our connectivity to the Internet is handled by a primary network provider and a back-up network provider.

Our build rules call for us to have double our highest-ever peak on our primary provider and the same on our backup. So, we have four times our highest-ever peak in terms of available capacity which is one of the ways we are both handling scalability in terms of ensuring we are ahead of the curve and also managing risk of failure - which falls more into the availability side of the equation."

"We don't have the same problems we used to in terms of infrastructure and bandwidth. We are still pushing what the carriers can deliver to us, but over time we have developed a situation and a level of contact with all of our suppliers that means we can meet our requirements quite easily.

"Certainly if we were coming in off the street and saying we need another 100Mbit of pipe or another palette of servers or something like that we would have all sorts of problems.

"But vendor relationships, particularly with server and network vendors and telcos, have been the key to ensuring that we have what we need at an appropriate time," Gauvin says.

While only a handful of companies will ever require the level of scalability of Hostworks, they could do worse than to take heed of the way the company has tackled the issue.

After all scalability is the ability to grow.

Join the newsletter!

Error: Please check your email address.

More about Great PlainsHostworksKPMGMicrosoftMSNQueensland HealthSAP Australia

Show Comments