Some of the nation's leading financial services companies said this week that their IT executives met earlier this year for the first time to share current disaster recovery schemes and discuss future technology recovery strategies. And what they found was that they had a lot in common -- including headaches.

"To start with, I found out I'm not alone. All banks are struggling with this," said Todd Baumann, director of enterprise business continuity at Huntington Bancshares Ohio.

The Technology Recovery Project involved an information exchange between Huntington Bancshares, Bank of America, Wachovia, BankOne, Comerica, US Bancorp, J.P. Morgan Chase & Co and RBC Financial Group.

IBM, Microsoft and Veritas Software also participated in the project, which was organized by the Financial Services Technology Consortium (FSTC) in New York and took place between November 2003 and this past June.

The banks and bank holding companies looked at mainframe, open-systems and storage-networking environments.

Firms were asked what recovery strategies they currently use, what they consider to be best practices and what cost/risk trade-offs and regulations are driving their choice of strategy. They were also asked what investments in disaster recovery they're making over the next year.

While the banks were unwilling to share specific strategies publicly for security reasons, Baumann said a common issue was the need to find a data recovery methodology that is efficient and scalable and meets the needs of internal customers.

"We'd all like to have an open checkbook to do everything right now. We'd like to do it at a price tag our companies are willing to spend," Baumann said. "It's not so much getting the money. It's putting together the right business case to say, 'Here's why we should be doing this.'"

Virginia Garcia, an analyst at TowerGroup said the discussion is unique among financial services firms, which have traditionally been squeamish about sharing IT data, because they consider it a competitive advantage and because they don't want weaknesses exposed.

But with disaster recovery spending totaling between 1 percent and 2 percent of a financial firm's budget -- that's roughly US$2 billion a year for U.S. banks -- building business continuity through best practices is becoming a necessity.

"This spending is growing well into double digits -- an increase of 17 percent a year," Garcia said. "There's a very concerted effort on the banking industry to get a better handle on risk management spending at the operational level."

Charles Wollmen, managing executive director for the FSTC's business continuity standing committee, said there were several revelations from the project.

Banks, for example, said they are more tightly integrating recovery activities into IT systems design and incorporating them into day-to-day production practices. Companies are also moving toward more automation to reduce recovery times and eliminate human errors.

Garcia agreed with those findings, saying banks are quickly moving away from tape backups and choosing disk-to-disk mirroring of data over a geographically dispersed area.

Firms also said they are moving toward internal bunker data centers and away from third-party recovery service providers, such as SunGard Data Systems and IBM. The firms all said they want to spread their primary and backup data centers further apart to deal with regional power outages.

"Having the data centers five miles away is not going to be good enough," Wollmen said. "They'd like to ... have data centers farther apart and still be able to do the backups and not lose data. It's more the issue of you want your cake and eat it too."

One idea floated by the banks in conjunction with longer-distance replication of data was to share physical disaster recovery facilities in remote locations, which would spread out the cost of building and running hot sites. "But the other issue is that you have so much at stake in these large data centers. If sharing IT increases, then risk would be a concern. So it's a balancing act involving risk and cost," Wollmen said.

On October 6, the FSTC plans to meet again to determine further disaster recovery initiatives needed in the financial services industry.