It is the ultimate in bursty traffic: more than 100 servers containing tens of thousands of Web sites doing a daily backup to a storage array.
It used to take up to 16 hours each day for the process to trickle through the Fast Ethernet pipes at HiWay Technologies, a Web site hosting company. With plans to expand to up to 2000 servers, the provider needed something faster.
The company had an idea: build a separate infrastructure, identical to the existing one, dedicated to performing daily data backups. This plan would have required a second network card in every server to hook into a second Fast Ethernet backbone. HiWay also considered installing Fibre Channel, but that also would have required a separate network.
Instead, HiWay decided to replace the Fast Ethernet backbone network with Gigabit Ethernet and send backup traffic over the same lines as other data. The company didn't have to install additional cards in its servers, and at $US120,000, this approach cost one-third as much as setting up a separate network, says David Hartman, manager of network systems at HiWay.
Last June, the company installed a Gigabit Ethernet backbone anchored by a pair of Foundry Networks' BigIron 4000 switches. A handful of Foundry FastIron switches link to the backbone and connect the servers through their 24 Fast Ethernet ports. Because these connections don't use Gigabit Ethernet, HiWay didn't have to upgrade the cards in the servers.
It now takes four to six hours to back up the company's 125 servers. That's still too long, Hartman says, but the bottleneck isn't the network.
The problem is the server that manages backups, a Silicon Graphics Origin 2000 machine connected to the backbone via a Gigabit Ethernet card. "We got rid of the network problem and pushed it to the CPU on the Origin 2000," Hartman says.
The server moves at a speed of only about 300Mbits per second, and Hartman is looking for Silicon Graphics to get that rate increased to about 750Mbit/sec by tweaking the network card and improving the speed of its server. Hartman's goal is to back up all the servers in just a couple of hours.
Backups are really the only reason HiWay chose Gigabit Ethernet. Without the bursts caused by the high-volume backups, less than 10 per cent of the backbone is used, Hartman says. Because the company hosts Web sites, most of the traffic is outbound to the Internet.
"The traffic in and out of the Internet doesn't warrant Gigabit Ethernet," Hartman says. At peak times, the traffic level reaches only about 70Mbit/sec through HiWay's three T-3 lines to ISPs. But installing Gigabit Ethernet gives the company some headroom for at least the next three years, Hartman estimates.
Right now, HiWay doesn't use any of the Layer 3 functions of its BigIron switches. But one reason the company chose Foundry equipment is that it supports Open Shortest Path First (OSPF) routing. OSPF is a routing protocol that can adapt quickly to network changes.
Today, HiWay manually sets up static routes to its servers via the Cisco 7505 routers it uses to connect to the Internet. But that will change. Using OSPF, the servers will be automatically discovered, reducing the administrative burden of manually defining them, Hartman says. This will be critical as the company adds hundreds of new servers.
HiWay also plans to use Virtual Router Redundancy Protocol, a standard that defines how a backup router can automatically take over when another router fails. This way, if HiWay loses one of its two Layer 3 switches, the second can pick up the slack.
But the company hasn't had much trouble with outages. In fact, of the switches it uses, only one port on one of the switches has failed.
"That's unbelievable as far as I'm concerned," Hartman says.