FRAMINGHAM (05/05/2000) - Rail traffic in 19 states began to return to normal last week after being brought to a standstill by an MCI WorldCom Inc. outage caused by what the carrier said was routine maintenance.
The telecommunications outage, which lasted about six hours on April 28, stopped hundreds of trains on the tracks and disrupted rail traffic throughout Jacksonville, Florida-based CSX Transportation Inc.'s (CSXT) system. It was, according to one analyst, the kind of single-point-of-failure problem that comes with dependence on public broadband systems and one that could bring almost any big company to its knees.
CSXT spokeswoman Kathy Burns said late last week that she anticipated that the rail system would be operating normally by last Friday. Both passenger and freight delays were experienced much of the week as far north as Chicago, though she couldn't quantify the impact on rail customers.
"Some trains didn't move for six and a half hours," Burns said.
MCI WorldCom spokeswoman Linda Laughlin in Clinton, Mississippi, said the outage occurred during routine maintenance of a digital access cross-connect system (DACS) in Jacksonville. Technicians were updating addressing information in the equipment, which served CSXT and other customers, when the problem occurred. "We were aware we had a problem at 4:25 a.m. and immediately notified CSXT," Laughlin said.
The DACS includes computer-based devices that route voice and data traffic over public telecommunications networks, based on circuit addressing information carried by the signals, according to Steve Oliva, manager of transport planning at Sprint Corp. in Kansas City.
Telecommunications analyst David Willis at Meta Group Inc. in Stamford, Connecticut, said that what the MCI technicians were doing was technically pretty simple.
But he said he was troubled that "this fairly minor change affected CSXT for several days."
Noted Willis, "You would expect better change-control processes [from MCI WorldCom]. They have been bitten by change processes before in the 1999 [frame-relay] outages." Those outages knocked out communications at the Chicago Board of Trade last August.
Laughlin said she agreed that this did involve change processes but stressed that it wasn't a frame-relay issue. She said MCI passes on lessons learned from any incidents to its technicians worldwide.
All of CSXT's systems - including voice, data, signaling and radio communications - were affected by the outage.
In a joint phone conference with Computerworld late last week, CSXT and MCI officials said they were teaming up to analyze the reliability of CSXT's telecommunications system. The use of more access points or even additional carriers are options being considered, Burns and Laughlin said.
Willis said it's "going to get a lot scarier" for large corporations that rely on current public broadband infrastructures. In some instances, he said, the options for falling back on another system either don't exist or are limited to narrowband backups.
There is one solution, Willis said, and that's to use multiple carriers or carriers that can provide parallel infrastructures. But that can double the cost. Most large companies have been unwilling to pay up, he said.