Systems fail. Employees burn out. Vendors disappoint. But it's those tough times that teach you how to lead. Five Fortune 500 IT executives tell what they learned from the most challenging experiences of their careers.

1. Rob Carter, executive vice president and CIO, FedEx

Eighteen years ago, Carter, of transportation and business services provider FedEx, was working on his MBA when he was asked to direct the development of a large-scale billing system at a telecommunications company. The difficulty of the project was compounded by the need to keep pace with business requirements that changed while the system was being developed. There was also the challenge of adapting the legacy code, with all of the surprises it contained.

"There were huge pressures on us, with a lot of fault being placed on the IT team for dates being missed," he says. "I saw the other side of the equation, where we were trying to build something that was constantly changing."

Bolstering his team's confidence was key to meeting those challenges. "I spent a huge amount of time celebrating our successes," Carter says.

Carter also notes that the biggest leadership challenges are those you face when your personal life is in turmoil. He knows. In 1997, Carter, then FedEx's vice president of corporate systems development, had just landed in Hong Kong on a business trip when he got word that his 9-year-old son, Philip, had died unexpectedly.

After that shattering loss, "just showing up at work and figuring out how to continue to keep my head up and be a leader was the most difficult 'assignment' of my career," he recalls.

The heartache made him a more compassionate manager, he says. "We all have seasons in our lives when we need to be surrounded and even protected. That's something I understand, as a manager, better than ever before."

Lessons learned:

-- Acknowledge that "big bang" system development isn't a wise business strategy. It's better to break up your deliverables into increments.

-- Remember that legacy applications often contain solid components and code that is salvageable. "It needs to be re-engineered, it needs to have a new front end or user interface, it needs some rearchitecting, but you don't have to start with a clean sheet of paper," Carter says.

-- Give your team a chance to blow off steam during an arduous project. Carter arranged for his employees to indulge in fishing, go-cart racing and paintball fights.

-- Stand up for your team. "Fight to make sure that your team has the opportunity to lead a balanced life," he says.