Network and systems management technologies have come a long way, but networks have come even further - making it difficult for network teams to quickly diagnose performance problems.

Last week at Interop, I attended several sessions hoping to learn about innovative ways to tackle the management challenges introduced with technologies such as virtualization, VoIP and wireless.

Unfortunately for me, I didn't hear a lot of new techniques to take on the challenges network managers face today and will face going forward. That's when I ran into Burton Group senior analyst Eric Siegel, who shared my sentiment as well as his take on how network managers can tackle the performance problems bound to happen on today's more complex networks.

In yet-to-be-released research, Siegel details how companies can use the technology they already have in-house and develop a team of expert diagnosticians with staff on hand to solve the 20 or so most confounding network and performance management issues. Talking with Siegel reminded me of episodes of House in which the head doctor and his group work to discover the illness causing the symptoms of their patient. In network terms, it would be more about the application, server, network and other reps trying to figure out what part of the network is causing slow performance. Is it DNS? An application making unnecessary SQL queries? What network disease is causing the symptoms of slow response time?

And while most IT organizations would find it difficult to shave off time for their most experienced staffers to begin this group, Siegel says dedicating $100,000 in staff time to pulling three to five subject area experts off whatever they are working on to create a team of heavy-duty problem solvers could save companies millions of dollars in software costs. By measuring the known good state ahead of time and creating a baseline of meaningful metrics, this team would be able to more quickly spot anomalies. The project would pay for itself almost immediately when two days of poor performance can be cut down to 20 minutes because the diagnosticians did this work upfront.

"You probably already have all the tools out there collecting the data you need to really isolate problems, and look around because the skills are most likely in-house as well," Siegel says. "Instead of spending $1 million on new software, put $100,000 of staff hours and human capital toward building a framework of people for rapid problem-identification and diagnosis."

This insight hit home for me. Most IT shops have several management tools in place, yet many network managers don't feel they get the information they need from the tools. By designating an expert team or a few key diagnosticians to be on call to address the seemingly unsolvable cases could prevent investment on redundant software and free up the entire IT staff from taking part in problem solving swarms. This idea brings together the need to follow processes such as those laid out in the best practice framework ITIL with tapping in-house talent for skills that are hard to come by.

The idea of building a team of experts in-house and designing the process to effectively problem solve without causing a panic is the best lesson I learned at Interop. But please don't get me wrong, a lot more than that happened at the show this year. For instance, we shot tons of video discussing many of the hottest topics with vendors at Interop, and many exhibitors turned out to share their latest wares.