Whether they're performing an analysis of an attack or observing the network traffic for large corporations, security professionals deal with a lot of data. Charged with protecting the electronic crown jewels, the task of sorting through the mountains of recorded information can seem daunting at times, but help appears to be on the horizon.

New technologies, demonstrated at the Black Hat Briefings conference in Las Vegas, can parse large and complex logs or data files and generate visual information based on those logs or data that may be able to help security professionals find the proverbial needle in the haystack.

Network traffic can be notoriously difficult to parse, and as the size of the network increases, the sheer volume of data that needs to be evaluated can be simply too much for a human mind to handle. To that end, a wide range of products -- from intrusion detection and intrusion prevention systems to firewalls that respond in real time to denial-of-service attacks -- serve the needs of IT workers responsible for maintaining the security of companies, governments, or other organizations.

However, there has never been a good way for the people who operate these technologies to step back and see the problem from 30,000 feet. That may be changing.

A "galactic" view of Internet traffic

One example of emerging security visualization tools at Black Hat was Lookingglass' ScoutVision, which offers a view of virtually the entire Internet. The Flash-based program's widest-angle view looks like a galaxy, with each of the 65,000 "stars" orbiting a galactic center representing chunks of IP address space called "autonomous systems" -- essentially the IP ranges of entire ISPs, as reported by the ISPs themselves using the BGP protocol.

Given a pair of nodes, the system can show all the possible routing paths for data that moves from one node to another, giving network administrators a quick visual representation of, for example, critical infrastructure choke points, which might need additional protection.

Users of ScoutVision set up watch lists of autonomous systems they want to monitor, and the program can alert them if it detects anomalous traffic, such as the massive amounts of SMTP traffic generated by a spambot. If the system detects an IP range on which there might be botnet machines, it highlights that autonomous system and provides additional data about the alert, including information about the precise IP address that triggered the alert.

Peering inside your binary files

Also at Black Hat, two security experts from West Point -- research programmer Erik Dean and assistant professor Greg Conti -- demonstrated two tools they've created that rapidly render the contents of binary data files into pixels in a resizable field. The tools, called VisBin and DanglyBytes, take the binary data from any file, manipulate the data mathematically, and render that data as a graphical image.

On a small scale, you can look at these kinds of files in a hex editor, Conti says, but "by creating graphical displays, you can show 900 times more information in the same amount of screen space."

Even though the images may look like television static, Conti says with experience and practice, researchers may be able to discern certain kinds of structures within binary files by scrutinizing repeating patterns that emerge in the visualizations. With the high-level view, a researcher can identify regions of the file where interesting information may be located and drill down to that location.

Although the DanglyBytes program is still in a very early beta stage, it's still able to render a number of different kinds of images, providing a surprising amount of information. In his demonstration, Conti browsed through a memory dump Windows created when his copy of the Mozilla Firefox browser crashed. Buried within the memory dump were copies of graphics that had been loaded in the browser when it crashed, including the Google and Wikipedia logos. "Every time I've done a visualization, I've been surprised with the result," Conti says. "I've found things I couldn't anticipate."