Where security belongs

A hacker, thief, or spy looking to break into (or just disrupt) one of your systems has a big bag of tools at his disposal. The more he knows about your operation, the more specific and sophisticated his intrusion techniques can become. The most damaging attacks exploit weaknesses in enterprise applications. The intruder's best ally is buggy or poorly written software, and unfortunately, almost all large applications contain exploitable bugs.

Usually, it traces back to a failure to check for exception conditions. A Web collaboration application might let users insert URL links into their postings but fail to check to make sure the link doesn't point back to a private file, directory, or device on the server. The name field on a customer response form might have room for 32 characters, yet the form will let the user enter additional characters beyond the limit. A server program that runs with elevated privileges, as many do, might not reduce those privileges before executing an external application or script. In these and similar cases, external hackers can trick an application into crashing, hanging, granting elevated privileges, or worst of all, executing uploaded code.

Some programming experts, particularly Java proponents, blame overly permissive programming languages. Native-compiled languages, especially C and C++, encourage the use of pointers. Pointers give programmers an easy and high-performance way to scan through data stored in memory, but most pointers don't know what they're pointing at. It's up to the programmer to test the validity of a pointer. If that test isn't done, the application can be tricked into listing or modifying arbitrary memory locations. Safety mechanisms are available in the form of data management libraries and tool add-ons (such as Compuware Corp.'s Bounds Checker). Many native compilers will automatically insert pointer validity checks if asked. But often, these safety measures go unused because they delay the project's deployment or slow down the executable.

Scripted and just-in-time compiled languages, including Java and Microsoft Corp.'s .Net languages, tightly control access to memory. In the cases of Java, .Net, and Visual Basic 6, the run-time environment is supposed to compensate for programmers' bad habits. Pointers don't exist (.Net allows pointers, but it marks the application as "unsafe") and memory management is taken completely out of the programmer's hands. Any holes found in the framework's run-time components are patched by the vendor, and all software written for that framework immediately becomes more secure. This only covers those exploits that do things no application should be permitted to do. If Sun and Microsoft lock too many doors in their frameworks, developers won't be able to deliver integrated enterprise applications.

The most promising solutions carry the concept of the virtual machine to the extreme: The whole computer is virtualized. None of the system's physical assets -- memory, disk, networking -- is directly accessible. Instead, a virtual machine process intercepts all application requests for resources. The VM can check requests for validity, write them into a log file, or transform malicious requests into harmless ones. If a DoS (denial of service) attack or exploit takes down a virtual machine, other VMs are unaffected. The hacked VM's complete running state can be frozen for later analysis. That post-mortem can identify both the intruder and the application weakness used to break in. In any case, the potential for damage is limited.

IBM makes consistent use of VMs in its mid- to large-scale servers. On Windows systems, VMWare extends this protection, and the 2.5 release of Linux will feature virtualized instances of Linux that run in nonprivileged user mode. Instead of trying to whip programmers to produce cleaner code or plug every potential hole in the run time, engineers should build exploit protection into operating systems.

The most promising approach to securing applications lies in building exploit protection into operating systems. A virtual machine process that checks all application requests for validity, and prevents direct access to system resources, can be used to identify and mitigate malicious requests.

Join the newsletter!

Error: Please check your email address.

More about CompuwareIBM AustraliaMicrosoftVMware Australia

Show Comments