Special version operating systems: secure but with a catch

Trusted operating systems are special versions of off-the-shelf operating systems, such as Windows NT and Unix that are enhanced to be more secure. Charles Kalko is a big fan of trusted operating systems. With PitBull from Argus Systems, he's used them to lock down vital functions that run his business-to-business barter site. The result: tighter security and more stable systems because fewer information technology administrators can make ill-advised tweaks to them.

But trusted operating systems are "nuclear bombs", says Kalko, a senior security engineer at Bigvine.com in the US. "They drop in and they solve a lot of problems very quickly, and they also create some of their own. If you don't know what you're doing, it could make your life miserable." Which means that IT managers should use them only when the benefit is worth the cost in training and management time.

An IT manager might use a trusted version of Windows NT on a Web server that contains or is linked to sensitive corporate information. But beware: trusted operating systems are usually harder to learn and administer than standard versions.

For example, because a trusted operating system can seal applications into unbreakable "compartments", one system administrator might think an application has crashed when in fact he just isn't authorised to monitor it. And because they split administrative power among many people, support staff need to coordinate more than they have in the past. "I scratch my head every day," says Kalko, trying to figure out, "What's going on here?"

New world order

What's going on is a fundamental change in how companies protect their applications and data. Today's Web economy demands that companies keep applications deep within their corporate infrastructure secure from hackers while keeping those same applications available to customers. Competition also requires companies to bring e-commerce systems online quickly, even if some components have known security bugs.

"You have to know which systems are critical for the business," says Chuck Ryan, director of information security at Molex, an electronics manufacturer also in the US. "You can't secure everything today."

Operating systems, especially on servers, can be weak points because of the fundamental role they play controlling basic functions such as how data is organised into files, written to disks or displayed on-screen.

Any off-the-shelf operating system can be made more secure, or "hardened", with simple procedures such as changing the administrator's password from the easy-to-guess "password" or turning off connections to the Web when they're not being used. But these common-sense fixes can be time consuming and may not protect a critical server from a determined hacker.

A truly trusted operating system is born, built from the ground up with security in mind. IT managers should look for the following three things in a trusted operating system, says Paul McNabb, chief technology officer at Argus Systems:

* A mandatory access-control policy. Consider the simple matter of creating and sharing a file - just fine if you're a legitimate user, possibly deadly if you're a hacker. "If you get into NT or Unix, the OS is not going to tell you if you can e-mail or share" that file, says McNabb. But using a mandatory access-control policy such as the one in PitBull, "you can configure the system in advance to say a . . . user can never get access to, or give away access to, certain resources", such as a file.

* An administration and privilege capability, which an administrator can use to control or eliminate the ability of a user or application that manages the system, or part of the system. "On a trusted OS, you can set up a program which does not have the capability to ever administer the system, even if that program should somehow be totally controlled by an attacker," McNabb says. This prevents a hacker who enters a system through one application from, say, disabling the password that protects other applications.

* Evaluation by an independent laboratory.

By these criteria, most commonly used operating systems such as Microsoft's Windows NT and Windows 2000, as well as the various flavours of Unix, aren't trusted systems, although Windows 2000 took an important step forward with its "system file protection", which safeguards some crucial components.

Trusted operating systems from major vendors such as Sun Microsystems and Hewlett-Packard have been around for a long time, but they have had a poor reputation for being hard to manage, lagging behind their commercial counterparts in key features. They were also incompatible with applications that their less-secure counterparts could run, says John Pescatore, an analyst at Gartner.

They were largely confined to high-risk environments in organisations such as banks and governments that could afford the staff to manage them.

Newer versions of those tools, such as HP's Virtual Vault (a secure version of HP-UX) and PitBull (which enhances the security features of Sun's Solaris, IBM's AIX and Windows NT) are easier to use, says Pescatore, but are still more expensive than their off-the-shelf counterparts. Still, the need for cost-effective and trusted operating systems is growing as more corporate systems are linked to the outside world.

Not only do operating systems ship with too many vulnerabilities, Ryan says, but many applications also add security holes as they install themselves.

Customers are "finding hundreds, if not thousands" of vulnerabilities, he says, ranging from weak password protection to user accounts or file structures that are "wide open" to hackers.

Compartmentalising security

Most trusted operating systems split the services they offer (such as file, print or network access) into compartments, or "sandboxes", and allow only certain end users, administrators or applications into those areas.

To be sure only genuine administrators can make such changes, trusted operating systems may require administrators to authenticate themselves using both a password and a secure ID card, and to enter the system only from certain host machines or network addresses, McNabb says.

Limiting the ability to make changes helped limit what Kalko calls "system drift" - undocumented changes to system configurations that not only open security holes but also make the systems less stable.

But creating these multiple levels of control can be confusing. Splintering the power to administer the system and to access the root directory (which allows access to all other directories and files) required 10 days of training for each of the 10 people on Kalko's administrative staff.

"It's not your standard systems administrator view where he can do anything he wants," Kalko says. "It [requires] a lot of communication within the team on who can do what, when, how."

Pulling the trigger

QSecure from Qiave Technologies (recently acquired by WatchGuard Technologies) locks down vulnerable portions of servers while they are in operational mode and provides a console for managing security across the network. In operational mode, even an authorised system administrator can't take actions that would compromise the system, founder and CEO Jack Danahy says.

QSecure also uses a "239-bit elliptical curve" encryption to transmit requests to the operating system kernel. "Every time you want to access one of your files on the file system, on your own box, first you have to re-authenticate yourself into the file system," Danahy says. As for ease of use, he claims a basic installation for an NT server requires only "five mouse clicks [and to] type your password twice".

Along a similar line, the current version of HP's Virtual Vault divides operating system functions into only four compartments "rather than separate every process into different compartments, [making] it hard to use", says Gary Sevounts, director of marketing, products and services at HP's Internet security division.

Sevounts says HP is planning an even easier-to-use product called Web Proxy that will have fewer configuration options than the current version but will serve as a secure front end to many popular Web servers.

It was such ease-of-use features that were most important for several IT managers.

"Since we're global, we need to be able to administer the software, potentially, from a centralised place," Ryan says. He says he also wants reports that tell him which vulnerabilities are most important, not just a list of '500 things wrong with the system'. "You can't give that back to the support people and say 'Fix this'."

Finally, Ryan says he wants tools that work across NT, Unix and perhaps even NetWare without needing specialised staff to monitor each platform.

Carl Tianen, director of global IT security at oil-services company Halliburton, says he was nervous about the cost of supporting a trusted operating system.

"Look at Windows NT and the effort required to administer an NT system. You start adding layers on top of that, and it could become very difficult," he says.

For such reasons, Pescatore suggests using secure operating systems mainly on servers that conduct financial transactions over the Web, and then only if a corporate security group is available to help system administrators support them.

Trusted operating systems become crucial, McNabb says, "when you have different types of people, different classes of users, on the same system, or you have different classes of networks attached to the same machine."

McNabb says examples include servers that are linked to both the Web and to internal systems, systems administering public-key infrastructure encryption systems and servers running firewalls.

"On a front-end Web server, they're pretty crucial," Kalko says. "I wouldn't use them anywhere else."

Trusted operating system products and pricingArgus Systems www.argussystems.comProduct: Trusted operating systems, which are enhancements to Sun Solaris, IBM's AIX and Linux.

Price: Runs from $US5000 for a trusted operating system running on a single-processor Web server to nearly $50,000 for an enterprise-level implementation.

Computer Associates


Product: eTrust Access Control, which can be used to harden Windows NT and various Unix variants. It can also control access to files, operation of critical application processes and access to network services.

Price: Servers start at $US4000.


www.hp.com/security/products/virtualvault and www.hp.com/security/products/webenforcer

Products: Virtual Vault (trusted version of HP-UX), which runs only on HP hardware, and HP Praesidium WebEnforcer, which is a tool for continuously monitoring and repairing Windows NT security vulnerabilities.

Price: Virtual Vault starts at $US15,000; WebEnforcer costs $3000 per server.

Qiave Technologies

(recently acquired by WatchGuard Technologies)www.qiave.comProducts: QSecure Enterprise Suite for Windows NT, Windows 2000 and Sun Solaris blocks any changes while the operating system is in operational mode; changes while the system is in administrative mode are allowed only after an exhaustive authentication process.

Pricing: Servers start at $US1295.

Hacker attacks on nontrusted vs trusted operating systemsA. In an attack against a conventional operating system, the hacker steals, guesses or decodes the administrator's password.

B. Posing as the administrator, the hacker is free to create, delete or e-mail files or directories and to open any application on the server to more attacks.

A. During an attack on a trusted operating system, the hacker steals, guesses or decodes the administrator's password.

B. But despite appearing to be the system administrator, the hacker can't tinker with operating system features that have been locked down during operation.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

More about Access OneArgusCA TechnologiesGartnerHalliburton AustraliaHewlett-Packard AustraliaIBM AustraliaMicrosoftMolexNT SecurityQiave TechnologiesSun MicrosystemsWatchguardWatchguard Technologies

Show Comments