Homegrown high-performance computing

High-performance computing enters the reach of today's enterprise

Once the domain of monolithic, multimillion-dollar supercomputers from Cray and IBM, HPC (high-performance computing) is now firmly within reach of today's enterprise, thanks to the affordable computing power of clustered standards-based Linux and Microsoft servers running commodity Intel Xeon and AMD Opteron processors. Many early movers are in fact already capitalizing on in-house HPC, assembling and managing small-scale clusters on their own.

Yet building the hardware and software for an HPC environment remains a complex, highly specialized undertaking. As such, few organizations outside university engineering and research departments and specialized vertical markets such as oil and gas exploration, bioscience, and financial research have heeded the call. No longer borrowing time on others' massive HPC architectures, these pioneers, however, are fast proving the potential of small-scale, do-it-yourself clustering in enterprise settings. And as the case is made for few-node clusters, expect organizations beyond these niches to begin tapping the competitive edge of in-house HPC.

The four case studies assembled here illustrate the pain and complexity of building a successful HPC environment, including the sensitive hardware and software dependencies that affect performance and reliability, as well as the painstaking work that goes into parallelizing serial apps to work successfully in a clustered environment.

Worth noting is that, although specialized high-performance, low-latency interconnects such as Myrinet, InfiniBand, and Quadric are often touted as de-facto solutions for interprocess HPC communications, three of the four organizations profiled found commodity Gigabit Ethernet adequate for their purposes -- and much less expensive. One in fact took every measure possible to avoid message passing and cutting-edge interconnects in order to enhance reliability.

New to the HPC market, Microsoft Windows Compute Cluster Server 2003 proved appealing to two organizations looking to integrate their HPC cluster into an existing Microsoft environment. So far, results have been positive.

Finally, one organization found that delegating much of the hardware and software configuration to a specialized HPC hardware vendor/integrator made the whole process considerably easier.

More about: ACT, Alpha, AMD, Apache, BAE Systems, BEA, Cray, Dell, EMC, F5, Hewlett-Packard, HIS Limited, HP, IBM, Intel, Linux Networx, MATLAB, Microsoft, Morgan, Morgan Stanley, NASA, Oracle, PLUS, SGI, Speed, SuperMicro, Transportation, VIA

Comments

Post new comment

The content of this field is kept private and will not be shown publicly.
Users posting comments agree to the Computerworld comments policy.
Login or register to link comments to your user profile, or you may also post a comment without being logged in.
Related Whitepapers
Latest Stories
Community Comments
Whitepapers
All whitepapers
Sign up now to get free exclusive access to reports, research and invitation only events.
Featured Download
/downloads/product/145/microsoft-security-essentials/

Microsoft Security Essentials

Microsoft Security Essentials provides your home PC with real-time protection. It constantly uses the latest technology ensuring that you will always stay up to date ...

Computerworld newsletter

Join the most dedicated community for IT managers, leaders and professionals in Australia