Breaking the Gigabit Ethernet bot

For network managers who've outfitted their servers with Gigabit Ethernet network interface cards in recent years, the results definitely have been of the "glass half-full" variety. While Gigabit Ethernet NICs have allowed servers to deliver more than predecessor Fast Ethernet, other system bottlenecks usually combined to keep overall throughput half, or less, of Gigabit Ethernet's potential maximum. Will Remote Direct Memory Access-based NICs change all this?

Vendor efforts to push the benefits of Gigabit Ethernet notwithstanding, the fact remains that while it is no big deal for Layer 2/3 infrastructure to run at wire speed, actual end-to-end communication is another matter. In short, going up the stack almost always slows you down. We have studied this issue for years focusing on determining the maximum throughput achievable between a pair of high-end machines running IxChariot - a standard network benchmarking test tool.

Even using machines outfitted with high-performance, 64-bit, 66-MHz PCI bus architecture, throughput for the most highly optimized bidirectional "file transfer" application topped out at around 750Mbit/sec - out of a possible 2Gbit/sec (1Gbit/sec each way).

So while this is far better than Fast Ethernet could offer, it is less than 50 percent of the theoretical maximum - and, worse, the test application deliberately only simulates file transfer (to isolate network performance). Because no data actually is read from or written to disk, performance with real applications likely would be worse.

Last year, we extended our study to benchmark the effective throughput when running actual applications but the results were still poor. When run again using Gigabit Ethernet, the results always went up - but marginally. The best results observed never broke 115Mbit/sec - or 11.5 percent of the theoretical maximum for unidirectional traffic.

Granted, things might be different in "server-to-server" applications, but my experience has shown that too many programmers seem to be blissfully unaware of how to write code that can take advantage of the underlying transport.

RDMA optimizes the process at the bottom of the stack. It is more efficient, reduces latency and offloads the server CPU. But will the presence of massive bottlenecks higher up in the stack make all of that irrelevant? The answer to that question might make all the difference in the world to vendors of RDMA products.

Kevin Tolly is president of strategic consulting and independent testing company The Tolly Group

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

More about Kevin TollySECTolly Group

Show Comments