The astonishing throughput of 10GbE (10 Gigabit Ethernet) can solve many problems in your enterprise. You can use the same 10GbE switch (but with different optics) for local data centre use or for metropolitan-area distances. If you have the fibre available, then real-time off-site backup becomes feasible. Likewise, if you’re simply running out of backbone capacity, 10GbE can provide the bandwidth you need — and then some.
But as we discovered in our recent tests, there’s more to 10GbE than jaw-dropping bandwidth. In fact, because 10GbE switches provide so much switching capacity, performance differences among switches become relatively unimportant. Considerations such as management software, traffic management capabilities, and distance capabilities could play as big a role as raw performance in your buying decision.
Today, 10 Gigabit Ethernet has arrived and is available from several manufacturers, two of which bravely accepted InfoWorld’s test invitation. In general, we found the technology works well. We also found that there’s more than one way to build a 10GbE switch. We looked at two core switches from two manufacturers that take different approaches to delivering 10Gig, and we found both to be significantly useful to enterprise networks, but in different ways. Foundry Networks, long a powerhouse in enterprise networking, provided us with its workhorse FastIron 800, equipped to handle the extra capacity afforded by 10Gig. In the FastIron solution, the company combines a new throughput engine with a mature switch chassis and excellent management software.
Startup Force10 Networks brought a new platform designed specifically for 10Gig. Because of its fresh design, Force10’s E600 provides unquestioned leadership in overall throughput, with the capability of moving data at a full 10Gbps per port. But in other areas, such as manageability, we encountered rough edges.
Our experience with the Force10 and Foundry switches showed us that 10Gig really works in enterprise scenarios. But it will pay to look closely at what you need a switch to do. For example, Force10’s E600 seems best suited to service providers, given its power requirements, availability of DC operation, and emphasis on raw performance. Foundry’s FastIron 800 doesn’t keep pace with Force10’s wire-speed throughput, but its greater management capabilities clearly meet enterprise needs. However, to be really useful, the FastIron requires an all-Foundry network.
Putting 10Gig to the test
We assembled two simulated enterprise networks at the University of Hawaii’s Advanced Network Computing Laboratory (ANCL) on the university’s Manoa campus. This lab is designed for research and testing of enterprise products, and it provides the infrastructure required to operate a complete enterprise in the lab.
Our approach was to base all switch testing on a model of real enterprise-traffic types. Instead of simply pumping a single type of packet of a single size through the switch for each test, we fed a wide variety of packets containing real-life data payloads through the switches, both directly and through attached edge switches. This raised the level of complexity considerably, but the result was a more realistic test of enterprise capabilities.
A relative newcomer to the switching arena, Force10 Networks has focused on high performance, blasting onto the scene with a chassis designed from the ground up to make full use of 10Gig technology. Unlike Foundry’s FastIron 800, which tops out at 8Mbps per port, Force10’s E-Series 10Gb switches provides true 10Gbps line-rate performance.
Force10 gave us two of its E600 switch chassis to act as a core. Because Force10 has no edge-switch product of its own, the company brought Cisco 7200 series edge switches to the test, each configured with 24 10/100 ports and dual fibre 1Gbps uplinks. Force10 then protected its video traffic on the edge by assigning higher priority diffserv values to the packets in the video stream. These values were recognised by the core E600 chassis and protected accordingly on their way to the opposite edge.
The E600 chassis is a truly cutting-edge switch design, providing 1.2Tbps of nonblocking switch fabric (more than four times that of the Foundry FastIron 800), according to the company. (We were not able to verify switch capacities in our tests.) The box has room for seven line-card slots, four redundant AC power supplies, and an optional dual-redundant DC power module. Besides 1Gbps and 10Gbps port options, Force10 also provides WAN capability with SONET (Synchronous Optical Network) cards providing throughput between OC-3 and OC-192.
Regarding Force10’s power architecture, we very much liked being able to run off multiple 100volt, 15amp feeds. Not only is this the same feed provided by most rack-mount UPSes, but it also allows the E600 to be used for things such as cheap computing clusters without having to spend large sums of cash on 208 volt, 30 amp feeds. Additionally, the power subsystem is intelligent enough to automatically shut down the switch if it doesn’t have enough power. For even greater power flexibility, we’d suggest that Force10 add L21-20 and twist lock 110 volt, 20 amp plug assemblies.
The E600 is based on what Force10 calls its EtherScale architecture. This combines six custom ASICs, allowing the E600 to support as many as 168 ports of Gigabit Ethernet or 14 ports of 10Gb Ethernet per chassis, giving customers more than enough bandwidth bang for their buck.
This EtherScale ASIC-based architecture managed to deliver nonblocking, line-rate forwarding across all the E600’s ports, both 1Gbps or 10Gbps ports, even when we applied QoS requirements to protect a video stream being routed from one edge and across our core to the other edge.
As for the switch’s firmware, Force10 runs its own real-time operating system software, FTOS, allowing the E-Series to perform full Layer-2 switching and Layer-3 routing across all ports.
In our performance testing, Force10 had something to prove. Dinged in previous tests for an exceptionally long latency period, Force10 was eager to show that its newest firmware had the right stuff to remove the tarnish from its reputation. Indeed, our tests indicated a significant improvement. Whereas previous benchmarks had pegged this architecture’s latency at roughly 40 microseconds, our tests showed the company has managed to reduce latency by nearly 50 per cent to slightly more than 20 microseconds.
Although Force10’s throughput numbers led the way and its latency numbers were certainly improved, the company still has work to do. For one, even at 20 microseconds, the E600’s latency numbers could stand improvement. And Force10’s FTOS software remains command-line only. This interface is closely based on Cisco’s IOS system, which means a short learning curve for most switch-management professionals. If you want a graphical management interface for this switch, however, you will have to rely on third-party software.
For those who live in a graphically managed world, this may seem like a real problem. But we weren’t overly dismayed. For one thing, Force10’s product line really isn’t mature enough to warrant a dedicated graphical management system. The company produces only core switches, which means you’ll need edge products from third-party manufacturers in any case. In our test, these products came from Cisco, and the E600s worked nicely with the CiscoWorks software over SNMP.
Overall, Force10’s E600 product provides blazing 10Gbps speed in an impressive package only slightly marred by latency and management concerns. And these minor troubles will disappear with more mature firmware revisions. For network managers needing 10Gig performance, the E600 is a much more effective solution than aggregating an equivalent number of 1Gbps pipe.
Whereas Force10 offers a new-sprung product designed for 10Gb Ethernet from the ground up, Foundry Networks’ FastIron 800 is an existing switch chassis upgraded for 10Gig traffic. This upgrade has performance consequences, but it also leverages existing Foundry firmware and management software for a more polished total solution.
Foundry brought several products to our test, starting with two FastIron 800 switch chassis equipped with a combination of 10Gbps and 1Gbps ports, which we linked with two 10Gbps connections to form our core. The FastIron 800 is a powerful chassis, with a rated maximum switching capacity of 256Gbps and the capability of carrying a maximum of 14 10Gbps ports.
The Foundry 800’s maturity shows in its back-end design and support for robust management standards. The switch can carry redundant management modules or blade configurations designed for hot swapping through rapid fault detection. Its power modules can also be configured as redundant or load sharing with the ability to mix AC and DC feeds in the same chassis.
The FastIron supports an alphabet soup of management protocols but has especially nice support for Spanning Tree protocols, including Rapid Spanning Tree. It also provides Single Spanning Tree support for attached devices as well as VLAN Spanning Tree in order to support multiple spanning trees within a single system for VLAN load sharing.
The chassis is a fully functioning, Layer-3 routing switch with support for RIP (routing information protocol), OSPF (open shortest path first), integrated switch routing, policy-based routing, and NAT. The box also pays attention to VoIP-oriented users with support for bandwidth management, granular QoS management capabilities, and low latency numbers.
On the edge, Foundry installed two of its EdgeIron 24G switches. These 1U boxes carry 20 10/100/1000Mbps RJ-45 ports and four combo 1Gbps RJ-45/mini-GBIC (gigabit interface converter) slots. Designed specifically for edge duty within the Foundry product family, the EdgeIron supports as many as 255 active VLANs and has a complete Layer-2 feature set including all the usual goodies such as support for 802.1Q, CoS (Class of Service), 802.3x flow control, and all the FastIron 800’s Spanning Tree protocols.
Foundry’s EdgeIron 24G served as our starting point for implementing QoS. The EdgeIron was configured to protect the video stream with a port-based 802.1Q scheme that the FastIron 800 core switches automatically picked up and protected. All this was quickly and slickly accomplished through a wizard-like interface within Foundry’s IronView Network Manager software. A key difference between Foundry and Force10 is the maturity of Foundry’s network management software. Traditionally, this has been a weak point for Foundry, with the company relying largely on its Cisco IOS clone command line, as does Force10 today. But Foundry has since done an impressive amount of work on its Foundry Manager suite.
The central console is Network Object Manager, an intuitive, tree-based view of all Foundry devices on the network broken out by device, preset device categories, or user-defined device categories. From the Network Object Manager, administrators can drill down to specific devices with just a few mouse clicks or set a number of global parameters using device categories as guides.
A handy feature, the ability to define port groups allows network managers to define groups of ports from anywhere on the network. Network Object Manager then manages the group as a single unit, showing port characteristics, IP subnets, fault monitoring, and more.
Unfortunately, although Foundry Manager can discover non-Foundry devices via an SNMP-based discovery module, it cannot manage or monitor anything within Foundry Manager that hasn’t fallen from the Foundry product family tree. You also won’t find support for advanced routing protocol configuration such as BGP (Border Gateway Protocol) route maps, but this should be coming soon. In the meantime, it is an excellent interface for most day-to-day switch management tasks. For end-to-end Foundry users especially, Foundry Manager is a one-stop management shop. You’ll find an excellent VLAN manager that will configure, modify, and deploy VLAN settings for multiple switches from a single wizard-like interface. There’s similar functionality for access control lists, password settings, and software images, all revolving around the device groupings within the Network Object Manager.
The change manager retrieves the configurations of all devices in the Network Object Manager at regular, user-defined intervals. If it detects a change in the configuration, it stores it for future comparison and fast rollback in the event of a problem.
In testing, the Foundry devices ran smoothly after a few hiccups. We lost a number of ports on one of the FastIron 800’s 1Gbps port blades but were able to use the remaining ports to complete the test. We also noticed a difference between a switch designed for 1Gbps running 10Gbps ports and one that’s designed for 10Gig from the ground up.
Nonetheless, the FastIron 800 was not slow. Foundry rates its box at a 220Mpps (millions of packets per second) switch capacity when equipped with 10Gbps ports. Whereas the Force10 E600 was able to blast 10Gbps at wire rate, the FastIron 800 averaged just under 8Gbps of actual throughput. On the upside, the FastIron had only a fraction of the Force10 switch’s latency, between 5 microseconds and 6 microseconds on average.
Current Foundry users needn’t be put off by the FastIron 800’s slightly slower throughput. With enterprise requirements for 10Gbps yet to be determined, it’s unlikely that a full 10Gbps bandwidth path will be an application requirement for a while. The FastIron 800’s existing throughput capacity should be enough for quite some time, and its polished feature set and excellent manageability more than make up for its performance numbers.
There is no winner here, and there definitely is no loser. Both of these 10GbE core switches fulfil their promises. Although they’re based on the same technology, we found the Force10 and Foundry switches to be very different. The right switch for you depends on the demands of your enterprise and its applications. For that, you’ll need to decide what your backbone traffic is like, how much capacity growth you expect, and how skilled your network managers are.
If you already have a mostly Cisco-based network, your engineers are comfortable with IOS, and you’re moving a lot of data no farther than 10km (Force10’s distance limit), then Force10’s E600 is a good choice.
If you’ve got Foundry gear or a heterogeneous infrastructure and your staff requires more intuitive management capabilities, Foundry’s FastIron 800 is the better choice. Only Foundry could cover distances as far as 40km, meeting the requirements of MAN and WAN applications.
No matter how specific your needs may be, it’s nice to know that 10GbE technology is working well enough to meet them. w
When asked for details of Australian availability, Force10 said in an e-mail that it operates from the US and handles products from there. It added that the company has come a long way since earlier tests were performed, which “don’t make sense any more”. The E600 is a 7-slot system; each slot can take one of the following flavours: 2-port 10 Gigabit Ethernet card, 24-port 1 Gigabit Ethernet card with pluggable optics or 24-port Gigabit Ethernet base T card (copper) and it also supports Sonet cards, the e-mail said.