Hackers of the world, unite! You have nothing to lose but the lousy stock firmware your routers shipped with.
Apart from smartphones, routers and wireless base stations are undoubtedly the most widely hacked and user-modded consumer devices. In many cases the benefits are major and concrete: a broader palette of features, better routing functions, tighter security, and the ability to configure details not normally allowed by the stock firmware (such as antenna output power).
The hard part is figuring out where to start. If you want to buy a router specifically to be modded, you might be best served by working backward. Start by looking at the available offerings, picking one of them based on the feature set, and selecting a suitable device from the hardware compatibility list for that offering.
In this piece we've rounded up five of the most common varieties of third-party operating systems, with emphasis on what they give you and who they're best for. Some of these are designed for embedded hardware or specific models of router only; some are designed as more hardware-agnostic solutions; and some are intended to serve as the backbone for x86-based appliances. To that end, we've presented them with the more embedded-oriented solutions first and the more generic-PC oriented solutions last.
DD-WRT is a popular router firmware choice not only with hobbyists and hackers, but router manufacturers as well. Buffalo, for instance, uses DD-WRT as the basis for many of its home and prosumer router offerings. The original product was created in 2005 for the Linksys WRT54G router, a device designed to accept Linux-based firmware, and the core software is available as a GPL offering. Note that there may be fairly major differences in implementation or presentation between the core version of DD-WRT and third-party, router-specific editions such as Buffalo's.
Supported hardware: DD-WRT supports Broadcom, ADM, Atheros, or Ralink chip sets, but be aware that not all devices using these chip sets are automatically compatible. Some may require unit-specific hackery to work; some may not work at all, period. The DD-WRT maintainers also keep a database of supported devices, along with a list in their wiki of both devices and features.
Features: DD-WRT provides a breadth of powerful features not normally found in consumer-grade routers, such as ChilliSpot (for creating commercial-grade Wi-Fi hotspots), the AnchorFree VPN system, and support for the NoCat wireless community network system. It also comes in a range of differently sized builds, from the 2MB "micro" build that supports only the most essential functions to the 8MB "mega" build that has, well, everything. This allows the firmware to be placed on devices of widely varying storage capacity.
Limitations: The core version of DD-WRT is updated very infrequently. If you want more frequent updates, you either must go with an interim beta or pick a manufacturer-supplied version with regular revisions.
Recommendation: DD-WRT is the best choice for most users. The fact that DD-WRT comes as a stock preload (albeit with mods) in many routers makes it easy to get your hands on a router with it both preloaded and tuned specifically to work with your hardware, as well as to keep it updated.
A commercial version of DD-WRT ships with many routers from Buffalo and other hardware makers. The unbranded version may vary in terms of presentation and feature set.
Originally devised as a replacement firmware for Broadcom-based routers, Tomato drew attention for its GUI, bandwidth-monitoring tools, and other nifty professional-level and tweak-able features.
Supported hardware: Hardware support is much the same as with DD-WRT, although pay close attention to exactly which builds are compatible with the particular hardware you're using.
Features: Many of the functions found in Tomato are also found in DD-WRT, such as sophisticated QoS controls, CLI access via telnet or SSH, Dnsmasq, and so on. That said, Tomato has been designed such that few configuration changes require rebooting, though that's a common complaint about any grade of router firmware, whether commercial or open source. There's also been a wealth of custom scripting developed by the Tomato community, such as redirecting the router's syslog to disk or another computer, backing up router settings, and much more.
Tomato has seeded a vast crop -- pun intended -- of spin-offs and offshoots, which deserve at least as much attention as the core project itself. Chief among them is Tomato USB, so named because it provides support for routers that have USB ports, thus allowing the mounting of removable media. Toastman compiles useful mods from a number of other Tomato firmware versions, such as an improved QoS module and IP traffic client monitoring tools. Teaman (also known by its Google Code project name, "tomato-sdhc-vlan") adds support for SDHC (Secure Digital High Capacity)/MMC media storage, 802.11Q VLAN tagging, and the experimental MultiSSID Web interface.
Limitations: The reason there are so many mods for Tomato is simple: The code for the original project hasn't been updated since 2010. To that end, any updates or new features come courtesy of the alternate builds described above. Updates for any one of these projects is also not guaranteed.
Also, because of the sheer number of Tomato forks, it can be difficult to pick the one that best fits your needs. Picking the right one for your hardware, though, shouldn't be too tough, thanks to the copious documentation of which devices fit which builds.
Recommendation: Tomato is best for moderately advanced users. Working with Tomato is on a par with dealing with DD-WRT, in terms of making sure you have the right hardware and following the flashing instructions to the letter. Tomato isn't used as a commercial pre-load, though, so don't expect to see it in an off-the-shelf router à la DD-WRT.
Tomato RAF is one of the many sub-breeds of the Tomato firmware. After development ceased on the original version, others have picked up the torch.
OpenWRT is a router firmware project that's like a full-blown Linux distribution for embedded systems. You can download the packages for a specific hardware configuration and build the code for that hardware using a supplied tool chain. This complicates the deployment process, but also provides enormous flexibility.
To save time, various prebuilt versions of OpenWRT are available for common hardware types and router platforms. This includes everything from generic x86-based systems to the Broadcom and Atheros chip sets used to power many open-firmware routers. The makers of OpenWRT recommend starting with an off-the-shelf version, then learning how to roll your own once you've found your footing.
Supported hardware: Lots. More than 50 hardware platforms and 10 CPU architectures are supported: everything from ARM mini-boards to full-blown x86-64 systems. They also have a buyer's guide for helping you choose proper hardware for your particular needs, in the event you're shopping for something specifically OpenWRT-compatible.
Features: In addition to broad hardware and platform support, OpenWRT includes support for the OLSR mesh networking protocol, which allows you to create mobile ad hoc networks out of multiple OpenWRT devices. Also, the software, once deployed, can be modified without reflashing the firmware. Packages can be added or removed as needed through a built-in package management system.
Various spin-offs of OpenWRT also exist, some with highly specific usage scenarios. The Cerowrt build, for instance, was created as part of the Bufferbloat project to address network bottlenecking issues in LANs and WANs. FreeWRT is even more developer-focused than the core OpenWRT builds, but has a handy Web-based image builder for those who want to create a FreeWRT firmware with a little guidance. And Gargoyle offers as one of its big features the ability to set bandwidth caps per host.
Limitations: The biggest strengths of OpenWRT are also its biggest limitations. It's best used by people who really, really know what they're doing. If you just want to replace your stock router firmware with something a little more current, steer clear.
Recommendation: OpenWRT is best suited for experts. This is the firmware for people who want as few limitations as possible on what they can do, are ambitious about using unusual hardware, and feel comfortable with the kind of tinkering that would normally go into rolling one's own personalized Linux distro.
Gargoyle is one of many breeds of OpenWRT, specifically offering special bandwidth-capping features. Like a miniature Linux distro, OpenWRT lends itself easily to this sort of respinning.
M0n0wall and PfSense
Among the other projects here, m0n0wall is closest in spirit to OpenWRT. It's a version of FreeBSD that works as a firewall or router, so it's much closer to a full-blown OS installation than a mere firmware layer.
Supported hardware: M0n0wall runs on embedded hardware systems with at least 64MB of RAM and 16MB of flash storage. It can also be run on commodity x86 PC hardware, and a high degree of compatibility is provided with common PC components thanks to the BSD driver library.
Features: All common router features are supported, including traffic-shaping and QoS tools, as well as features useful on high-end networks such as VLAN tagging and polling. Boot time is fast: 30 seconds tops to a full working system on flash-based hardware. Most useful is the sophisticated Web interface, which includes the ability to upgrade the firmware directly through a browser.
M0n0wall is also the basis for a number of spin-off builds. Most notable among them is PfSense, which is intended for full-blown PC-style systems rather than embedded hardware. If you plan to repurpose a PC as a router, start with pfsense instead of M0n0wall, as you'll have something built directly for PC hardware and you can keep the amount of tinkering to a minimum. Another version, M0n0wall-mod, adds new WAN options such as the ability to configure DHCP and PPTP over a separately enumerated WAN interface.
Limitations: One drawback of M0n0wall is its support for only a small number of wireless chip sets. That said, any 802.11 hardware based on the Atheros chip set should work.
Recommendation: Those repurposing old PC hardware as a firewall or router should check out pfsense. If you're looking to fashion an embedded hardware router into a multipurpose network device, M0n0wall is a good choice.
Based on FreeBSD, M0n0wall and spin-off PfSense (setup screen shown) are designed for both embedded hardware and full-blown x86 boxes.
Vyatta is a Linux-based network operating system available in both a core open source implementation and a commercial edition. The commercial edition can be had in the form of a software subscription or by purchasing a hardware appliance. Vyatta is commonly deployed as a small- or branch-office gateway, as a VPN concentrator, and as a bridge between data centers or between data centers and clouds.
Supported hardware: Like OpenWRT and M0n0wall/PfSense, Vyatta comes in incarnations that allow it to be used in stock 32-bit x86 PC hardware, so any such system can be transformed into a high-efficiency router, firewall, and network services box.
Features: You name it, Vyatta probably has it. Among the most recent additions as of the March 2012 build of the product are a GUI and dashboard that displays graphical statistics (in the commercial edition only), RFC-compliant VRRP, an enhanced connection tracking and logging subsystem, and stateful inspection firewall -- professional features all. Vyatta also comes built for drop-in use in many virtualization environments, with support for VMware vSphere 5 templates just added too.
Limitations: The single biggest limitation of Vyatta, vis-à-vis the other products in this roundup, is that it's designed entirely for x86 devices. And not just any old x86 device, either, but one with a fairly large amount of storage (1GB minimum), especially by embedded-device standards. In short, Vyatta is best for full-blown PCs. Also, while a 64-bit edition does exist, it's still considered experimental at this time and shouldn't be used for production systems.
Finally, a number of key features, including the Web interface, are only available in the commercial edition. The command-line interface is designed for network admins and may pose a hurdle to more casual users. A free 30-day trial of the commercial edition is available, though.
Recommendation: Vyatta is a business-oriented product with routing and security features beyond the needs of most small offices and home offices. That said, those building a network appliance using full-fledged x86-based PC hardware -- especially for larger environments -- will find everything they need in Vyatta.
Vyatta's Web-based configuration panel is one of the big assets for its commercial edition, but its open source core version preserves all the under-the-hood functionality for this x86-based network OS.