Open-source tools keep Twitter up and running

To curb disruptions and scale up its service while keeping costs down, Twitter has drastically changed its core infrastructure and has adopted some open-source tools.

Twitter processes about 6,000 messages per second -- that's more than 500 million a day and about 3.5 billion a week. The company set a record earlier this year, with 143,000 messages per second during the airing of a movie in Japan, said Chris Aniszczyk, Twitter's head of open-source computing, at a recent conference in Europe.

When Twitter debuted in 2006, it used a monolithic Ruby on Rails application. That worked fine until 2008, when the microblogging service started to suffer a lot of "fail whales" -- the company's term for service disruptions.

The company decided to break the one application it had been using for everything into different services, Aniszczyk said. It also started using open-source tools such as Apache Mesos, a cluster manager; Netty, for creating high-performing protocol servers; and Scalding, for writing big data jobs.

And in another move, Twitter switched its core infrastructure to a Java virtual machine. The end result, said Aniszczyk, is a performance improvement and fewer disruptions.

This version of this story was originally published in Computerworld's print edition. It was adapted from an article that appeared earlier on

Read more about open source in Computerworld's Open Source Topic Center.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags open sourcemanagementNetworkingtwittersoftwareinfrastructure management

More about ApacheTopic

Show Comments