Liminal Existence

Scaling Twitter, the Talk.

Simon Willison linked to an interview with Alex Payne, one of my co-workers on Twitter. This caused a bit of a stir, so apparently there’s some interest in our experience scaling Twitter, and Rails.

We’ve been extremely happy with Rails, and make use of the multitude of helpers that it offers us - like any application on any stack, though, providing fast response times to a (rapidly) growing number of users is a challenge. The solutions are often tightly coupled to the application and its characteristics, and while scaling the most trafficked Rails site in the world, we’ve run into situations where existing solutions weren’t enough.

This process has led us to build a number of tools that help us deal with our load, and just as soon as we find some spare time, we’ll be releasing many of them. In the meantime, you can find out first what sorts of challenges we’ve encountered and solutions we’ve come up with at my talk at the SDForum Silicon Valley Ruby Conference next weekend (April 21-22nd).

I’ll be focusing on ActiveRecord and database optimization, caching, and of course, Messaging. I’ll also touch on some areas where we haven’t had great successes (yet), and hopefully someone from the audience will shout out that there’s some totally obvious and awesome thing that we haven’t thought of, and it’ll save us weeks of work (no, I’m serious. Does someone want to take bets?).