Twitter Copes with Web2.0rhea

1
Twitter Copes with Web2.0rhea

• The Twits are catching up

Alex Carian ISM 158 May 6, 2010
2
From TheRegister.co.ukTwitter It's the end of
the sysadmin as we know it

• "This is a whole new world For the longest
  time, people ran large data systems on a kind of
  ad hoc basis. We're in a world now where so many
  people are depending on the real-time web. A
  system administrator is not just a system
  administrator anymore. 1

John Adams, Ops Engineer
3
The Real-Time Web?
You have to use analytics. You have to grab
data. You have to look at where a site is
trending and where things are going so you can
scale... If you don't start doing this work
early, if you don't start collecting this early,
you will fail. - John Adams, Twitter Ops Engineer
- Ad hoc solution designed for a specific
problem or task, cannot be adapted to other
purposes 3

• Initially, the core of Twitters code ran
• Ruby on Rails (application framework)
• Apache/Mongrel (web servers)

4
Twitters Aches and Pains
- Despite the benefits of the RoR environment,
Twitter experiences outages, most notably during
Steve Jobs Keynote address at Macworld 08 3

• A developer, in his spare time, ports a section
  of the website code into the Scala programming
  language
• Allows for object-oriented and functional
  programming paradigms
• Task that originally took 2.5 hrs now takes
• 20 seconds!

5
Functional Bits

• Twitter maintains Ruby on Rails for building
  user facing features and MySQL for database
  management1
• Now entrusts back end infrastructure to Unicorn,
  a RoR application server
• Grocery Store model
• Still open source
• Still based on Mongrel (the former app server)

6
Solipsism 2.0

• Twitter saw a 1358 increase in traffic in 2009
  (comScore)
• Utilizes a monitoring system known as Ganglia
• open source program grown out of UCB and NSF
  grants
• Tracks 15,000 points of site performance,
  including grid load, cluster load, of hosts
  up/down

7
TWITPOCALYPSE?!

• Many suspected Twitters service would come
  unwound after reaching the Unique Tweet ID limit
• (231 4,294,967,295)

The Fail Whale

• Twitters ops team mined site data to predict
  when the limit would be reached
• Allowed the team to implement necessary changes
  to avoid complications
• Current number of Tweets posted 13,475,000,000
  5

8
Another Word on Sysadmin 2.0
- Hard drive? Thats so 2009. Or 1959
"Another discovery that we made when trying to
increase the scale of Twitter was that disk is
the new tape," he said. "With any sort of social
networking operation - juggling followers,
sending mail, etc. - disk is extremely slow. 1
-John Adams

• Twitter now uses memcached
• Data and objects cached in RAM to reduce the
  number of times an external data source (such as
  a database or API) must be read. 4
• Dont overdo it.

9
Twitters Open Back End

• Company uses open sourced technology in most
  operations
• RoR, Unicorn, Ganglia,
• Developed several tools at home
• Kestrel, message queue server
• FlockDB, follower database
• Both now released as open source

Will Twitter maintain its open source track
record?
10
Sources Cited

• Metz, Cade. Twitter It's the end of the
  sysadmin as we know it. The Register, May 4,
  2010. lthttp//www.theregister.co.uk/2010/05/04/twi
  tter_and_the_end_of_the_sysadmin_as_we_know_it/gt
• Metz, Cade. Twitter jilts Ruby for Scala The
  Register, April 1, 2009. http//www.theregister.co
  .uk/2009/04/01/twitter_on_scala/
• Riley, Duncan. Twitter Fails Macworld Keynote
  Test TechCrunch, Jan 2008. lthttp//techcrunch.com
  /2008/01/15/twitter-fails-macworld-keynote-test/gt
• Ad hoc. Wikipedia
• memcached. Wikipedia
• Twitter Twitpocalypse Status. Accessed May 6,
  2010. lthttp//www.twitpocalypse.comgt

11
Sources Cited

• Metz, Cade. Twitter It's the end of the
  sysadmin as we know it. The Register, May 4,
  2010. lthttp//www.theregister.co.uk/2010/05/04/twi
  tter_and_the_end_of_the_sysadmin_as_we_know_it/gt
• Metz, Cade. Twitter jilts Ruby for Scala The
  Register, April 1, 2009. http//www.theregister.co
  .uk/2009/04/01/twitter_on_scala/
• Riley, Duncan. Twitter Fails Macworld Keynote
  Test TechCrunch, Jan 2008. lthttp//techcrunch.com
  /2008/01/15/twitter-fails-macworld-keynote-test/gt
• Ad hoc. Wikipedia
• memcached. Wikipedia
• Twitter Twitpocalypse Status. Accessed May 6,
  2010. lthttp//www.twitpocalypse.comgt

12
(No Transcript)