Cloudstone: Multiplatform, Multilanguage Performance Measurement for Web 2'0

1
Cloudstone Multi-platform, Multi-language
Performance Measurement for Web 2.0

• Will Sobel, Shanti Subramanyam, Akara
  Sucharitakul, Jimmy Nguyen, Hubert Wong, Arthur
  Klepchukov, Sheetal Patil, Armando Fox, David
  Patterson
• UC Berkeley RAD Lab Sun Microsystems

2

• For better or worse, benchmarks shape a field.
  David Patterson
• Goal of this talk feedback on proposed
  methodology for how and what to measure for
  interactive apps in cloud-like environments

3
Toward Web 2.0 Cloud Computing Benchmarks Goals

• Goals
• reproducible consistent
• systematic exploration of design space of
  deployment optimization choices
• end-to-end, full stack (vs. microbenchmark)
• Approach
• realistic Web 2.0 app
• flexible, realistic workload generator
• suggested metrics reporting methodology

4
Olio social events app
5
Cloudstone

• 100 open source (Apache Foundation)
• Olio, Rails PHP versions of social events app
• Events, users, comments, tag clouds, AJAX
• representative sound practices on each stack
• Faban, Markov-based workload generator
• per-operation latency SLAs
• time-varying workloads, distributed operation
• instrumentation meet 90 or 99ile for all
  per-op SLAs over all 5-minute windows
• Big caveat no apples to apples comparison

6
Concurrent Users//Month

• Max concurrent users per dollar per month,
  while meeting 90 99 SLA response time
• Report 1, 12, 36 months
• captures cap-ex vs. op-ex, depreciation,
  long-term contracts, etc.
• Report log10(size of user base)
• avoids unfair advantage for very large sites

7
Example EC2

• 1 EC2 compute unit1GHz Opteron
• M1.XLarge 15GB RAM, 8 CU (4 cores)
• C1.XLarge 7 GB RAM, 20 CU (8 faster cores)
• Both cost 0.80 per instance-hour

8
EC2 users//month
9
EC2 max concurrent users
10
Integration with RAD Lab research

• Now SCADS running on Cloudstone
• Future ActiveRecord-like middleware layer to
  make SCADS largely transparent
• Future more comprehensive benchmarking of other
  clouds, platforms
• .NET/IronRuby, GlassFish/JRuby, GRails, etc.
• Encourage outside adoptionpublish EC2 AMIs

11
Keeping the pipeline full

• Previously announced plans to generate users of
  RAD Lab infrastructure
• student projects in RoR courses
• hire best students as TAs, RAs in RAD Lab
• Status
• 3 undergrads from previous iterations (Hubert
  Wong, Arthur Klepchukov, Jimmy Nguyen)
• Contributed to Cloudstone, SCADS, Director,
  running the course

12
SaaS Using RoR, Fall 2008Course Goals

• How does SaaS development, deployment, testing,
  operations differ from conventional SW process?
• How do rich frameworks like Rails apply big
  ideas in CS to achieve high productivity,
  reusability, extensibility?
• Major Vehicle design, develop deploy (on EC2)
  a SaaS app of your choice, in small teams

13
Selected project demos including LOC test LOC

• Flash-it (909 / 1163)
• Courses Snowman (1730 / 582)
• Hearsay (714 / 381)
• MedicalMooz (1499 / 2456)
• already has external customers
• WeJoinIn (from previous iteration)used to
  coordinate campus voter reg. drive

14
Comparison to other courses

• Design course for seniors
• vs. 2-unit P/NP seminar for freshmen/sophs
• Open-ended project
• vs. fill in blanks programming
• Focus on SaaS
• vs. Android, Java desktop apps, etc.
• Projects expected to work, and most do
• vs. working pieces but no artifact

15
Real SaaS Experience

• Introduce key SaaS concepts absent from
  shrinkwrap SW
• horizontal scaling load balancing
• consistency vs. availability
• service level agreement
• performance bottlenecks solutions (caching,
  asynchronous/AJAX UI, etc.)
• Deploy on EC2 (Amazon donation!)
• real scaling experiments, e.g. watch database tip
  over, then measure effect of add caching

16
Future

• Close content gap between current SW Eng.
  syllabus and this course
• Provide SCADS prototype for project use
• More use of PeepCode screencasts
• Challenge Open source Critical mass SaaS
  developers High productivity tools gt
  documentation cant keep up with technology
• Interview course alumni 5 years later
• Feedback from employers/industry
• RoR programming club

17
Summary

• Very positive response from students
• Show value of CS big ideas to achieve high
  productivity
• intellectually-significant content in
  deployment/ops/performance concerns in SaaS
  absent from current courses
• Managing complexity of large SW systems and how
  moving parts work together