Cap'n, I'm givin' 'er all she's got

1
"Cap'n, I'm givin' 'er all she's got!"

• What to do when your servers are at their limit.

2
Scotty

• How many times did Scotty have to be a hero?
• He just needed bigger, more efficient engines!

3
Web server capacity

• Your web server is an engine.
• With heavy usage, performance degrades
• The server can stop responding altogether

4
Our Story

• Heres how we reached capacity
• and re-designed our system to make sure it
  doesnt happen again.

5
Over Capacity

• Site access growing at 25 - 50 per semester.
• First week of school was the biggest problem,
  especially opening day.
• More key resources on-line.
• Poor diagnostics
• Seemed slower

6
First steps--bulk up
7
Bulking up the server

• Add disk--Disk is cheap.
• Add memory--Memory is cheap.
• Get a bigger machine!

8
Short-term fix

• Got us through a few semesters, but we hit the
  new limit.
• Next size machine was very expensive
• Bigger hardware wasnt working

9
Where are the resources?

• Perl CGI scripts many on-line resources that
  were developed over time.
• CGI was easy to write, but very resource
  intensive.
• Then we added database interaction, opening and
  closing DB handles each time.

10
Each CGI hit

• spawn external process
• invoke perl
• possibly make DB connection
• return content to web server
• clean up

11
Where can we improve?

• Look for ways to speed up CGI
• already many scripts.
• Wanted to leverage existing code.

12
mod_perl Solution

• In our case, mod_perl was a very good solution.
• Able to use most of our existing code.
• Works with apache, and that was the server
  platform we were running.

13
mod_perl Solution

• ApacheRegistry to speed up the CGI scripts
• ApacheDBI to cache DB connections.

14
Getting Started

• But its complicated--how do we get started?

15
Call the Consultants

• Called Perl trainers to give us mod_perl courses.
• Helped us with training and code review.
• Worked with us on server configuration.
• Confirmed for management that the technology was
  accepted and widely used.

16
Analysis What a server does.

• Receives request from browser
• Invokes some resource to generate content to
  respond
• Some responses are fast, others are slow
• Some are dynamic, others are static

17
Where are you slow?

• We looked at what parts of our response cycle
  were slow
• 80/20 rule says you 80 of your performance
  problem is in 20 of your cycle
• Find those parts.
• Make sure you solve the right problem.

18
mod_perl speed

• For us, mod_perl accelerated the slow parts.
• mod_perl handles
• persistent perl interpreter
• keeps code in memory
• ApacheDBI keeps persistent DB connections.
• Slowest parts are all very fast now.

19
Where are you slow, part II

• Needed to convert our CGI scripts to run under
  mod_perl.
• Again, 80/20. Generated a top 10 list from our
  server logs.
• Put effort into converting those 10 resources.

20
Technical Parts

• End of manager portion of presentation

21
All that memory?

• mod_perl loads a bunch of code into memory
• Apache processes are large
• Running out of memory?

22
Server modifications

• Server does different things application
  processing and serving flat files (images and
  HTML)
• Dont need a huge application process to serve an
  image or HTML page.

23
Solution split the server

• Set up two apache servers
• one proxy server on the front
• one application server on the back

24
Front server

• Runs on normal ports (80 and 443)
• Responds to all requests
• Caches images and serves them if it can
• Sends app requests through to back
• Small footprint very few modules and no perl.
• Many children available

25
Back server

• Application server running on alternate port
  (8080)
• Only responds to front server
• Large processes with mod_perl and all loaded code
• Small number of these processes

26
(No Transcript)
27
SSL Resources

• SSL is CPU intensive (key generation for
  encryption between browser and server)
• Can go with faster CPUs
• Offload processing to dedicated hardware
  component (cryptographic accelerator cards)

28
Were set, right?

• Web server is wicked fast
• Hardly working even during heavy load
• Were donearent we?

29
What if the server goes down?

• Enterprise had two engines
• If one goes down, there is a backup
• We needed the same thing so a machine failure
  wouldnt interrupt service.

30
Redundancy

• At least two components for each one we had
• So the web server machine had to become at least
  two machines.

31
Cold Backup Machine?

• Could buy a second machine and turn it on in case
  of failure (cold backup).
• Server was big and expensive, so a backup would
  also be expensive.
• The backup would just sit there.

32
Server Farming?

• Created a series of parallel servers all running
  at the same time.
• Often called a server farm.

33
Basic Farm Layout
34
Extra Hardware

• Some sort of load balancing
• Can be software running on a box
• We use dedicated hardware
• Multiple boxes
• Instead of one big server, many small ones
• Each is cheaper
• Impact of losing one is smaller

35
Code Challenges

• Apps must act the same if user is routed to a
  different server
• Cant rely on flat files, unless you have shared
  file system.
• Could use sticky settings to keep users on the
  same machine, but this can get messy.

36
Deployment

• Same code to all machines
• Same server config on all machines
• Automated our deployment process and combined
  with version control

37
Where we are today

• 3 Cisco load balancers
• 4 Sun web farm machines
• mod_perl
• some regular cgi
• html and images
• Oracle database in cluster (2 machines)
• Automated code deployment system

38
Extra Content

• Additional considerations

39
SSL negotiation

• Probably want to do SSL on front server
• SSL is CPU intensive
• If you run servers on different machines, you
  dont want to contend with app server.

40
SSL negotiation

• Dont encrypt traffic from front to back
  server--unnecessary performance hit.
• If you are running servers on the same machine,
  not a problem.
• If on different machines, network must be secure.

41
IP Stickiness

• If your load balancer has IP sticky, use it.
• Prevents renegotiation for the same user getting
  moved around.
• Can be huge hit.
• Use keep-alive also for images.

42
Server Tuning

• Watch your server to determine how to fine-tune.
• Apache very flexible, but needs to be configured
  properly.
• Continue to look for that big performance
  problem.
• Keep metrics to identify if something changes.

43
Shared Database

• May need database so share data across all
  machines.
• Critical dependency.
• Need redundancy/high availability at this level
  too.

44
High Availability

• Storage Area Network (SAN)
• Special device with
• Guaranteed high availability
• Fiber networking for very fast access.
• Large storage.

45
Server Monitoring

• Apache has modules for monitoring
• Very useful and worth the performance hit
• ApacheVMonitor
• server-info, server-status, perl-status