Infrastructure to Support Extreme Self-Service

1
Infrastructure to Support Extreme Self-Service

• Session 20653
• March 15, 2006
• Alliance 2006 Conference
• Nashville, Tennessee

2
Presenter

• Carol Jordan
• I have been working on PeopleSoft applications
  since 1997, and have been at Stanford since
  November 2000.
• I am the Infrastructure Manager for the team
  supporting Student and HR Systems.
• In addition to the PeopleSoft products, our group
  also supports Kronos, Resource 25, OnBase
  Imaging, Resumix and a custom Workflow
  application.

3
Overview

• Before PeopleSoft
• - Limited functionality for students
• - using a web-enabled mainframe application
• allowing only 35 concurrent student sessions
• With PeopleSoft HR/SA and Portal
• Many functions for students
• and for faculty, advisors and staff
• and no portal sessions are turned away

4
Agenda/Contents

• In this presentation
• Extreme self-service defined
• Our rough start
• Where we are now, how we got here
• What we learned

5
Overview Stanford University

• Stanford University is a private university
  located 30 miles south of San Francisco and just
  north of Silicon Valley.
• There are currently 6,700 undergraduate and 8,000
  graduate students enrolled, with 1,775 faculty.
• Founded in 1891, the university was established
  by Jane and Leland Stanford in memory of their
  son, Leland Stanford, Jr., who died of typhoid
  fever at the age of 15.

6
Overview Application Administration
The Application Administration team is part of
Administrative Systems, supporting many of the
applications used campus-wide. There are three
AppAdmin teams, supporting PeopleSoft, Oracle
Financials, and Reporting. The team supporting
PeopleSoft is made up of 5 System Analysts, 3
Oracle DBAs and 3 Windows System Administrators.
Developers and other support staff are part of
a separate team within Administrative Systems.
7
Overview Application Administration

• PeopleSoft products currently installed
• Campus Solutions 8 SP1
• PeopleTools 8.22.05
• Enterprise Portal 8.8 SP1
• PeopleTools 8.44.03
• Enterprise Learning Management 8.8 SP1
• PeopleTools 8.45.12
• All on Oracle 9.2.0.6

8
Extreme Self-Service, defined

• Lots of functionality for students, faculty,
  advisors and staff
• and
• Lots of users close to 35,000 logins on our
  busiest days

9
Extreme Self-Service, defined

• Lots of functionality for students
• File or adjust study list and elect grading
  options
• Request an official transcript to be mailed
• Print history of courses and grades
• Print Enrollment Certification
• Declare major and minor
• Apply to graduate
• Update addresses, emergency contacts
• Maintain FERPA elections
• View and accept financial aid
• View student bill and print statement
• View advisors
• View degree progress, milestones, program summary
• Coming in Marchenter or read course evaluations
• Coming in Aprilpay student bill, add money to
  card plan
• Links to other applications Student Housing, IT
  Services bill (Pinnacle), Office of Development

10
Extreme Self-Service, defined

• Lots of functionality
• for Faculty
• Review class lists
• Send e-mail to students in a class
• Submit and change grades
• Assign a grade proxy
• Review previously-submitted grades
• Coming in Marchread students course evaluations
• for Advisors
• View lists of current and past advisees
• View study lists, grades and unofficial
  transcripts for current advisees
• Place and release holds of current advisees
• for Staff
• View on-line pay statement
• Make W-4 elections
• Enter direct-deposit elections
• Register for training, take on-line tests
  through PeopleSoft ELM
• Link to applications Kronos, on-line W-2

11
Extreme Self-Service, defined
Lots of users
12
Our Rough Start

• Upgrade to Version 8 went OK
• First couple of months went OK
• but the system could not support the sustained
  demand of our busiest days the first two days
  of the new academic year

13
Our Rough Start

• Stanford Daily articles with headlines like,
  Axess Problems Plague First Week
• What students saw
• log-in attempts that failed
• connection refused errors between HR/SA and
  Portal
• pages that didnt load

14
Our Rough Start

• CIO magazine article Big Mess on Campus
• Stanford's IT was still struggling with
  integrating the enterprise systems when the newly
  launched PeopleSoft Web portal (called Axess)
  crashed last fallAxess couldn't handle the load
  of all the returning students trying to log in

15
Where we are now

• We are able to handle high load without issues
• Business offices are planning to roll out new
  functionality using the Axess portal
• and weve managed to stay out of the papers for
  a while (fingers crossed)

16
Obligatory Hardware Diagram
17
How We Got Here

• Upgraded hardware, added hardware
• Server hosting custom single sign-on was the
  biggest, earliest bottleneck
• Original web servers were old and slow
• Got the OK to order new hardware after capacity
  issues hit
• Lessons
• Go with your gutif youre worrying about it, fix
  it if you cant fix it, have a contingency plan
• Fight the requests to minimize costs by re-using
  old hardware

18
How We Got Here

• Web-server changes for HR/SA and Portal
• Upgraded JRE
• Upgraded Jolt and WebLogic to the latest rolling
  patch
• Reduce network disconnect timeout
• Used WebLogic console to fine-tune JVM heapsize
  (768M) and thread-count (100) on Windows web
  servers
• Set Windows PIA service to restart automatically
  if it goes down
• Installed two logical web-servers per physical
  server to improve memory utilization under
  Windows
• Separated administrative and self-service users,
  to set different time-outs, breadcrumbs, and
  other configuration parameters for different
  types of users

19
How We Got Here

• Application-server changes for HR/SA and Portal
• Upgraded JRE
• Upgraded Tuxedo to latest rolling patch
• Reduced client-cleanup timeout
• For Portal and ELM implemented shared
  application-server cache
• For Portal and ELM re-configured Tuxedo domains
  to eliminate spawning always start the maximum
  number of PSAPPSRVs
• 35 per server for each of two Portal application
  servers
• 8 per server for each of two ELM application
  servers
• 15-25 per server for each of two HR/SA
  application servers

20
What we learned

• We cant predict load with certainty
• High load is loosely tied to the beginning of the
  term, when fees are due and students need to know
  where their classes are
• so we implemented defensive monitoring to send
  an e-mail when at 400 logins per quarter-hour
• Sample e-mail textThe Axess login count as of
  0016, 03/13/2006 is 550 in a 15-minute interval.

21
What we learned

• Because we count logins, we can see patterns in
  user activity
• 03/12/2006 2230 57
• 03/12/2006 2245 66
• 03/12/2006 2300 94
• 03/12/2006 2315 80
• 03/12/2006 2330 105
• 03/12/2006 2345 240
• 03/13/2006 0000 550
• 03/13/2006 0015 197
• 03/13/2006 0030 181
• 03/13/2006 0045 128
• 03/13/2006 0100 100
• 03/13/2006 0115 90
• 03/13/2006 0130 76
• 03/13/2006 0145 57

Spring enrollment opened at 1201am on Monday 3/13
22
What we learned

• Monitor at many levels
• At the web-server layer
• We monitor the load-balanced site
• and individual (logical) web servers
• and the HTTP port used by the load-balancer
  probe
• At the application-server layer
• We monitor for Tuxedo processes
• UNIX SysAdmins monitor memory usage, add swap
  space as needed
• We monitor for CPU utilization (sample e-mail
  follows)

23
CPU monitor sample e-mail

• Info from sar
• Time 004010
• usr 77
• sys 10
• wio 0
• idle 14
• Top output
• load averages 4.76, 9.29, 15.18 004010
• 106 processes 101 sleeping, 1 zombie, 4 on cpu
• Memory 8192M real, 1143M free, 6577M swap in
  use, 18G swap free
• PID USERNAME THR PRI NICE SIZE RES STATE
  TIME CPU COMMAND
• 26073 a2k_prd 4 0 0 203M 187M cpu0
  440 10.43 PSAPPSRV
• 9800 a2k_prd 4 58 0 256M 238M sleep
  1349 8.83 PSAPPSRV
• 10991 a2k_prd 4 50 0 245M 227M sleep
  1118 7.61 PSAPPSRV

24
What we learned

• Use your PeopleSoft tools!
• PeopleSoft Performance Monitor for Portal and
  ELM
• WebLogic console for web servers

25
What we learned

• Keep Tools up-to-date watch for BEA updates
• Move AppMessaging to a dedicated host keep
  back-end processing out of the transactional
  infrastructure
• Nobody hits logoff configure your web
  servers to close idle connections quickly
• Performance testing is mandatory for new
  projects
• Configuration review, top-to-bottom, with
  PeopleSoft Consulting was helpful

26
Questions?
27
Contacts

• Carol Jordan
• Infrastructure Manager, Student and HR Systems
• Administrative Systems
• Stanford University
• E-mail cajordan_at_stanford.edu
• Jeff Butler
• Lead System Analyst, Student HR Systems
• Stanford University
• E-mail jbutler_at_stanford.edu
• Ross Shimabukuro
• System Analyst, Student HR Systems
• Stanford University
• E-mail ryshima_at_stanford.edu

28
This presentation and all Alliance 2006
presentations are available for download from the
Conference Site
Presentations from previous meetings are also
available