FY09 Tactical Plans for Database Services

1
FY09 Tactical Plans forDatabase Services

• J.Trumbo
• Sept. 24, 2008

2
FY09 Tactical Plan for Database Services

• CD DocDB Document 2889-v1
• 2009 Tactical Plan Database Services
• Tactical Plan Leader J.Trumbo
• Organizational Unit home LSCS/DBI/DBA

3
FY09 Tactical Plan Strategies and Goals

• Tactical Plan Strategies
• Develop stronger expertise and support for open
  source databases, reducing the dependency on
  Oracle, with a goal of reducing development and
  support costs.
• Tactical Plan Goals
• Continue to provide a stable, professional
  operation of laboratory scientific and
  information systems databases, providing maximum
  uptime and technical support.

4
FY09 Tactical Plan Objectives

• Tactical Plan Objectives
• 1. Database Systems Operations
• Support running experiments at appropriate
  levels.
• 2. Database Support.
• Promote Postgres and Oracle as a 1st tier
  database and MS-SQL and MySQL as second-tier
  databases.
• Support production, integration and development
  on all database platforms. Configure, tune, test,
  and maintain databases.
• 3. Agile response to new and rapid shifts of
  responsibilities and demands accompanying the end
  of Run II data taking and reduction of effort in
  the Run II experiments.

5
FY09 Tactical Plan Objectives

• 4. Kerberize Oracle using Advanced Security
  Option.
• 5. Maintain expertise for new features. Work with
  databases on vm environments, oracle rac testing,
  etc.
• 6. Implement Oracle Internet Directory (OID).
• 7. Complete upgrade to Oracle 10 for the
  information systems databases.
• 8. Upgrade information systems hardware for
  databases, including major apps, computer
  security, 24.7 applications and nimi development.
  
• 9. Continued Deployment of the 3par San
  technology.
• 10. Ramp SDSS support down to maintenance levels.

6
FY09 Tactical Plan Past Action Items

• Resolution of Past Action Items SDSS
• Planning hardware/software layouts for the next
  DR7 and final DR8 releases. Obtaining new
  machines and licenses promptly. Reconfigured
  number of existing machines to meet our hardware
  requirements for the future maintenance of the
  SDSS machines to keep less machines in the rack.
• Implemented 2 mirror sites for DR5 Data Release
  in the Libraries of JHU and University of
  Chicago.
• SDSS team working close with team of the
  librarians to setup SDSS help desk support using
  new web application Question Point.
• SDSS DBA team successfully completed transition
  CasJobs web service from SQL2K to SQL2K5.
• Completed incremental loading from DR6 to DR7.1
  (BestDR7.1, TargDR7.1, SegueDR7.1 - total more
  than 6TB data) and released it to the SDSS
  collaboration on February 14 - two weeks early.
• Working close with team of developers from JHU
  and Princeton to implement and test new code
  related to schema modifications for DR7.2 with
  major changes in data model and merging
  SegueDR7.1 to BestDR7.2.
• Completed loading 3TB data into new database
  Stripe82, this is a new
• addition to the SDSS databases in respond to the
  SDSS scientists to
• merge stripe 82 runs into one database.
• Successfully upgraded 7TB RunsDB database from
  SQLServer2000 to SqlServer 2005 environment.
• Working constantly on SDSS documentation and
  improving monitoring tools to maintain stable
  production environment.

7
FY09 Tactical Plan Past Action Items

• Resolution of Past Action Items Open Source
• To be clear, points below include effort from
  sysadmins as well as dbas!
• Established best practices, standards, how tos
  for open source implementations.
• Cross training as much as possible.
• Established postgres general purpose boxes for
  dev/int/prd. Installed several users including
  content_mgmt, ilc ground motion and edu office
  lasso project dbs.
• Took mgmt of mysql general purpose boxes. they
  need work to upgrade and bring to stds, work is
  pending.
• Migrating support of gratia db to dba.
• Benchmarked MyISAM vs InnoDB engines for Gratia.
• Evaluated several backup products for mysql
  operating under InnoDB engine.
• Established nova dev/int/prd postgres boxes.
• Took over support of NIMI. Modifying to
  standardized installs, upgrade os and db.
• Continuing cross training of group. Attempted to
  set up an onsite postgres class for developers
  and dbas, but furloughs disrupted attendance
  planning.
• Assisted minerva with mysql and postgres installs
  on general purpose boxes.
• Established the Content Mgmt postgres database
  dev/int/prd.
• Still not quite transitioned minos production
  mysql db to dba.

8
Service Activity Database Support, including
Oracle, SqlServer Open Source

• Support running experiments at appropriate
  levels, insure a stable, performant, and smooth
  running environment.
• Goals Related to this Activity
• Install, configure, and test new databases,
  perform version upgrades of existing databases.
• Administer and operate production and development
  database information systems.
• Participate in a 24x7 on-call rotation schedule
  for production issues.
• Perform database backups and data restores
• Disaster recovery for databases.
• Replication support and maintenance
• Diagnose problems, and monitor database
  performance.
• Perform tuning of SQL queries.
• Install and configure Oracle Internet Directory
  (OID) for Oracle v10 dbs.

9
Service Activity Database Support, including
Oracle, SqlServer Open Source

• Goals Related to this Activity
• Move database to 3par including fcdfora6,
  fcdfora4, fcdfora1, minos
• Configure and implement 5 new d0 online database
  boxes to replace current d0 online, test new
  cluster software and failover.
• Configure and implement new cdf offline
  production box. Reuse current production box for
  development replacement. Dependant on hardware
  delivery.
• Configure and implement new d0 offline production
  box. Reuse current production box for
  development replacement. Dependant on hardware
  delivery.
• Begin planning upgrade of Oracle databases to
  11g.
• Design and implement ITL asset management
  database. Scope undefined and unknown.
• Implement ust1_fts, the cms database on new
  hardware
• Move the esh dev/int/prd database to a Unix
  platform from Windows.

10
Service Activity Database Support, including
Oracle, SqlServer Open Source
Goals Related to this Activity - Train dbas in
both mysql and postgres to a level of production
support, this includes formal classes as well as
on the job training. - Retrofit toolman
application for open source database platforms,
postgresmysql, if possible, providing a
monitoring tool for open source
platforms. - Rebuld mysql installations on
flxd01flxd04 so that products are not owned by
root. - Implement new machines to replace
flxd01flxd04. Dependant on hardware purchase.
- Implement new machine for nimi dev database.
Dependant on hardware purchase 1 fte
month - Upgrade minos production and development
to v5 of mysql. - Implement new Minerva
production machine - Cross train dbas on MS
SqlServer technology.
11
Service Activity Database Support

• Key Metrics
• Uptime Statistics
• http//www-css.fnal.gov/dsg/external/database_upti
  me/
• http//www-css.fnal.gov/dsg/internal/SDSS/metrics/
  metrics.htm
• Issues and Risks (specific to this activity,
  includes allocation impact)
• Hardware failure
• Aging Hardware

12
Project Activity Kerberosize Oracle with ASO

• Continue Kerberos for Oracle Implementation
• Goals Related to this Activity
• continue oracle aso kerberos project
• implement Kerberos authentication for DBA tasks
• install aso for users as needed
• define an action plan to support aso

• Key Milestones
• Implementation into production databases.
• Metrics, of kerberized dba tasks, of
  kerberized user tasks
• Issues and Risks (specific to this activity,
  includes allocation impact)
• Manpower

13
Project Activity Complete Upgrade to Oracle 10

• Upgrade Infrastructure Databases (miscomp) to
  Oracle 10
• Goals Related to this Activity
• os upgrade
• Information systems databases upgrade 10gR2

• Key Milestones
• Os upgrade, databases upgraded.
• Metrics, of instances upgraded
• Issues and Risks (specific to this activity,
  includes allocation impact)
• Application readiness

14
Project Activity Upgrade Infrastructure Hardware

• Upgrade infrastructure with new hardware
• Goals Related to this Activity
• plan separation of information systems apps,
  including 24.7 instances and misprod1.
• plan for data movement to new databases
• Implement new databases, including oid repository
  and new remedy.

• Key Milestones
• New hardware implementation successes
• Metrics, of upgraded
• Issues and Risks (specific to this activity,
  includes allocation impact)
• manpower

15
Project Activity Migrate Databases to 3Par

• Move databases to 3Par array
• Goals Related to this Activity
• Move dev/int cdf offline databases to 3par to
  replace aging and unsupported disk array.
• Move dev/int d0 luminosity databases/snapshots to
  3par to allow for a full and complete testing
  environment. (complete)
• Move fcdfora6 replica to 3par
• Assess and move if possible the cdf online
  dev/int environments of aging array.
• Assess the use of 3par snapshotting to
  replace/supplement oracle rman.

• Key Milestones
• Number of successful moves successes
• Metrics, uptime, saving on time and materials
  repairs to unsupported hardware.Issues and Risks
  (specific to this activity, includes allocation
  impact)
• Issues and Risks (specific to this activity,
  includes allocation impact)
• 3par readiness

16
Project Activity SDSS to Maintenance Mode

• Ramp down SDSS to maintenance levels.
• Goals Related to this Activity
• Load and Release DR7
• Load and Release DR8
• Loading Runs Database
• Standardizing the Idrea Monitoring.
• Support the transition from SDSS operations to
  long-term maintenance.
• Consultancy JHU and UC for hosting DR5
• UC DR7 implementation

• Key Milestones
• Meet release dates
• Metrics, less time spent on SDSS, uptime stats
• Issues and Risks (specific to this activity,
  includes allocation impact)
• Manpower,

17
Project Activity DES Standby database

• Move databases to 3Par array
• Goals Related to this Activity
• Test and develop expertise in oracle rac.
• Test and develop expertise in auto storage mgmt.
• Implement development stand by database at fermi,
  master in champaign.
• Install Oracle and implement production standby
  database at fermi
• Investigate and implement proper rac, standby
  monitoring tools.

• Key Milestones
• Meet schedule release dates.
• Issues and Risks (specific to this activity,
  includes allocation impact)
• Hardware availability, MOU

18
FY09 Resource Request, Preliminary Allocation
Level 0 Activity Application Support NOTE
FTES ARE FTE MONTHS
FY09 Tactical Plan Activity Level 1 FTE Request At Activity Level 1 MS Request At Activity Level 1 FTE Allocation At Activity Level 1 MS Allocation At Activity Level 1
Database Services D0 (onoffline) CDF (onoffline) Minos DES Standby database Nova Gratia Infrastructure CMS (2days_at_month) Gen.Support, Open Src 18.25 18.25 2 6 1 1 12 .1 24.5 0 0
  Oracle License Travel/Edu/Conf New Hdw/Licenses 0 0 0 370,000 79,250 93,000
 

 
 
Total

19
FY09 Resource Request, Preliminary Allocation
Cont.
FY09 Tactical Plan Activity Level 1 FTE Request At Activity Level 1 MS Request At Activity Level 1 FTE Allocation At Activity Level 1 MS Allocation At Activity Level 1
SDSS complete dr7 SDSS maintenance Dr 7 at u of c 3 6 .5 0 0 0
 
 
Total 93.5 542.250 0 0
20
Impact of Preliminary Allocation

• SDSS will be annually funded 60k for continuing
  support. In addition 3 fte months are allotted
  for Oct/Nov, Oct (2 ftes) completing public
  release, Nov. (1fte) tying up loose ends.
• Additional funding needs to be procured if DR7 at
  U of C is to be completed, needing hardware and
  manpower.

21
Template Guidance (not part of the presentation)

• Activity Activity Level 1 or lower explicitly
  listed in Tactical Plan
• Activities have been classified in Tactical Plans
  as either being more Project-like or Service-like
• Projects finite defined lifetime, defined
  deliverable. Includes RD work.
• A.k.a Upgrades,
• Services same operation or task repeatedly.
• A.k.a. Operations, Maintenance, Support,
• Whether an activity is a project or service
  determines which format slide to use later.
• Terminology
• Obligation Budget Allocated budget. This may
  differ from the budget request.
• YTD Obligations Fully incurred expenses. Does
  not include RIPs, Req.s in CD, etc.
• Current FY08 Forecast Estimate of all
  obligations to be incurred by the end of FY08.
• MS Equipment Operating. Operating includes
  Travel, Material Purchases, Professional Services
  (contractors), Maintenance (licenses), and Other.
• RIP Requisitions in Progress