FY09 Tactical Plans for Database Services

1
FY09 Tactical Plans forDatabase Services

• J.Trumbo
• Sept. 24, 2008

2
FY09 Tactical Plan for Database Services

• CD DocDB Document 2889-v1
• 2009 Tactical Plan Database Services
• Tactical Plan Leader J.Trumbo
• Organizational Unit home LSCS/DBI/DBA

3
FY09 Tactical Plan Strategies and Goals

• Tactical Plan Strategies
• Develop stronger expertise and support for open
  source databases, reducing the dependency on
  Oracle, with a goal of reducing development and
  support costs.
• Tactical Plan Goals
• Continue to provide a stable, professional
  operation of laboratory scientific and
  information systems databases, providing maximum
  uptime and technical support.

4
FY09 Tactical Plan Objectives

• Tactical Plan Objectives
• 1. Database Systems Operations
• Support running experiments at appropriate
  levels.
• 2. Database Support.
• Promote Postgres and Oracle as a 1st tier
  database and MS-SQL and MySQL as second-tier
  databases.
• Support production, integration and development
  on all database platforms. Configure, tune, test,
  and maintain databases.
• 3. Agile response to new and rapid shifts of
  responsibilities and demands accompanying the end
  of Run II data taking and reduction of effort in
  the Run II experiments.

5
FY09 Tactical Plan Objectives

• 4. Kerberize Oracle using Advanced Security
  Option.
• 5. Maintain expertise for new features. Work with
  databases on vm environments, oracle rac testing,
  etc.
• 6. Implement Oracle Internet Directory (OID).
• 7. Complete upgrade to Oracle 10 for the
  information systems databases.
• 8. Upgrade information systems hardware for
  databases, including major apps, computer
  security, 24.7 applications and nimi development.
  
• 9. Continued Deployment of the 3par San
  technology.
• 10. Ramp SDSS support down to maintenance levels.

6
FY09 Tactical Plan Past Action Items

• Resolution of Past Action Items SDSS
• Planning hardware/software layouts for the next
  DR7 and final DR8 releases. Obtaining new
  machines and licenses promptly. Reconfigured
  number of existing machines to meet our hardware
  requirements for the future maintenance of the
  SDSS machines to keep less machines in the rack.
• Implemented 2 mirror sites for DR5 Data Release
  in the Libraries of JHU and University of
  Chicago.
• SDSS team working close with team of the
  librarians to setup SDSS help desk support using
  new web application Question Point.
• SDSS DBA team successfully completed transition
  CasJobs web service from SQL2K to SQL2K5.
• Completed incremental loading from DR6 to DR7.1
  (BestDR7.1, TargDR7.1, SegueDR7.1 - total more
  than 6TB data) and released it to the SDSS
  collaboration on February 14 - two weeks early.
• Working close with team of developers from JHU
  and Princeton to implement and test new code
  related to schema modifications for DR7.2 with
  major changes in data model and merging
  SegueDR7.1 to BestDR7.2.
• Completed loading 3TB data into new database
  Stripe82, this is a new
• addition to the SDSS databases in respond to the
  SDSS scientists to
• merge stripe 82 runs into one database.
• Successfully upgraded 7TB RunsDB database from
  SQLServer2000 to SqlServer 2005 environment.
• Working constantly on SDSS documentation and
  improving monitoring tools to maintain stable
  production environment.

7
FY09 Tactical Plan Past Action Items

• Resolution of Past Action Items Open Source
• To be clear, points below include effort from
  sysadmins as well as dbas!
• Established best practices, standards, how tos
  for open source implementations.
• Cross training as much as possible.
• Established postgres general purpose boxes for
  dev/int/prd. Installed several users including
  content_mgmt, ilc ground motion and edu office
  lasso project dbs.
• Took mgmt of mysql general purpose boxes. they
  need work to upgrade and bring to stds, work is
  pending.
• Migrating support of gratia db to dba.
• Benchmarked MyISAM vs InnoDB engines for Gratia.
• Evaluated several backup products for mysql
  operating under InnoDB engine.
• Established nova dev/int/prd postgres boxes.
• Took over support of NIMI. Modifying to
  standardized installs, upgrade os and db.
• Continuing cross training of group. Attempted to
  set up an onsite postgres class for developers
  and dbas, but furloughs disrupted attendance
  planning.
• Assisted minerva with mysql and postgres installs
  on general purpose boxes.
• Established the Content Mgmt postgres database
  dev/int/prd.
• Still not quite transitioned minos production
  mysql db to dba.

8
Service Activity Database Support, including
Oracle, SqlServer Open Source

• Support running experiments at appropriate
  levels, insure a stable, performant, and smooth
  running environment.
• Goals Related to this Activity
• Install, configure, and test new databases,
  perform version upgrades of existing databases.
• Administer and operate production and development
  database information systems.
• Participate in a 24x7 on-call rotation schedule
  for production issues.
• Perform database backups and data restores.
• Disaster recovery for databases.
• Replication
• Diagnose problems, and monitor database
  performance.
• Perform tuning of SQL queries.
• Install and configure Oracle Internet Directory
  (OID) for Oracle v10 dbs.

9
Service Activity Database Support

• Key Metrics
• Uptime Statistics
• http//www-css.fnal.gov/dsg/external/database_upti
  me/
• http//www-css.fnal.gov/dsg/internal/SDSS/metrics/
  metrics.htm
• Issues and Risks (specific to this activity,
  includes allocation impact)
• Hardware failure
• Aging Hardware

10
Project Activity Kerberosize Oracle with ASO

• Continue Kerberos for Oracle Implementation
• Goals Related to this Activity
• continue oracle aso kerberos project
• implement Kerberos authentication for DBA tasks
• install aso for users as needed
• define an action plan to support aso

• Key Milestones
• Implementation into production databases.
• Metrics, of kerberized dba tasks, of
  kerberized user tasks
• Issues and Risks (specific to this activity,
  includes allocation impact)
• Manpower

11
Project Activity Complete Upgrade to Oracle 10

• Upgrade Infrastructure Databases (miscomp) to
  Oracle 10
• Goals Related to this Activity
• os upgrade
• Information systems databases upgrade 10gR2

• Key Milestones
• Os upgrade, databases upgraded.
• Metrics, of instances upgraded
• Issues and Risks (specific to this activity,
  includes allocation impact)
• Application readiness

12
Project Activity Upgrade Infrastructure Hardware

• Upgrade infrastructure with new hardware
• Goals Related to this Activity
• plan separation of information systems apps,
  including 24.7 instances and misprod1.
• plan for data movement to new databases
• Implement new databases, including oid repository
  and new remedy.

• Key Milestones
• New hardware implementation successes
• Metrics, of upgraded
• Issues and Risks (specific to this activity,
  includes allocation impact)
• manpower

13
Project Activity Migrate Databases to 3Par

• Move databases to 3Par array
• Goals Related to this Activity
• Move dev/int cdf offline databases to 3par to
  replace aging and unsupported disk array.
• Move dev/int d0 luminosity databases/snapshots to
  3par to allow for a full and complete testing
  environment. (complete)
• Move fcdfora6 replica to 3par
• Assess and move if possible the cdf online
  dev/int environments of aging array.

• Key Milestones
• Number of successful moves successes
• Metrics, uptime, saving on time and materials
  repairs to unsupported hardware.Issues and Risks
  (specific to this activity, includes allocation
  impact)
• Issues and Risks (specific to this activity,
  includes allocation impact)
• 3par readiness

14
Project Activity SDSS to Maintenance Mode

• Ramp down SDSS to maintenance levels.
• Goals Related to this Activity
• Load and Release DR7
• Load and Release DR8
• Loading Runs Database
• Standardizing the Idrea Monitoring.
• Support the transition from SDSS operations to
  long-term maintenance.
• Consultancy JHU and UC for hosting DR5
• UC DR7 implementation

• Key Milestones
• Meet release dates
• Metrics, less time spent on SDSS, uptime stats
• Issues and Risks (specific to this activity,
  includes allocation impact)
• Manpower,

15
FY09 Resource Request, Preliminary Allocation
Level 0 Activity Application Support
16
Impact of Preliminary Allocation

• SDSS will be annually funded 60k for continuing
  support. In addition 3 fte months are allotted
  for Oct/Nov, Oct (2 ftes) completing public
  release, Nov. (1fte) tying up loose ends.
• Additional funding needs to be procured if DR7 at
  U of C is to be completed, needing hardware and
  manpower.

17
Template Guidance (not part of the presentation)

• Activity Activity Level 1 or lower explicitly
  listed in Tactical Plan
• Activities have been classified in Tactical Plans
  as either being more Project-like or Service-like
• Projects finite defined lifetime, defined
  deliverable. Includes RD work.
• A.k.a Upgrades,
• Services same operation or task repeatedly.
• A.k.a. Operations, Maintenance, Support,
• Whether an activity is a project or service
  determines which format slide to use later.
• Terminology
• Obligation Budget Allocated budget. This may
  differ from the budget request.
• YTD Obligations Fully incurred expenses. Does
  not include RIPs, Req.s in CD, etc.
• Current FY08 Forecast Estimate of all
  obligations to be incurred by the end of FY08.
• MS Equipment Operating. Operating includes
  Travel, Material Purchases, Professional Services
  (contractors), Maintenance (licenses), and Other.
• RIP Requisitions in Progress