Oracle 11i Upgrade Experiences

1
Oracle 11i Upgrade Experiences

• Migrating GEMMS and Financials to Oracle 11i
• February 13, 2002

2
Who Are We?

• Foremost Farms is a dairy cooperative owned by
  about 4,600 dairy farmer members in Illinois,
  Indiana, Iowa, Michigan, Minnesota, Ohio and
  Wisconsin. In 2001, they supplied more than 4.9
  billion pounds of milk. We're among the top 10
  dairy cooperatives in the United States, and our
  annual sales were 1.1 billion in 2000.
• We have about 1,700 employees in 26 locations
  across the Upper Midwest who process and market a
  full line of dairy products and ingredients for
  the wholesale and retail marketplaces. We
  manufacture packaged fluid milk chilled,
  ready-to-serve juices cheese sour cream butter
  and a host of value-added whey ingredients.
• We implemented GEMMS and Oracle Financials in
  1997.

3
System Profile

• We currently support a production database of
  about 100 Gbytes.
• Financials Modules
• Accounts Payable
• Accounts Receivable
• Fixed Assets
• General Ledger
• Pre-Upgrade Versions
• Oracle database - 7.3.4
• Oracle Financials - 10.7 prod 16.0
• GEMMS - 4.10.5
• We were also a character-based environment
• Post Upgrade Versions
• Oracle 11i version 11.5.4
• We have also had to implement many family packs
  and mini-packs to correct issues in the base
  version of the product. The major packages are as
  follows
• Oracle Process Manufacturing - Family Pack F
• Accounts Payable - Mini-Pack F
• Accounts Receivable - Mini-Pack E

• OPM Modules
• Item Master
• Batching
• Purchasing
• Formulas

• Inventory
• Quality Control
• Order Fulfillment

• General Ledger - Mini-Pack D
• Fixed Assets - Mini-Pack F
• Purchasing - Mini-Pack B

4
System Profile - (Cont.)
5
Project Approach Overview

• This presentation will provide an overview of how
  we approached the migration, the methodology we
  used to execute and some of the lessons we
  learned during the project.
• Project Authorization/Definition
• Define scope of project and setup Project Team
  and worked with Executive Staff to set
  expectations and gain their support.
• Project Analysis/Design
• Design Project Plan
• Educational Sessions
• Develop Business Scenarios for Testing
• System Testing Testing Testing
• Develop Implementation Execution Plans
• Develop Post-Implementation Support Plans
• Develop Training Strategy and Course Work
• Execute Implementation Plans

• Infrastructure Design and Implementation
• Convert Custom Applications

• Develop Migration Validation Plans
• Develop Blackout Plans

6
Project Team
Steering Committee - Executive Management
Business Process Owners - Functional Area Managers
Project Manager

• Training Support Team
• I/S Lead
• Consultant Support
• Cross Functional Team

• Infrastructure Team
• I/S Lead
• Consultant Support
• Cross Functional Team

• Financials Team
• I/S Lead
• Consultant Support
• Cross Functional
• Team

• OPM Team
• I/S Lead
• Consultant Support
• Cross Functional
• Team

Sales Order Business Lead
Procurement Business Lead
Manufacturing Business Lead
7
Communications

• One of the real challenges for this project was
  the demands for communications and coordination
  between the many resources involved and effected
  by this project. To facilitate communication we
  followed these steps
• Executive Status Meetings- Quarterly
• Information Services - Weekly Status/Situation
  Meeting
• Project Team Meetings - The entire project team
  meet every two weeks
• Project Team Leads - Weekly Status/Situation
  Meeting
• Business Team Leads - Weekly Status/Situation
  Meeting

8
Oracle 11i Migration Project Process
Train User Community
Understand Functionality Differences
Convert Existing Data to 11i
Business Processes Mapped to 11i
Create Training Plans
Migrate Prod. to Oracle 11i
Oracle Applications
Production Test Runs 12
Understand Hardware Differences
Hardware Configuration Finalized
Create Test Environment
Migration Test Runs 1-2 (New Prod. H/W)
Migration Test Runs 2-4 (New Prod. H/W)
Technology Infrastructure
Convert Customized Applications Oracle
Forms/Reports, Make Sheet, B.O. Reports, QC
Upload, etc.
Continue Customization Conversions
Unit Test Upgraded Customizations
Document Data Model Differences
Custom Applications
Complete all Testing for Customizations
10/31
1/1
March
April
May
June
August
September
July
October
9
Oracle Applications Path - Education/Setup

• Using our consulting partner we formulated a 2
  day educational overview session for the project
  team.
• We brought up the Vision database (Oracle demo
  database) and allowed the project team to
  familiarize themselves with the new
  functionality.
• We began the process of setting up a limited set
  of our data into the Vision database.
• The steps noted above
• Familiarized the user community with the
  application functionality and navigation.
• It also helped the technical team to become
  familiar with the 11i components and all of us to
  became familiar with the 11i problem resolution
  process.
• We also were able to start to develop our
  Impact/GAP analysis documents.

10
Oracle Applications Path - Business Scenario
Development

• The objective of the business scenario method of
  testing is to develop testing scenarios as
  similar to real life business processes as
  possible
• We identified 3 levels or tiers for testing as
  follows
• Tier 1 A set of iterations that followed
• The sales order process - setup an item, enter a
  sales order, schedule it, manufacture it, ship
  it, invoice for it, receive payment, apply cash
  and post.
• The purchase order process - setup an item,
  create a purchase order, receive the purchase,
  put item into inventory, receive an invoice, pay
  for it, post payment (this process was performed
  for expense and fixed assets items both new items
  and old).
• Tier 2 These iterations were more in depth
  versions of the Tier 1 scenarios that checked
  variations of the above work (different types of
  customers, items, newly created, migrated, etc.).
• Tier 3 very specific iterations (more of the
  traditional test plan type of testing) to test
  specific situations, known bugs, etc.
• Remember to test old old old new new new
• This testing methodology was successful for us
  because
• It was developed by a cross functional team.
• It was well documented and repeatable.
• Results from successive migration testing were
  consistent so issues were identifiable.

11
Technology Infrastructure Path -Multiple
Migrations

• We researched the possible hardware
  configurations using
• Our vendor - HP and Oracle documentation and
  technical resources
• NCOAUG - Some valuable information was gained
  during these conferences.
• Other companies in the process of migrating to
  Oracle 11i
• We finalized our hardware components and
  purchased a new production system.
• To help minimize the blackout time we completed
  the Multi-Org implementation in our production
  environment 2 months prior to going live with
  11i.
• We completed 6 migrations before our actual
  implementation.
• Two migrations with 11.5.3 - familiarization,
  proof of concept, debug
• Two migrations with 11.5.4 - process
  optimization, blackout period minimization, use
  of our testing process, functionality
  stabilization, change process freeze.
• Two real time migration exercises -
  representative time frames, validate resource
  schedule and migration validation plan
• For all migrations we kept a very detailed
  migration spreadsheet. For this work we started
  with Oracles migration worksheet and then
  modified to it to include some additional
  required information.

12
Technology Infrastructure Path - (Cont.)
13
Custom Applications Path - Customizations

• We were able to use Oracles Technical
  documentation for the Financials area to
  understand the differences between 10.7 and 11i.
  Unfortunately, we had to develop it for the OPM
  area.
• Over a period of about 3 months we converted
• 300 custom Oracle reports
• 500 Business Objects reports
• A dozen bolt on applications (Oracle Forms)
• We came from a Forms 4.5 and Reports 2.5
  environment to 6i.
• For the Financials area we found for the most
  part that table changes were hidden behind views
  so we had very few changes to make.
• Items we had to look out for in the OPM were
• Changed Date Field Names
• User_id field from fnd_user instead of op_code
  field, added_by , etc.
• Null fields versus spaces

14
Stress Testing

• We benchmarked our original system (number of
  users and application being run at 5 minute
  intervals through out the day) to better
  understand our environment.
• We performed the following series of tests
• Load Test 1 Goals/Objectives
• Set up an isolated environment that is
  representative of the proposed future
  environment.
• Provide ability to measure Citrix server, WAN,
  and Unix usage/load with 40 users.
• Load Test 2 Goals/Objectives
• Perform the same test as load test 1 but add a
  mix of our other applications running on the
  Citrix servers simultaneously.
• Load Test 3 Goals/Objectives
• Determine how much memory is required when people
  are logged into the environment but are idle.
• Determine the load differences on the application
  server versus the database server
• Determine if performance is acceptable at the
  256K WAN speed.
• Load Test 4 Goals/Objectives
• Test to see how many users we can get logged onto
  the system
• Understand the load on the Citrix boxes imposed
  by the B.O and Client/Server Oracle reports.
• Determine performance when users are direct
  connected to the system and running concurrent
  manager processes.
• Determine if a 128K WAN connection is viable.
• Gather more controlled results from focused tests
  of reports, concurrent manager, Citrix response
  etc.

15
Training Strategy

• Training Preparation
• We utilized a Train the Trainer model. We
  organized training teams with I/S trainers and
  manufacturing personal from the project team.
• We rewrote and put online over 600 procedures.
• The majority of the Financials users were part of
  the project team so we held only one Financials
  related class. The balance of the training was
  received OJT over the course of the project.
• Plant Lead Training
• We trained over 60 Plant Leads a month before
  the go live date. They were given practice
  assignments to complete after their training to
  continue their application familiarization.
• We were able to test drive our training material
  (and make corrections) during the Plant Lead
  training.
• End User Training
• The End User training was accomplished during the
  2 weeks prior to going live.
• We ran multiple training sessions simultaneously
  at different facilities which allowed us to
  complete the training and still keep our business
  operating.
• We added 3-4 plant leads to the training teams to
  assist with questions and to provide a higher
  teacher to student ratio.
• Class sizes were kept to 12 users per class.

16
Blackout Planning

• Being prepared for the blackout period and
  communicating goals and tasks from our Executive
  staff to all of the plants greatly contributed to
  the success of our migration.
• We broke this task into four stages
• Planning
• Goals - Complete as much business as possible
  before the blackout period to minimize business
  impact during and after the blackout.
• Pre-Blackout
• Goals - Get your activities caught up in the
  system as much as possible.
• Enter in the system all staged blackout period
  activity.
• Blackout
• Goals - Record transactions accurately and in a
  format that will facilitate entry into 11i.
• Complete as many non-11i related functions as
  possible to free up time for post blackout
  activities
• Post-Blackout
• Goals - Activities are prioritized (over 4 days)
  to expedite our ability to invoice completed
  shipments, balance system work loads to ensure
  good response times, and resume normal activities
  on the system as soon as possible.

17
Blackout Activity Overview
Day 2
Day 4
Day 1
Blackout
Day 3
Pre-Blackout
Planning

• Caught up on all blackout period production and
  receipt processing

• Complete remaining blackout day 1,2
  transactions for invoicing

• Complete October business

• Get your activities caught up in the system as
  much as possible

• Record transactions
• accurately to help re-entry

• Enter as many blackout shipments as possible to
  facilitate invoicing

• Complete as much business
• as possible before blackout

(Goals)

• Complete as many
• non-GEMMS functions as possible

• Enter in system all
• staged blackout activity

• Enter blackout shipments/File

Shipments
Ship using prepped or manual documents/File

• Get caught up

Bulk Fluid

• Prep for blackout shipments

• Enter day 1 activity

Business as usual
Contact customers

• Enter staged blackout shipments/File

Finished Goods
Day 2 shpmts follow blackout procedures
Enter blackout day 1,2,3 shipments/File

• Get caught up

Ship using prepped or manual documents/File
Business as usual
Contact customers

• Prep for blackout shipments

• Day 1shpmts follow blackout procedures

Shipment File
Coordinate blackout purchase orders
Manual purchase orders/File
Enter day 1 purchase orders
Enter blackout day 2 purchase orders/File

• Get caught up

Purchases
Business as usual

• Prep for blackout purchases

PO/Receiving File
Receipts
Get caught up
Intake log
Enter day 1intake log activity
Enter blackout and day 2 receipts from log
Business as usual
Bulk Fluid
Non-Bulk Fluid
Get caught up
Manual/File
Enter day 1receipts
Enter blackout and day 2 receipts/File
Business as usual
PO/Receiving File

• Enter blackout production
• Day 2 follow blackout procedures

• Get caught up
• Stage for blackout days 1, 2

Cheese - EMS
Follow blackout procedures
Enter days 1,2,3
Production
Business as usual
Ingredient - manual logs

• Continue to upload blackout QC

Upload days 1,2,3 of finish goods bulk fluid
Quality Control
Print customer specification report
Enter info into QC upload , no upload functions
available
Business as usual
Upload blackout QC
Physical Inventory 9/30
Physical Inventory 10/31
Enter Inventory Adjustments
Inventory
Record inventory adjustments
Follow blackout procedures
Follow blackout procedures
18
Post Implementation Support Structure

• We tapped into the expertise developed during the
  testing and educational phases of the project to
  put a 3-tier 24x7 support structure in place.
• Tier 1 - Plant Leads
• First point of contact for all issues at their
  Plant. Expected to resolve the majority of the
  how do I do this questions.
• Escalates issues they cant answer to their
  assigned Tier 2 person.
• Tier 2 - Training Team and other non-I/S Project
  Team Members
• Each person was assigned a functional area
  (purchasing, inventory, etc.) and 2-3 plants.
  Which allowed us to support 29 facilities and
  still complete normal duties.
• Responsible for documenting, managing and
  communicating resolution of issues to/from their
  assigned plant leads.
• Expected to answer the majority of the plant
  questions. Escalate issues they cant answer to
  the appropriate Tier 3 person.
• Tier 3 - I/S Team Members
• Responsible for researching and correcting
  complex issues.
• Communicates issue resolutions to Tier 2 person.
• During the first 2 weeks after implementation we
  held nightly (gradually decreasing the frequency)
  issue review meetings to insure dissemination of
  information.

19
Support Structure - (Cont.)
Plant 1 Users
Plant 2 Users
Plant 26 Users
gtgtgtgt
Plant Lead
Plant Lead
Plant Lead
Tier 1
Purchasing /Receiving Contact
Shipping Contact
Inventory Contact
Tier 2
I/S Financials
I/S OPM
I/S Logon Issues
I/S Desktop Printer Issues
Tier 3
20
Migration Validation

• After completion of the migration we performed a
  series of setup and validation steps to insure
  that all data was migrated correctly and the
  setup information in 11i was consistent with the
  setup of Financials 10.7 and GEMMS 4.10.05.
• Table record count validation This process
  consisted of an export of the row counts of all
  tables in the database both prior to the
  conversation and again after the conversation was
  complete.
• To insure accuracy we shadowed the critical
  manual setup steps required for A/R, G/L,
  purchasing, pricing, etc.
• Personnel from A/R, A/P, G/L and F/A departments
  used pre-migration screen prints and reports to
  compare to the corresponding migrated screens and
  reports.
• Cost accountants validated that each piece of the
  MAC was setup correctly.
• The final validation step consisted of running
  our Tier 1 iterations on the migrated system.
• This validation process took about 4 hours for
  the team to complete.

21
Migration Challenges

• Oracles fast pace of software revision (which
  doesnt stop upon completion of the migration).
• You can expect to apply many patches to correct
  items that are not functioning correctly
  (depending on how fully you use the system).
• Purchasing and receiving interface between OPM
  and Financials.
• Application kickouts, lockups and general
  performance issues (although this seems to be
  getting much better in later versions of the
  infrastructure).
• Time commitment from user community to insure
  that the application functions correctly and to
  learn to navigate in the new environment.
• Develop/Testing limitations because you can only
  have two environments per server (excluding the
  Vision database).
• Oracle 11i has become a complicated environment
  patches need to be evaluated carefully, applied
  judiciously and tested very thoroughly.

22
Oracle Support

• Metalink Searches
• Look for your problem many have already been
  identified
• Metalink TARS
• We had around 250 TARS during project (and
  continue to open more)
• Log a TAR as soon as a problem is identified and
  duty manage critical TARS to raise priority
• Conference Calls
• Get an Oracle Support and Development contact
  early in the process.
• Separate contacts one for Oracle Financial and
  the OPM group
• Set up regular conference calls to review open
  TARS
• We found Oracle to be very responsive and helpful
  (particularly the OPM group) to issues using this
  process
• Bugs
• Oracle can reproduce problem and will determine
  if it should be corrected.
• Patches
• Many already exist for known problems
• Development works on a bug to create a patch for
  release
• Many fixes may be rolled into one patch
• Read me files offer little help determining what
  is in patch
• No way of knowing patches exist to apply to
  prevent problems

23
Common Purchasing OPM Receiving

• This is the area where we had the most problems
  in the migration.
• It is also the focus of the area where we
  continue to fight issues.
• A fair amount of setup is required even for our
  simple process.
• Multiple Flexfields
• Workflow and Approval Concurrent Processes
• PO, GL and Inventory Periods
• Profile Options
• Test Combinations of Migrated and New POs
  Receipts
• Two items of particular interest when
  troubleshooting interface errors
• po_interface_errors.interface_header_id
• rcv_transactions_interface.header_interface_id

24
Lessons Learned

• Unlike most implementations the upgrade to Oracle
  11i was a very DBA intensive task. You must have
  a strong, structured, detail orientated DBA to
  insure success.
• We were able to save a tremendous amount of time
  during the blackout period (our first migration
  would have taken 6.5 days in real time versus the
  4 days the actual migration took) by doing tasks
  ahead of time (Multi-Org, data cleanup, etc.),
  performing tasks in parallel and moving tasks to
  post migration (on-line help).
• We were able to avoid the tedious and time
  consuming task of setting up responsibilities for
  all users by exporting the setups from our test
  environment and importing them into the
  production environment and scripting the user
  setup.
• This project is very manageable given sufficient
  planning, time and cross functional resources.
  You must document in detail and not jump into the
  migration without taking the time to think and
  plan out each step(s) before executing.
• Plant/User ownership and accountability was
  critical to our success.

25

• Questions?