INT407 Success Story for e-Biz Impact Upgrade

1
INT407 Success Story for e-Biz Impact Upgrade
Kathy KemperSystem Architect, Group Health
Cooperativekemper.k_at_ghc.org / (206)
448-2776August 5, 2003
2
Agenda
Goal Give you something to take home and use

• Background
• Upgrade Project
• What could we have done to make the upgrade
  easier?
• What caused us grief?
• What really worked?
• How we handled some technical issues
• Suggestions on Project Organization
• Resources
• Future Steps
• QA

3
Background information
Group Health Cooperative Who are we
What our Systems are like
Getting started
Project Focus
4
Introduction
Group Health Cooperative

• Who we are
• Non-profit health care system that coordinates
  care and coverage
• Second largest consumer-governed health care
  system in US
• Founded in 1947 by union members doctors to
  provide quality, affordable healthcare
• Statistics
• Enrollment 580,000 one in 10 Washington
  residents
• Medical Practitioners 1,000
• Total Staff 10,000 Washingtons 3rd largest
  private employer
• 26 medical centers, 3 specialty centers, 1
  hospital
• Contractual agreements with over 6,000 community
  practitioners 50 community hospitals
• 20 counties in WA, 2 in Idaho

5
Application suite

• Heathcare applications
• lab, rx, radiology, hospital, electronic medical
  record, cancer screening
• Insurance applications
• membership billing, claims, Medicare
• baseline
• patient master-file, practitioner master-file
• Corporate
• payroll, employee master-file, GL
• Health Research
• data warehouse, SAS
• Misc
• employee disease-exposure tracking

6
System Environment.

• System Environment
• 10 year policy of purchasing packages based on
  user-requirements, not shared technology
• Multiple vendors, hardware, databases
• MVS Master-file systems
• Highly tailored home-grown applications
• Hardware MVS, Tandem, 20-someSolaris boxes,
  Windows NT, IBM-AIX, and Dec-Alpha
• Databases Sybase ASE ASA, Oracle, MUMPS, VSAM
• All glued together with over 120 TDM / Impact
  Interfaces

7
Interface Diagram TDM Impact
8
Production Statistics 1/2
Production Servers TDM-1 TDM-2 e-BIZ-1
Daily volume 33,000 77,000 22,000
App Controllers 50 31 2
Prod Objects 75 18 2
Acquisition Aims 21 11 2 / 7 (multi-srv)
Delivery Aims 75 18 2
9
Production Statistics 2/2
Production Servers TDM-1 TDM-2 E-BIZ-1
Socket connections 71 12 9
Sybase deliveries 24 12 --
Batch/file 12 5 --
Openlink POs 3 -- --
MQ (c-based) 2 1 --
10
Types of Interfaces- examples

• Types of Interfaces at GHC
• Distribution
• Patient master-file distributes to 12
  destinations
• Collection
• Online Medical Record gets data from 15 sources
• Round-trip glue packages together
• Referral-request system sends request to Referral
  System, which sends back approval

11
Types of interfaces Distribution
12
Types of interfaces Collector
13
Types of interfaces Round-Trip
14
TDM Impact Environment

• Hardware, System Software
• TDM Impact Sun Solaris 5.6
• 2 test servers
• 2 production servers
• Production e-Biz Impact Environment
• e-Biz Impact Sun Solaris 5.8
• 1 test server
• 1 production server

15
AIMS - types

• AIMS - types
• Generic aims c-based
• 33 sockets acquisition aims
• 60 sockets delivery aims
• 3 MQ-client delivery aims
• 6 flat-file delivery aims
• Custom Parse/deliver aims
• 36 Sybase updates rpcs

16
Timeline

• Getting started May 02-Jan 03
• Reflection Dec 02- Feb
  03
• Focus and moving
  forward Feb 03 - present

17
Getting started May 02-Jan 03

• Focus was on the technical part of the project

18
Getting Ready- thru Feb 03
Project Flow

• ? New UNIX boxes no bridging
• ? Side-trip tried to get ready early, w/
  contractor help with msg-ide
• ? Attended class (too early - ? )
• ? Installed 1st release it on our own using
  documentation
• ? Trial and error 1st release
• ? Hired Sybase Consulting, mostly helped with
  msg-ide stdization
• ? Lots of help from Tech Support
• ? Trial and error 5.3.1.8
• ? Lots of help from Tech Support

19
Reflection Dec 02- Feb 03

• Began to think more like a real project, less
  like an internal IT project..

• Who are the customers
• What do they need/want
• How can we best handle the conversion of old to
  new system? 

20
Reflection technical learnings

• Much stays just the same
• TRAN-IDE MSG-IDE tools work the same (with
  improvements)
• Prod Objects need no changes (except if using
  Openlink/CL-DBI)
• Msg Aims need minor tweaks
• Most programmer-concepts stay the same tools
  change
• Authorization DB for security
• Bsmtab sfm.cfg translate to configurator
• Some are more complex
• Change-control becomes more important
• Xml cfg files must stay as a set

21
Reflection customer requirements

• Focus on customer
• Minimize unnecessary changes
• Keeping what makes sense
• Finding mental roadmaps to the known
• Keep conversion simple
• 2 new Unix boxes

22
Focus and moving forward Feb 03 - present

• Our customers
• Operations staff
• Interface-engine programming staff
• Interfacing-systems programmers/testers

23
Operations Staff responsibilities

• Monitor console for problematic aims
• Can pause, continue restart aims
• Paged if critical apps are late

24
Operations Staff similarities

• Kept sfm organization
• sfm1 sfm8
• Kept aim names
• eg csrlab1, csrsyb2
• Cross-reference document showing TDM functions
  and related e-BIZ function
• Pause/Continue
• Restart
• Login to cluster

25
Operations Staff differences

• Differences
• Global console shows 1 big vertical list, instead
  breaking up by sfm
• Sosetting up windowing is important
• To help, we created documentation for
  Trouble-spots

26
Operations sfm console Instructions 1/2
To view the destinations /sources for an sfm  Find the sfmfor each SFM, youll see an application controller level (for example ac_sfm6) and the sfm level (sfm6). If you dont see the levels, you might need to expand them ( -)   Left-click on the sfm-level to highlight it. On the right side of the MMC console youll see the Sources/Destinations view.   If you dont see the Sources/Destinations view, right-click on the sfm-level. This brings up a drop-down box. Go to the VIEWs option and select Sources/Destinations.   This shows the same information as in the TDM console whether a destination is active (blue/green) or problematic (red), and details about when the last transaction was processed,
27
Operations sfm console Instructions 2/2
To continue a destination Find the sfmfor each SFM, youll see an application controller level (for example ac_sfm6) and the sfm level (sfm6). If you dont see the levels, you might need to expand them ( -)   Confirm that its down by looking at the Sources/Destinations view as above.  On the left side of the MMC console, under the sfm directory, left-click to highlight the desired destination. Then right-click to bring up a drop-down box. Go to the ACTION option and pick continue.
Everything is PROBLEMATIC Contact the IE Team. We may need to restart the cluster, which requires us to login to the UNIX box itself.
28
Operations Troubleshooting 1/2
Message about having to login to do an action You need to login to the cluster in order to take any actions. (You can view without logging in.) Expand the directory structure to the cluster level (prd1), if its compressed Left-click on the clustername to highlight it. Right-click to bring up the dropdown-box. Select ACTION, login Enter the operatec userid and password and click OK If you have problems, contact the IE Team for help.
The ACTION you need is grayed out The IE Team hasnt setup permissions for operatec to do this function for this aim. Contact the IE Team so we can add this to the Authorization Database
29
Operations Troubleshooting 2/2
MMC Console hangs or doesnt reply Try exiting and restarting the global console. Exit Global Console-MMC. You might need to ctl-alt-delete to the Task-Mgr and stop MMC that way. Then click on Global Console ICON to restart. Youll probably need to re-login (though sometimes the login stays active)
EBIZ-5 icon is gone entirely If PC was fully re-imaged,EBIZ-5 may have been lost as EBIZ-5 did not come as part of RADIA push. (EBIZ-5 is currently hand-installed plans are to make it Radia-pushable fall-2003) Contact the IE Team. IE Team will provide CD and instructions, or will come to AOC to reinstall.
30
Focus and moving forward Feb 03 - present

• Our customers
• Operations staff
• Interface-engine programming staff
• Interfacing-systems programmers/testers

31
Interface Engine Programming Staff

• Keep sfm organization the same, for familiarity
• Kept application controllers organized by
  system to ease managing of connections
• all the DataWarehouse aims ACSYB1,
• all the LAB-system aims ACLAB1,
• all the CICS feedsACCICS1
• Kept concept of generic aims to reduce
  programming, testing and ongoing maintenance time
• Made cheat-sheets to aid staff
• for implementation workflow,
• For config-file maintenance
• For batch command-line utilities

32
Focus and moving forward Feb 03 - present

• Our customers
• Operations staff
• Interface-engine programming staff
• Interfacing-systems programmers/testers

33
Interfacing-systems programmers/testers

• Respect their schedules/priorities by offering
  ability to do staggered installs
• Use generic pre-tested aims for generic
  connections (socket, MQ, flat files)
• Try to minimize new-code that requires extensive
  test-plans, user commitment, etc
• Especially for our 40 custom parse/deliver aims
  that write to Sybase databases

34
What didnt work / what was hard to do 1

• What was the hardest part?

• If only Id realized that, in general, the
  functions stayed the same. and only the
  presentation-layer and the underlying technicals
  changed!

35
What could have made upgrade easier -- 1

• Similarities/Differences between TDM-Impact
  e-Biz

Component TDM-Impact EBIZ-Impact
Security/access cntl security broker ASA Authorization DB Authorization Tool ( odbc setup on PC)
Monitoring control - console SFM Console console commands MMC Global Console command SNMP (and odbc setup on server)
Monitoring control command line UNIX Command w/ root access Ims cnc commands no root access
Configuration Mgt Vi, notepad, ftp (bsmtab, sfm.cfg, linktab) Configurator, ftp (set of files-change mgt)
Development tran-ide, msg-ide, c-based aims Tran-ide, msg-ide, rare use of c-based aims
36
What didnt work/ was hard to do 2

• Being an early adopter was hard

• Documentation incomplete, sometimes misleading
• Code had some bugs
• Global console wasnt ready for operations staff
  to use.
• Didnt have a mental-roadmap of how e-biz worked.
  (wish I had attended Aishas Techwave 2002
  presentation)

37
What didnt work/ was hard to do 3

• New config file sets require careful coordination

• What if we both want to work on the config files
  at the same time?
• How do I make sure Ive got the most recent set?
• I sure hope no one edits the XML file outside the
  Configurator
• Wish I had kept root access for when my
  colleague forgot to chmod gw

38
Keeping what worked before

• We wanted to keep what worked well for us in
  TDM-4
• Standardize, standardize, standardize!
• Generic, reusable aims
• Keeping our naming standards from TDM-4
• Keeping our sfm/app-controller structures from
  TDM-4

39
What didnt work/ was hard to do 4

• Figuring out how to keep our structure

• Side-trip to msg-aims showed us how much we liked
  our existing setup
• having generic aims
• Multiple aims organized under a controller
• Can we re-use aims in e-BIZ5?
• Sybase Consulting showed us the way we finished
  the code

40
Building re-usable generic aims

• Building re-usable generic aims
• Requires some modifications to get them to work
  then can reuse.
• Use standard pipe names delpipe, acqpipe
• Need to handle the customizations by writing odl
  code
• Pick up name of aim from clGetargv
• For acquisition, get sfm and name of engine from
  Command-line in cfg file
• Build lastID file using serial-nbr from servproc
  parameter
• .sample snips follow..
• Use just one copy of aim/Project File point to
  it from cfg file

41
Re-usable aims socket acquisition

• 
  Acquisition AIM

Note that the acq_hl7.prj file sits in a
separate directory, to be shared by all
acquisition aims
42
Re-usable aims socket acquisition code snip 1

• .NNSY_ROOT/config/prd1/acq/acq_hl7.prj

Code snip showing how to get aim name note that
arg1 (clGetArgv(0,xx) is always aimname
//Set source to clGetArgv to pickup aimname from
Config file clGetArgv(0,Source)
Code snip showing how to get nbr of command line
args In this example, there are 3 args aimname,
sfm flavor, ENGINE
//Set rv to the number of lines rvclGetArgc()

43
Re-usable aims socket acquisition code snip 2

• .NNSY_ROOT/config/prd1/acq/acq_hl7.prj

Code snip showing how to pickup SFM flavor
if (clGetArgc() gt2)
clGetArgv(1,sfmFlav_in) erm (" AIMs
Cmd line parm1(sfmFlav_in)s!!!",Source,
sfmFlav_in) // Convert to Integer sfmFlavatoi_st
andin(sfmFlav_in) erm (" after integer
conv") if (sfmFlav -1 )
erm("")
erm(" ERROR-SFM sfmFlavor not numeric
") erm(""
) return 0
44
Re-usable aims socket acquisition code snip 3

• .NNSY_ROOT/config/prd1/acq/acq_hl7.prj

Code snip showing how to make a debug-file unique
to each aim
// Check debug debugON 0 if
(debugFlag.load(Source ".bug") ! -2)
erm(".....DEBUG IS ON.....") debugON 1
else erm(".....DEBUG IS OFF.....")
(the variable Source was set to aimname in
previous slide)
45
Re-usable aims socket acquisition msg-ide

• .NNSY_ROOT/config/prd1/acq/acq_hl7.prj

• Use standard name for Pipe in MSG-IDE, to tie to
  configurator ODL properties

46
Re-usable aims socket Delivery

• Delivery
  AIM - socket

Note that the del_hl7.prj file sits in separate
directory, to be shared by all delivery-socket
aims
47
Re-usable aims socket delivery code snip 1

• .NNSY_ROOT/config/prd1/del/del_hl7.prj

Code snip showing how to get aim name note that
arg1 (clGetArgv(0,xx) is always aimname
//Set source to clGetArgv to pickup aimname from
Config file clGetArgv(0,Source)
48
Re-usable aims socket delivery code snip 2

• .NNSY_ROOT/config/prd1/del/del_hl7.prj

. //Build name of lastID file from sfm
flavor fileNm "lastID." flvStr ID.set
("") Res ID.load(fileNm) if (Res
(-2)) Stat0 '\000' ID.save(fileNm) erm(
" Warning lastID.s file not found
",fileNm) else erm (" getting
SerialNbr from lastid file s.",fileNm) Stat
ID erm ("SERVAYT SerialNbr from
files is serial_nbrs.",fileNm, Stat)
lastID filename lastID.13
49
Re-usable aims socket delivery code snip 3

• .NNSY_ROOT/config/prd1/del/del_hl7.prj

Code snip showing how to make a debug-file unique
to each aim
// Check debug debugON 0 if
(debugFlag.load(Source ".bug") ! -2)
erm(".....DEBUG IS ON.....") debugON 1
else erm(".....DEBUG IS OFF.....")
(the variable Source was set to aimname in
previous slide)
50
Re-usable aims socket delivery msg-ide

• .NNSY_ROOT/config/prd1/del/del_hl7.prj

• Use standard name for Pipe in MSG-IDE, to tie to
  configurator ODL properties

51
Re-usable aims flat-file-out Delivery

• Delivery AIM
  flat file out

Note that the del_file.prj file sits in
separate directory, to be shared by all flat-file
delivery aims
52
Re-usable aims socket delivery code snip 1

• .NNSY_ROOT/config/prd1/del/del_hl7.prj

Code snip showing how to pick up outbound
filename from command line
rv clGetArgv(0, Source) rvclGetArgc() erm
("clinit - clGetArgcd",rv) if (rv 1)
//1 indicate no filename specified //
pickup output filename from aimname
outputFileName Source ".dat" else
//pickup filename from Command line rv
clGetArgv(1, outputFileName) erm("
.....(command line param-1)gtgts",
outputFileName) erm(" pgms gtgt
output file name is s", Source,
outputFileName)
53
Re-usable aims socket delivery code snip 2

• .NNSY_ROOT/config/prd1/del/del_file.prj

Code snip showing how to write to flatfile
(append)
Databuf.debug() // print databuf to
xlog  Databuf.add("\015") // Add a CR (0d) to
the end Databuf.add("\012") // Add a LF (0a)
to the end   rvDatabuf.write(outputFileName)
// write does an append erm ("
outputfiles. rvd", outputFileName,
rv)   if(rv gt 0) //databuf.write returns nbr of
bytes written, or negative unix err code
erm (" ..... starting to save serial_nbr in
lastid file ") Result
LastID.save(lastID_filename) return
1 else rm("
") erm("
Couldn't write to file (unix-rvd), abending
with 2 return code", rv) erm("

") return -2
54
Re-usable aims flatfile delivery

• .NNSY_ROOT/config/prd1/del/del_file.prj

• Specify servayt servproc to get record from sfm

55
Re-usable aims flatfile delivery

• .NNSY_ROOT/config/prd1/del/del_file.prj

MSG-IDE No clcomm, no ACK logic, no clpipe, etc

• Configurator No pipes/resources needed!

56
What was hard to do 5

• Figuring out what do do with our c-based
  parse/deliver-to-sybaseDB aims
• Choices
• 1) make them thread safe
• Too scary how to confirm they are thread-safe.
  Too labor-intensive
• 2) convert to tran-ide for parse build stored
  proc params msg-ide for cl-dbi calls.
• A lot of work and testing for us and downstream
  programmers, plus cl-dbi had bugs to start.
  Thruput unknown
• 3) make them stand-alone pgms called from msg-ide
• Less work and testing once std approach built
• Thruput unknown

57
What did we do for our C-based sybase aims?

• Kept the business-logic
• Parse
• Build sybase procedure calls/parameters
• Made pgms standalone by
• Removing servayt/servproc logic
• Wrapping std main logic around them
• Builr msg-ide to call the standalone pgm
• Passing Hl7 msg return code via a file
• Thanks to Sybase Consulting for this idea
• Thru-put is about the same as before
• approx 1 hl7 msg per second

58
What was hard to figure out 6?

• Ports?
• Arent they just for delivering acquiring?

59
What could have made upgrade easier?

• Ports? How many? How do I decide on a port ?

Clusters App controllers?
Acquisitions?
Deliveries?
SNMP emanate?
ASA authorization DB?
60
SNMP Cluster monitoring.

• Ports for SNMP Cluster monitoring
• 1 set of 2 ports for each group of clusters you
  want to see together on global console
• At GHC..
• Test (tst1, kkem, rwong)
• Production (prd1 port xxxx)
• QA (prd1x port xxxx)

61
SNMP setup 3 places

• Script to startup SNMP emanates
• 2 ports sequential,
• Set these up in SNMP startup scripts
• cluster-level in cluster startup script
• Global Console create agent each PC

62
SNMP setup 3 places

• Script to startup SNMP emanates
• 2 ports sequential,
• Set these up in SNMP startup scripts
• cluster-level in cluster startup script
• Global Console create agent each PC

63
SNMP setup 3 places

• Script to startup SNMP emanates
• 2 ports sequential,
• Set these up in SNMP startup scripts
• cluster-level in cluster startup script
• Global Console create agent each PC

64
Setup Ports for Authorization DB 3 places

• Authorization DB
• 1 port per cluster for those you want to monitor
  via global console
• Server setup
• Set this up on odbc.ini file for the cluster
• Then use the name to reference it in the
  configurator

• SNIP from odbc.ini file /opt/cnet/Impact-5_3

65
Setup Ports for Authorization DB 3 places

• Authorization DB
• Server setup (cont)
• Set this up on odbc.ini file for the cluster
• Then use the DSN name to reference it in the
  configurator

66
Setup Ports for Authorization DB 3 places

• Authorization DB
• Try automating by writing a Windows-cmd Script
• (no line-breaks, just let it flow)

• _at_echo OFF
• setlocal
• set NNSY_ROOTC\Sybase\ImpactClient-5_3
• set PATHNNSY_ROOT\bin\wrappersNNSY_ROOT\asa7
  03\binPATH
• set MYDBimpdb_ghcdev_prd1x
• set MYHOST199.99.99.186
• set MYPORT4364
• dbdsn -ws MYDB_dsn -c "DBNimpactENGMYDBAST
  OPNoINTNoDBGYESLOGc\sybase\odbcsetup_MYDB
  .txtDMRFNOLINKSSharedMemory,TCPhostMYHOST
  portMYPORT"

67
Setup Ports for Authorization DB 3 places

• Authorization DB
• Or set up ODBC manually on each PC

68
Setup Ports for Authorization DB 3 places

• PC Authorization DB
• So, once odbc entry is setup for each
  authDB,when you login to Authorization tool, you
  can pick from the odbc datasources on your PC

69
Ports for Cluster and Controllers

• Ports for Cluster Controllers
• 1 port for cluster, plus one for each controller
• This info is entered into Configurator at Cluster
  level
• Can set a range of ports
• These seem to be assigned dynamically, but you
  can see them on global console.

70
Ports for Cluster and Controllers

• Ports for Cluster Controllers
• These seem to be assigned dynamically, but you
  can see them on global console. (ports 7500-7550
  setup as initial range)

• Put picture f global onsole with ports here

71
Ports, continued

• No easy way to see list of all ports
• Spread out on .cfg files in multiple directories
• In configurator, must open each aim go down
  levels
• Aids
• Use a worksheet to plan keep for documentation
• Enter ports on /etc/services file (unix)
• Consider turning tracking off and moving all
  files up to App Controller level
• Consider writing script to read thru .cfg files
  and extract host port info.

72
Port Worksheet.

• Plan out your ports, clusters, global console
  groupings

Description Cluster a Cluster b Cluster c
Box hostname Gh-ieprod Gh-ietest Gh-ie-test
Clustername prd1 prd1x tst1
Box IP address  xxx.xx.xxxx  xxxxxx  xxxxxx
Auth DB (odbc.ini)  4361  4362  4363
SNMP emanate (snmp script)  6001/6002 6003/6004 6003/6004
Cluster Controller Ports - range or indiv nbrs (configurator-cluster level)  7500-7550  7500-7550  7551-7559
Acquisition-aim ports      
Aim 1  12000  12000  15000
Aim 2  12001  12001  
Aim 3  12003  12003  
Etc
73
Other things to think about

• And a few miscellaneous things..

74
Other things that worked well

• Turning tracking off to pull all files up into
  the Application-controller level.
• Adding a chmod gw to the deploy and harvest
  scripts
• Writing scripts to setup odbc on client hl7
  respository authorization DB
• Take advantage of multiple clusters on 1 box QA,
  RD, Development, etc
• Cloning is easy
• Grab the set of config files.
• Open up in configurator and change cluster name,
  ports, authdb
• Deploy to new cluster directory
• Copy scripts, prod-objects, msg aims, etc
• Start it up

75
Startup Scripts

• Scripts youll need to get things running on
  server
• Setup SNMP by running a script at system startup
• Script to startup Authorization DB (one per
  cluster)
• Startup shutdown scripts
• Log rollover scripts or use the auto roll
• On client
• ODBC setup scripts for HL7 repository
  Authorization DBs

76
Resources
What really worked for us

• Resources
• Migration Class - http//www.sybase.com/detail?id
  1018591
• Sybase Tech Support
• EBIZ newsgroup http//webforums.sybase.com/nntp/n
  d000103.nsf/(All)?OpenView
• Documentation (though it needs upgrading)
• Consulting assistance

77
Newgroup sample page
78
Sybase support solved cases

• Screenshot of solved cases
• Requires login/password for Sybase support
• Note keywords you may have to search a bit

79
Sybase Tech Support
What really worked for us

• Sybase Tech Support worth the
• Fast response to new cases
• Can enter cases online! Or by phone
• Relentless follow-up til solved
• Online posting of others solved cases for query
• Knowledge of latest fixes
• Knowledge of already-discovered problems
• Availability of developer documentation to fill
  gaps in official documentation

80
Future Steps

• Future Steps
• Finish the conversion!
• MQ series replacing sockets using MQClient
  IDE
• Finish CL- DBI database interfaces-objects
  template
• XML (HL7-version 3)
• Maybe add HIPAA support, replacing Mercator

81
Wish List

• What wed like Sybase to add to the product

82
Questions?
83
The END

• Thanks!