Quick and Easy OpenEdge AI File Management

1
Quick and EasyOpenEdge AI File Management
PAUL KOUFALIS
PRESIDENT PROGRESSWIZ CONSULTING
2
Progresswiz Consulting

• Based in Montréal, Québec, Canada
• Providing technical consulting in Progress,
  UNIX, Windows, MFG/PRO and more
• Specialized in performance tuning, system
  availability and business continuity planning
• and security of Progress-based systems

3
Agenda

• Introduction
• After-Imaging (AI) Basics
• Pre-OpenEdge Implementation Challenges
• The New AI File Management Daemon
• Step-by-Step Implementation
• Understanding the Results
• A Few Final Steps
• Questions

4
Introduction

• What is After-Imaging?
• Quick Definition
• A logging system that stores all information
  required to reapply all changes made to a
  database between two points in time

5
Introduction
OpenEdge AI Gremlins Hard at Work!
6
Introduction

• Recipe for Recovery
• One backup
• All AI files generated since that backup
• Result
• One fully recovered database with near-zero data
  loss
• Near-zero because open transactions will be
  rolled back

7
Introduction

• Why bother?
• Lets see, because you like your job?
• WaitwaitI knowdata in a database is like,
  important, right?

8
Introduction

• But seriously folks
• Most business only backup once per day
• Without AI, any changes made since the last
  backup are lost
• Best of all, its 100 free
• Both Workgroup and Enterprise
• And Personal DB too!

9
After-Imaging (AI) Basics

• Basic transaction process
• Begin Transaction
• Make some change
• Before and after value written to BI
• Before and after value written to AI
• Repeat
• End Transaction
• Note that writes to the data files are not in the
  list
• It will happen eventually

10
After-Imaging (AI) Basics

• What happens to all these BI/AI notes?
• OpenEdge automatically manages the reuse of BI
  file space
• AI files must be archived before allowing
  OpenEdge to reuse them

11
After-Imaging (AI) Basics
Data Files (.d)
BI File
AI File 1
Full
AI File 2
Full
. . .
AI File n
Full
User Processes
Now What?
12
After-Imaging (AI) Basics
AI File 1
Full
Empty
Safe Storage
AI File 2
Full
Copy/Archive
Empty
. . .
AI File n
Full
Empty
13
After-Imaging (AI) Basics

• Sounds simple enough
• When full, copy/archive AI files to a safe
  location
• Mark them as empty so OpenEdge can reuse the
  space in the files
• Whats the problem?

14
Pre-OpenEdge Implementation Challenges

• Problem 1 Scripting
• Commands are simple enough
• rfutil sports C aimage extent full
• rfutil sports -C aimage query extnum by name
  sports.a4
• cp sports.a4 /archive/sports.extnum
• rfutil ltdbnamegt -C aimage empty
• But adding the logic around them requires work

15
Pre-OpenEdge Implementation Challenges

• Scripts to automate AI file management can be as
  simple or complex as you like
• Depends how smart you want them to be
• Reality Not that difficult
• Plenty of example scripts available online
• Search the peg (www.peg.com)

16
Pre-OpenEdge Implementation Challenges

• Problem 2 Performance Impact
• Ye Olde Days of Yoreyes
• Today Please
• The effect is minimal unless you have really old
  junky hardware
• Of course, there are exceptions
• But if youre that busy you really need AI!

17
Pre-OpenEdge Implementation Challenges

• Resist the If it aint broke mentality
• If you are not running AI your system IS broken
• With modern hardware the performance issue is
  moot
• And now with the AI File Management Daemon, so is
  the scripting issue

18
The New AI File Management Daemon

• Available as of 10.1A
• Automates
• Rotation and reuse of AI files
• Archiving of full extents
• The ltahemgt intelligent ltahemgt naming of archived
  AI extents
• More on this later
• Its about time!!!

19
The New AI File Management Daemon

• Can be activated in one of two modes
• Time interval AI file rotation occurs every x
  seconds
• Min 2 minutes
• Max 24 hours
• Full AI File Rotation occurs when the currently
  active AI file is filled
• Typically used with fixed length AI extents

20
The New AI File Management Daemon

• Two configurable parameters
• -aiarcdir Directory where full AI extents will
  be copied
• Can specify multiple directories in case writes
  to the first fail
• -aiarcinterval Time between AI file rotations
• If not specified, rotation occurs when extent
  fills

21
How it Works

1. Daemon wakes up and checks for full AI files or
   elapsed aiarcinterval
2. Full AI files are copied to the archive directory
   and renamed appropriately
3. Once successfully copied, AI files are marked as
   empty and available for reuse

AI Daemon
DB
AI Files
Archived AI Files
BI File
22
Step-by-Step Implementation

• Four ways to activate AI and AI File Mgmt
• Note Online activation as of 10.1B

After-Imaging AI File Management
OFFLINE OFFLINE
ONLINE OFFLINE
OFFLINE ONLINE
ONLINE ONLINE
23
Add AI Extents

• Can be done offline or online as of 10.1A
• Create add.st file
• Three variable length AI files in /db/prod
• a /db/prod
• a /db/prod
• a /db/prod
• Apply to database
• prostrct add sports add.st
• prostrct addonline sports add.st

24
Enable AI AI Mgmt Daemon Offline

• Take a full backup of your database
• probkup sports /backup/sports.probkp
• Enable after-imaging
• rfutil sports C aimage begin
• Enable AI Management Daemon
• rfutil sports C aiarchiver enable

25
Enable AI AI Mgmt Daemon Offline

• Start your database
• Be sure to include the aiarcdir and, if
  required, -aiarcinteval startup parameters
• proserve sports pf sports.pf
• aiarcdir /aiarch,/aiarch2
• -aiarcinterval 900

26
Enable AI AI Mgmt Daemon Online

• Enabling AI the AI File Management Daemon
  offline required three distinct steps
• Backup
• Enable AI
• Enable AI File Management
• As of 10.1B, do it all in one command

27
Enable AI AI Mgmt Daemon Online

• probkup online sports/backup/sports.probkp
• enableai
• enableaiarchiver
• -aiarcdir /aiarch,/aiarch2
• -aiarcinterval 900

28
Mixing Online and Offline

• Parameters enableai and enableaiarchiver are
  optional
• Can use one or other in probkup online cmd
• Of course, AI must be enabled in order to enable
  AI Archiver

29
Mixing Online and Offline

• Example
• Backup offline
• probkup sports /backup/sports.probkp
• Enable AI offline
• rfutil sports C aimage begin
• Start DB
• proserve sports pf sports.pf
• Enable AI Archiver online
• probkup online sports/backup/sports.probkp
• enableaiarchiver
• -aiarcdir /aiarch,/aiarch2
• -aiarcinterval 900

30
Disabling the AI Archiver Daemon

• To permanently disable the AI Archiver
• rfutil sports C aiarchiver disable
• To stop the daemon temporarily without disabling
  the functionality
• I.e. for maintenance
• rfutil sports C aiarchiver end

31
Disabling the AI Archiver Daemon

• No documented way to restart AI Archiver
• I.e. there is no aiarchiver begin
• Will start automatically at next proserve
• Or run the command
• _mprshut sports -C aimgt

32
AI Archiver Status

• Activated?
• Use proutil ltdbnamegt -C describe
• Database Features
• ID Feature Active
  Details
• ---- --------------------------------- ------
  -------
• 8 After Image Mangement/Archiver Yes
• 9 64 Bit DBKEYS Yes
• 10 Large Keys Yes
• 11 64 Bit Sequences Yes

33
AI Archiver Status

• Running?
• Look in promon RD 1 4 1
• 10/22/08 Status All Processes
• 71724
• Usr Name Type Wait Trans id Login
  time
• 0 Paul Kou BROK -- 0 0
  10/22/08 1709
• 5 AIMD -- 0 0
  10/22/08 1709
• 6 Paul Kou MON -- 0 0
  10/22/08 1716

34
Changing Parameters Online

• Both parameters (-aiarcdir and aiarcinterval)
  can be changed online
• rfutil sports C aiarchiver setdir /aiarchnew
• rfutil sports C aiarchiver setinterval 1800
• The second option is particularly useful
• Decrease interval during high activity
• Increase interval during quiet times

35
Understanding the Results

• Enabling AI Archiver affects three output
  streams
• DB Log File
• AI Archive output directory (-aiarcdir)
• AI Archiver log
• New

36
Database Log File

• AI Archiver entries show up as AIMGT
• Example
• 2008/10/22_at_172716.078-0400 P-5316
  T-912 I AIMGT 5 (13213) A new archive
  interval 120 has been set.
• Please dont go blind!
• AIMGT 5 (13213) A new archive interval 120 has
  been set.

37
Archived AI Files

• The good
• All the information you need is stored in the
  file name
• The bad
• All the information you dont need is stored in
  the file name
• The ugly
• C!TEMPoewrkdbpk.20081022.164924.00002.0000001
  7.pk.a2
• Now thats a file name!

38
Archived AI Files

• C!TEMPoewrkdbpk.20081022.164924.00002.00000017
  .pk.a2
• a b c d e
  f
• The full file path of the source DB
• The date of the last backup
• The time of the last backup
• The sequence number of the last backup
• The AI sequence number
• The original AI file name

39
AI Archiver Log

• AI Archiver creates its own log
• ltDBNAMEgt.archival.log
• Whoa!!
• A little bit challenging to read

40
AI Archiver Log

• 0001,C\TEMP\oewrk\db\pk,20081022,181456,20081022,
  164924,2,36,C\TEMP\oewrk\db\pk.a3,C\TEMP\oewrk\d
  b,C\TEMP\oewrk\db\C!TEMPoewrkdbpk.20081022.16
  4924.00002.00000036.pk.a3
• 0032,C\TEMP\oewrk\db\pk,20081022,181652,1,3,20081
  022,181652,38,1,pk.bkp.3,BACKUP_SET_20081022_18165
  2
• 0001,C\TEMP\oewrk\db\pk,20081022,181656,20081022,
  164924,3,37,C\TEMP\oewrk\db\pk.a1,C\TEMP\oewrk\d
  b,C\TEMP\oewrk\db\C!TEMPoewrkdbpk.20081022.16
  4924.00003.00000037.pk.a1
• Black text is AI file rotation log entry
• Blue text is a probkup
• Note the backup timestamp and sequence number
  changed in AI file rotation entry

41
Careful!

• AI Archive files and log show AI file 37
  associated with backup 3
• C\TEMP\oewrk\db\C!TEMPoewrkdbpk.20081022.164
  924.00003.00000037.pk.a1
• Intuitively believe AI 37 should be applied to
  backup 3
• Incorrect
• AI 37 is last AI file before backup 3

42
Carefulcontinued

• From DB Log File
• Full backup started.
• Switched to ai extent C\TEMP\oewrk\db\pk.a2.
• This is after-image file number 38 since the last
  AIMAGE BEGIN
• Therefore AI file 38 is the first AI file to
  apply after restoring a backup
• Not a problem
• Rfutil utility will complain
• expected file 38

43
Carefulcontinued

• UNC pathnames in -aiarcdir
• I.e -aiarcdir \\bkpserver\aidir
• There was a bug on Windows
• Not sure if fixed in 10.1C

44
A Few Final Steps

• AI Archive Directory
• Ideally, directory mounted from another server
• NOT in the same building as production
• Cleanup AI files older than x days
• Delete files older than x days after saving to
  tape
• 5 is good starting number
• You will have to script this yourself

45
A Few Final Steps

• Monitoring
• One more thing to monitor
• _mprshut ltdbnamegt -C aimgt
• Log File Maintenance
• ltDbnamegt.archival.log
• Archive and truncate periodically

46
Thats It!

• Youre done! Congratulations!
• Restores and roll-forwards are easy
• AI Files are semi-intelligently named
• AI Archiver log file shows which files go with
  which backup
• Just remember to cleanup AI Archive directory

47
Questions
Questions?
48
Progresswiz Consulting

• Questions or comments? Send me an email
• pk_at_progresswiz.com