Disk%20I/O%20Performance%20focusing%20on%20Caching

1
Disk I/O Performancefocusing on Caching
Guy Peleg President Maklee Engineering guy.peleg_at_m
aklee.com
2
Why should I care?

• Computers became very fast over the last decade
• I/O performance remained (almost) the same
• Disk I/O is an extremely! slow operation

3
Why should I care?

• 1.6Ghz rx7640 single cell
• CPU cycle is 0.5 ns
• Memory latency is 250 ns
• EVA5000 read latency 5 10 ms
• Now lets change the scale and assume 0.5 ns 1
  sec

4
Why should I care?

• 1.6Ghz rx7640 single cell
• CPU cycle is 1 second
• Memory latency is 8.3 minutes
• EVA5000 read latency 23 days !!!!!

5
Why should I care?

• Test program written in C
• Creates a 1GB container file
• Opens the newly created file
• Positions at EOF
• Writes a small record
• Closes file
• 1.6Ghz rx2600
• OpenVMS V8.3-1H1

6
Why should I care?
88 difference
Seconds to complete the test program less is
better
7
I/O Best Practices

• The fastest I/Os are those that are avoided
• OpenVMS info desk Oct. 2004
• Cache as much as you can
• Caching was invented to attempt and reduce the
  need for physical I/Os.
• This session is all about providing techniques
  for avoiding/reducing I/O

8
High Water Marking

• The file highwater mark attribute guarantees that
  a user can not read data that was not written by
  the user.
• The init command turns this feature on by default
• Tradeoff between security and performacne
• INIT/NOHIGH
• SET VOLUME/NOHIGH

9
The impact of highwater mark
Seconds to create a container file less is better
10
Data caching

• Data could be cached by
• The Application
• RMS (or Database cache)
• OpenVMS file caching (XFC)
• Hardware (disk controller)
• No one knows your application better than you !!
• Try to satisfies I/O requests as close to the
• application as possible

11
XFC

• eXtended File Cache
• Introduced with OpenVMS V7.3
• Replaces VIOC
• 64 bit
• No closed file limit
• Will cache I/Os larger than 35 blocks

12
XFC

• Supports readahead
• (currently) does not support writeback
• May grow up to 50 of physical memory
• Grows and shrinks according to free physical
  memory.
• May be forced to return memory instantly
• No caching for files opened for write cluster
  wide

13
XFC Reserved Memory

• Using reserved memory for XFC is slightly faster
• Reserved Memory for Permanent memory
• SYSMANgt RESERVED_MEMORY ADD VCCMIN_CACHE_SIZE/SIZ
  E1500 /ALLOCATE/NOPAGE_TABLES/NOZERO/NOGLOBAL
• VCC_MAX_CACHE must be greater than the reserved
  memory size.

14
XFC SYSGEN paraeters

• mc sysgen sh vcc
• Parameter Name Current Default
  Min. Max. Unit Dynamic
• -------------- ------- -------
  ------- ------- ---- -------
• VCC_FLAGS 2 2
  0 -1 Bitmask
• VCC_MAXSIZE 6400 6400
  0 3700000 Blocks
• VCC_MAX_CACHE -1 -1
  0 -1 Mbytes D
• VCC_MAX_IO_SIZE 256 127
  0 -1 Blocks D
• VCC_MAX_LOCKS -1 -1
  50 -1 Locks D
• VCC_READAHEAD 1 1
  0 1 Boolean D
• VCC_WRITEBEHIND 1 1
  0 1 Boolean D
• VCC_WRITE_DELAY 30 30
  0 -1 Seconds D
• VCC_PAGESIZE 0 0
  0 -1 D
• VCC_RSVD 0 0
  0 -1 D

15
show memory /cache/full

• sh mem/cache/full
• System Memory Resources on
  13-MAY-2008 124419.80
• Extended File Cache (Time of last reset
  8-MAY-2008 225733.05)
• Allocated (GBytes) 1.35 Maximum
  size (GBytes) 4.00
• Free (GBytes) 0.21 Minimum
  size (GBytes) 0.00
• In use (GBytes) 1.13
  Percentage Read I/Os 26
• Read hit rate 95 Write
  hit rate 0
• Read I/O count 2901784 Write
  I/O count 8118032
• Read hit count 2781426 Write
  hit count 0
• Reads bypassing cache 180 Writes
  bypassing cache 6860600
• Files cached open 848 Files
  cached closed 993
• Vols in Full XFC mode 0 Vols in
  VIOC Compatible mode 3
• Vols in No Caching mode 0 Vols in
  Perm. No Caching mode 0
• I/O Statistics - Distributions (MAX_IO_SIZE 127)
• -------------------------------------------------
• Transfer Size Reads Read Hits Writes
• 1 Block IO 193447 191602 435529

16
Show memory /cache/full

• 27 Block IO 4670 4655 14815
• 28 Block IO 5657 5625 15013
• 29 Block IO 5218 5192 14984
• 30 Block IO 4717 4696 14841
• 31 Block IO 4708 4679 14344
• 32 Block IO 527966 467655 996501
• 33 Block IO 48 33 2269
• 63 Block IO 466 432 6366
• 64 Block IO 1075132 1065615 22811
• 65 Block IO 2 0 1144
• 124 Block IO 200562 190862 193467
• 125 Block IO 1 0 493
• 126 Block IO 1 0 504
• 127 Block IO 0 0 512
• gt127lt256 Block IO 21 263932
• gt255 Block IO 49 176683

17
Forcing XFC to shrink

• show mem/cache
• System Memory Resources on
  13-MAY-2008 124254.59
• Extended File Cache (Time of last reset
  13-MAY-2008 101301.11)
• Allocated (MBytes) 222.32 Maximum
  size (MBytes) 4096.00
• Free (MBytes) 0.19 Minimum
  size (MBytes) 3.12
• In use (MBytes) 222.12
  Percentage Read I/Os 87
• Read hit rate 29 Write
  hit rate 0
• Read I/O count 90334 Write
  I/O count 12864
• Read hit count 26537 Write
  hit count 0
• Reads bypassing cache 589 Writes
  bypassing cache 9969
• Files cached open 383 Files
  cached closed 631
• Vols in Full XFC mode 0 Vols in
  VIOC Compatible mode 2
• Vols in No Caching mode 0 Vols in
  Perm. No Caching mode 0
• SYSBCKgt mc sysgen
• SYSGENgt SET VCC_MAX_CACHE 100
• SYSGENgt W A
• SYSGENgt EXIT

18
Forcing XFC to shrink

• sh mem/cache
• System Memory Resources on
  13-MAY-2008 124322.10
• Extended File Cache (Time of last reset
  13-MAY-2008 101301.11)
• Allocated (MBytes) 95.63 Maximum
  size (MBytes) 100.00
• Free (MBytes) 0.02 Minimum
  size (MBytes) 3.12
• In use (MBytes) 95.60
  Percentage Read I/Os 87
• Read hit rate 29 Write
  hit rate 0
• Read I/O count 90391 Write
  I/O count 12894
• Read hit count 26584 Write
  hit count 0
• Reads bypassing cache 589 Writes
  bypassing cache 9996
• Files cached open 383 Files
  cached closed 181
• Vols in Full XFC mode 0 Vols in
  VIOC Compatible mode 2
• Vols in No Caching mode 0 Vols in
  Perm. No Caching mode 0

19
Cache pollution

• The Cache is a limited resource
• Avoid polluting the cache
• OpenVMS attempts to reduce pulltion
• VMS BACKUP
• Caching disabled for all backup IO
• Some utilities are not cache safe
• COPY
• CONVERT
• Consider disabling caching temporarily
• vcc_max_io_size 0
• mount/nocache
• set file/cacheno
• OpenVMS provides DCL commands to view the
  contents of the cache

20
Free Hot File Tracking Utility

• sh mem/cache(volume,topqio)
• System Memory Resources on
  26-APR-2008 013915.03
• Extended File Cache Top QIO File Statistics
• _1DGA242 (DISKSEC6), Caching mode is VIOC
  Compatible
• _1DGA242VMSCOMMON.SYSEXERIGHTSLIST.DAT1
  (open)
• Caching is enabled, active caching mode is Write
  Through
• Allocated pages 9 Total QIOs
  107
• Read hits 92 Virtual
  reads 107
• Virtual writes 0 Hit rate
  85
• Read aheads 0 Read
  throughs 107
• Write throughs 0 Read
  arounds 0
• Write
  arounds 0
• _1DGA242VMSCOMMON.SYSEXEVMSOBJECTS.DAT2
  (open)
• Caching is enabled, active caching mode is Write
  Through
• Allocated pages 0 Total QIOs
  9

21
Free Hot File Tracking Utility

• _1DGA142 (DISKMVMS), Caching mode is VIOC
  Compatible
• _1DGA142VMSCOMMON.SYSLIBDECCSHR.EXE1
  (open)
• Caching is enabled, active caching mode is Write
  Through
• Allocated pages 303 Total QIOs
  1646
• Read hits 1561 Virtual
  reads 1646
• Virtual writes 0 Hit rate
  94
• Read aheads 0 Read
  throughs 1642
• Write throughs 0 Read
  arounds 4
• Write
  arounds 0
• _1DGA142VMSCOMMON.SYSLIBLIBRTL.EXE1 (open)
• Caching is enabled, active caching mode is Write
  Through
• Allocated pages 143 Total QIOs
  1165
• Read hits 1123 Virtual
  reads 1165
• Virtual writes 0 Hit rate
  96
• Read aheads 0 Read
  throughs 1164
• Write throughs 0 Read
  arounds 1
• Write
  arounds 0

Avoid caching files that pollute the cache
22
I/O response time

• SDAgt xfc show volume/brief
•  
• Summary of XFC Cached Volumes (CVBs)-------------
  -----------------------Volume Name      
  CVB                Open   Closed      Total      
  Read       Read      Write      ... Response
  (Milliseconds)...                                
      Files    Files       I/Os       Hits     
  Count      Count          Hits       disk   
  AverageDISKFOOBAR       FFFFFFFEE01895E0     
  0        0          0          0         
  0          0       (N/A)      (N/A)     
  (N/A)DISKGP           FFFFFFFEE0189380     
  0        0          0          0         
  0          0       (N/A)      (N/A)     
  (N/A)DISKORACLE       FFFFFFFEE0189120    
  26        3    1872255          0          0   
  1872255       (N/A)       0.0000    
  0.0000DISKORA2        FFFFFFFEE0188EC0    
  73      177   22015701   14108183   21116834    
  898891        0.0232     0.5811    
  0.2236DISKSYS      FFFFFFFEE0188C60     
  0        0          0          0         
  0          0       (N/A)      (N/A)     
  (N/A)DISKORAIDX      FFFFFFFEE0188A00     
  0        0          1          0         
  1          0       (N/A)      (N/A)     
  (N/A)DISKSYSBCK    FFFFFFFEE01887A0     
  2        0          0          0         
  0          0       (N/A)      (N/A)     
  (N/A)DISKLEEHE        FFFFFFFEE0188540     
  4        3   27676052   27667501   27674665      
  1387        0.0118     0.4007    
  0.0120DISKMIKA     FFFFFFFEE01882E0     
  0        0          0          0         
  0          0       (N/A)      (N/A)     
  (N/A)DISKSYS831H1     FFFFFFFEE0188080   
  313      183    2736618    2668894   
  2713025      23594        0.0179     0.5425    
  0.0308

SDAgtXFC SHOW VOLUME/BRIEF
23
The XFC overhead
Elapsed time to copy 150MB file, rx2600, HSG80,
OpenVMS V8.3
24
The XFC overhead
Seconds to create a container file less is better
25
Shadowed RAM disk

• Applications that mainly read data would benefit
  from using shadowed RAM disk.
• The Shadow server will read from memory and will
  write to both devices.
• Forces data to remain resident in memory.
• Significantly boosts performance when files are
  opened cluster wide by multiple users.
• XFC will not help

26
Physical Disk Vs. RAM disk

• C application that processes records read from
  sequential file
• Each I/O 124 Blocks
• RX2600, OpenVMS V8.3, HSG80

Elapsed time to read 250MB file (less is better)
27
RMS

• RMS holds great potential for reducing I/Os
  improving performance
• The C RTL uses RMS
• Most C applications would benefit from RMS tuning

28
RMS

• RMS parameters related to performance
• FAB/RAB parameters (should you have access to the
  code)
• ASY, RAH, WBH, DFW
• ALQ DEQ
• MBC MBF
• NOSHR, NQL, NLK
• SET RMS
• /SYSTEM /PROCESS
• /BUFFER_COUNTn
• /BLOCK_COUNTn
• SYSGENgt SET RMS_SEQFILE_WBH 1
• Global Buffers

29
gZIP RMS

• gZIP is written in C I/Os eventually reach RMS
• 1.6 Ghz rx2600, MSA30, OpenVMS V8.3
• Test 1
• Compress 5.67 GB saveset
• Decompress 2.74 gZIP archive
• Default O/S RMS settings
• Test 2
• Compress 5.67 GB saveset
• Decompress 2.74 gZIP archive
• SET RMS/BLOCK127/EXTEN60000/BUFFER8,
  RMS_SEQFILE_WBH1

30
gZIP RMS
Elapsed Time in Minutes (less is better)
31
FTP Performance
32
Impact of RMS tuning
Seconds to create a container file less is better
33
I/O Consolidation - PC Sampling Before

• PC Sampling Information (CPU_CYCLES)------------
  -------------------------Start VA          End
  VA             Image                              
                 Count   Percent-----------------
  -----------------  -------------------------------
  ---------    -----------  --------FFFFF802.894000
  00 FFFFF802.89415FFF  Kernel Promote
  VA                                     2    
  0.78FFFFFFFF.80000100 FFFFFFFF.800111FF 
  SYSBASE_IMAGE                                    
     10     3.89FFFFFFFF.80011200
  FFFFFFFF.80068FFF  SYSPLATFORM_SUPPORT           
                         1     0.39FFFFFFFF.800A00
  00 FFFFFFFF.801D3CFF  SYSTEM_PRIMITIVES_MIN       
                           86    33.46FFFFFFFF.801
  D3D00 FFFFFFFF.802537FF  SYSTEM_SYNCHRONIZATION_MI
  N                           21    
  8.17FFFFFFFF.80365600 FFFFFFFF.80451FFF 
  EXCEPTION                                         
      1     0.39FFFFFFFF.80452000
  FFFFFFFF.80543EFF  IO_ROUTINES                    
                         6     2.33FFFFFFFF.805532
  00 FFFFFFFF.80699DFF  PROCESS_MANAGEMENT          
                           43    16.73FFFFFFFF.806
  99E00 FFFFFFFF.807A25FF  SYSVM                   
                               2    
  0.78FFFFFFFF.807A2600 FFFFFFFF.807F05FF 
  LOCKING                                           
     14     5.45FFFFFFFF.80847600
  FFFFFFFF.80927CFF  F11BXQP                        
                        22     8.56FFFFFFFF.80927D
  00 FFFFFFFF.80B40CFF  RMS                         
                           10    
  3.89FFFFFFFF.80B9CC00 FFFFFFFF.80C091FF 
  SECURITY                                          
      4     1.56FFFFFFFF.80CB0A00
  FFFFFFFF.80D2C9FF  SYSXFCACHE                    
                        11     4.28FFFFFFFF.810C1B
  00 FFFFFFFF.811080FF  SYSDKDRIVER                
                            1    
  0.39FFFFFFFF.8406E000 FFFFFFFF.8425FFFF 
  LIBRTL                                            
      3     1.17FFFFFFFF.84260000
  FFFFFFFF.84285FFF  LIBOTS                         
                         2     0.78FFFFFFFF.844FA0
  00 FFFFFFFF.845ABFFF  PTHREADRTL                 
                           18    
  7.00                                     Not
  Found                                            
  4

34
I/O Consolidation - PC Sampling After

• PC Sampling Information (CPU_CYCLES)------------
  -------------------------Start VA          End
  VA             Image                              
                 Count   Percent-----------------
  -----------------  -------------------------------
  ---------    -----------  --------00000000.000000
  00 00000000.7ADCBFFF  Process Space               
                        28936    34.50FFFFF802.894
  00000 FFFFF802.89415FFF  Kernel Promote
  VA                                   236    
  0.28FFFFFFFF.80000000 FFFFFFFF.800000FF 
  SYSPUBLIC_VECTORS                                
     13     0.02FFFFFFFF.80000100
  FFFFFFFF.800111FF  SYSBASE_IMAGE                 
                      3272     3.90FFFFFFFF.800112
  00 FFFFFFFF.80068FFF  SYSPLATFORM_SUPPORT        
                           56    
  0.07FFFFFFFF.800A0000 FFFFFFFF.801D3CFF 
  SYSTEM_PRIMITIVES_MIN                             
  7848     9.36FFFFFFFF.801D3D00
  FFFFFFFF.802537FF  SYSTEM_SYNCHRONIZATION_MIN     
                      2349     2.80FFFFFFFF.8027C7
  00 FFFFFFFF.8029CAFF  SYSEI1000.EXE              
                            2    
  0.00FFFFFFFF.80365600 FFFFFFFF.80451FFF 
  EXCEPTION                                         
    153     0.18FFFFFFFF.80452000
  FFFFFFFF.80543EFF  IO_ROUTINES                    
                      3664     4.37FFFFFFFF.80543F
  00 FFFFFFFF.805531FF  SYSDEVICE                   
                          838    
  1.00FFFFFFFF.80553200 FFFFFFFF.80699DFF 
  PROCESS_MANAGEMENT                                
  5334     6.36FFFFFFFF.807A2600
  FFFFFFFF.807F05FF  LOCKING                        
                         1     0.00FFFFFFFF.80927D
  00 FFFFFFFF.80B40CFF  RMS                         
                            2    
  0.00FFFFFFFF.80B9CC00 FFFFFFFF.80C091FF 
  SECURITY                                          
    479     0.57FFFFFFFF.80CB0A00
  FFFFFFFF.80D2C9FF  SYSXFCACHE                    
                         3     0.00FFFFFFFF.80D58E
  00 FFFFFFFF.80E1C6FF  SYSCLUSTER                 
                            1    
  0.00FFFFFFFF.81049500 FFFFFFFF.810C1AFF 
  SYSPKMDRIVER                                     
      4     0.00FFFFFFFF.810C1B00
  FFFFFFFF.811080FF  SYSDKDRIVER                   
                         1     0.00FFFFFFFF.8110BF
  00 FFFFFFFF.8113D3FF  SYSLAN_CSMACD.EXE          
                            1    
  0.00FFFFFFFF.8113D400 FFFFFFFF.811B03FF 
  SYSPEDRIVER.EXE                                  
      6     0.01FFFFFFFF.81356900
  FFFFFFFF.813BCAFF  SYSLOADABLE_IMAGESSYSLTDRIVE
  R.EXE                  2     0.00FFFFFFFF.813E3E
  00 FFFFFFFF.816028FF  TCPIPINTERNET_SERVICES     
                            6    
  0.01FFFFFFFF.81640300 FFFFFFFF.817141FF 
  TCPIPNFS_SERVICES                                
      2     0.00FFFFFFFF.8406E000
  FFFFFFFF.8425FFFF  LIBRTL                         
                     18395    21.93FFFFFFFF.842600
  00 FFFFFFFF.84285FFF  LIBOTS                      
                         2251    
  2.68FFFFFFFF.844FA000 FFFFFFFF.845ABFFF 
  PTHREADRTL                                       
  9452    11.27FFFFFFFF.845AC000
  FFFFFFFF.850BDFFF  DECCSHR                       
                       555     0.66               
                        Not Found                   
                          633

35
RMS Global Buffers

• RMS Global Buffers provide another layer of
  caching between RMS XFC
• Does it make sense to use both?
• YES !

36
RMS Global Buffers and/or XFC?

• RMS Global buffers WILL cache files open for
  writeshare in a cluster.
• RMS GB require knowledge of application I/O to
  apply them effectively.
• SET FILE/GLOBAL_BUFFERn
• XFC is S2 Space based and has no effective limit
  to its size other than physical memory available
• Prior to V8.3 RMS GB are limited to a max value
  of 32767 buffers per file.

37
RMS Global Buffers AND XFC make a good pair

• Remember slide 13?
• RMS is closer to the application.
• Use global buffers for files open for shared
  write in a cluster RIGHTSLIST.DAT is a good
  example with a common system disk.
• XFC will cache as much as it can once RMS passes
  I/O to it

38
The Impact of RMS Global Buffers
39
The Impact of RMS Global Buffers
40
File Expansions

• The volume extension size specifies the number of
  blocks to be added when RMS automatically extends
  a file.
• Larger values result in fewer file extensions.
• SET VOLUME/EXTENSIONn
• FABW_DEQ
• Maximum extension size 65,535 blocks

41
File Expansions

• sh dev dka100/full
• Disk 1DKA100 (STRIKE), device type DEC RZZ28B,
  is online, allocated,
• deallocate on dismount, mounted,
  file-oriented device, shareable, available
• to cluster, error logging is enabled.
• Error count 0
  Operations completed 57
• Owner process "SYSTEM" Owner UIC
  SYSTEM
• Owner process ID 00000222 Dev Prot
  SRWPL,ORWPL,GR,W
• Reference count 2 Default
  buffer size 512
• Total size 14.05GB Sectors
  per track 32
• Total cylinders 28786 Tracks
  per cylinder 32
• Logical Volume Size 14.05GB Expansion
  Size Limit 16.96GB
• Allocation class 1
• Volume label "XBC3_RESD" Relative
  volume number 0
• Cluster size 4
  Transaction count 1
• Free space 106.93MB Maximum
  files allowed 2947659
• Extend quantity 5 Mount
  count 1

42
Oracle 10g Database Export
20 Improvement
Minutes to complete database export less is better
43
Larger Block Size with RMS

• Going beyond 127 blocks I/O requires using RAB64
• Implemented in VMS COPY
• /BLOCK qualifier
• The Port drivers currently support a maximum of
  256 blocks per I/O
• Larger I/Os are segmented

44
Impact of larger block size
Seconds to copy 5GB file from disk to disk less
is better
45
Evolution of the test program
Seconds to create a container file less is better
46
FastPath

• Spread I/Os across all CPUs
• show fast
• Fast Path preferred CPUs on MAKLEE 13-MAY-2008
  212041.82
• HP rx3600 (1.59GHz/9.0MB) with 4 active CPUs
• Device Fastpath CPU
• EIA0 1
• EIB0 0
• FGC0 1
• FGA0 0
• FGB0 3
• FGD0 3
• PKA0 2
• OpenVMS TCP/IP is currently running on CPU 1

47
Summary

• Disk I/O is an extremely slow operation
• Dont stall the application
• Use Asynchronous I/Os whenever possible
• Consolidate your I/Os
• Perform larger and fewer I/Os
• Cache information as close to the application as
  possible

48
Questions?

• See us at www.maklee.com for
• Performance improvements
• Oracle Tuning
• Platform Migration
• Custom Engineering solutions
• Custom Training