32bit Virtual Memory Constraints in Windows 2000 and 2003

1
32-bit Virtual Memory Constraints in Windows
2000 and 2003

• Mark Friedman
• Demand Technology
• 1020 Eighth Avenue South, Suite 6, Naples, FL
  34102 USA
• phone [phone] fax [phone]
• e-mail markf_at_demandtech.com
• http//www.demandtech.com

2
Outline

• Topics
• Virtual Memory concepts
• Virtual Memory constraints in 32-bit Windows
• Physical Address Extension (PAE)
• Leaking processes
• Page faults/sec

3
Virtual Memory Manager

• 4 GB virtual per process address space
• Lower 2 GB - Private
• Page 0 reserved
• Code pages
• Heap
• Upper 2 GB - System
• System code
• Shared dlls
• System cache

4
Virtual Memory Manager

• All address spaces share the virtual addresses
  from the same upper 2 GB of the System area
• All virtual addresses in the lower 2 GB User area
  are unique
• Shared memory in the System range can be used for
  IPC

5
Virtual Memory map
6
Virtual Memory map (Win2K)

• No hard limits on the sizes of the pools
• One reserved area of virtual memory can fill up
  faster than the others

7
Virtual Memory Manager

• Commit Limit
• Limit on the number of Virtual Memory pages that
  the system will allocate
• Sizeof (RAM)
• paging file(s)
• Memory allocations start to fail as Committed
  Bytes in Use ? 100

8
Virtual Memory Manager

• Commit Limit
• But page files are extendible!
• Should you or shouldnt you?

9
Virtual Memory Manager
10
Extended Virtual Addressing

• Boot.ini /3 GB switch
• Virtual storage constraint relief for some
  applications
• Squeezes the System VM into 1 GB
• Physical Address Extension (PAE)
• Supports 36-bit real addresses on Xeon processors
• Address Windowing Extensions (AWE)
• Permits processes to address real memory gt 4 GB

11
Virtual Memory Manager

• Boot.ini /3 GB switch
• Virtual storage constraint relief for some server
  applications
• SQL Server
• Exchange 2000
• Etc.
• /uservaSizeInMB subparameter, where SizeinMB can
  be any value between 2048 and 3072

12
Physical Address Extension

• PAE is supported on most recent Intel servers
• Physical addresses are 36-bits
• Up to 64 GB of RAM can be installed
• Page Table Entries (PTEs) remain 32-bits
• Process address spaces are still limited to 4 GB.
  How can they exploit 64 GB of RAM?
• Expand sideways, using multiple address spaces
• Large Memory Enabled (LME) device drivers
• Address Windowing Extensions (AWE)
• Boot.ini /pae switch
• Limitation system addresses can be no higher
  than 16 GB

13
Physical Address Extension

• Expand sideways, using multiple address spaces
• e.g., MS SQL Server

14
IIS 6.0 (expand sideways)
Application Thread pools
OS kernel
HTML Get Request
http.sys
W3wp.exe
Object cache (.htm, .gif, .jpg, etc.)
ASP ASPX Requests
Real Addresses!
Web application Gardens (User mode Processes)
15
Address Windowing Extensions (AWE)

• API that allows processes to address real memory
  locations outside their 4 GB virtual addressing
  range
• Create and manage memory Overlays
• Used in conjunction with PAE

AllocateUserPhysicalPages
Virtual Alloc
MapUserPhysicalPages
16
AWE
Note Frequent unmapping and remapping of
Physical Memory blocks is expensive!
17
AWE support

• MS Exchange /3 GB support only
• MS SQL Server 2000 /3 GB, PAE, AWE
• Multiple process instances
• awe enabled 1
• Also Set max server Oraclememory
• Oracle /3 GB, PAE, AWE
• AWE_WINDOW_MEMORY
• SAS /3 GB, PAE
• Work library can be placed in PAE memory

18
Win64 Virtual Memory
19
Virtual Memory Manager

• Real Memory allocation Counters
• Available Bytes
• Pool non-paged Bytes,
• Pool Paged Resident Bytes,
• System Cache Resident Bytes,
• System Code Resident Bytes,
• System Driver Resident Bytes
• Instantaneous Counters (all reported as Bytes)
• Trimmed Pages on the Modified List are not
  counted (assumed to be small)
• Cache Bytes is actually the (pageable) System
  Working Set
• Pool Paged Resident Bytes System Cache
  Resident Bytes
• System Code Resident Bytes System Driver
  Resident Bytes

20
Real Memory allocation Counters
21
Virtual Memory Manager

• Memory allocation Counters
• Available Bytes
• Available KBytes
• Available MBytes
• Pool non-paged Bytes,
• Pool Paged Resident Bytes,
• System Cache Resident Bytes,
• System Code Resident Bytes,
• System Driver Resident Bytes

22
Virtual Memory map
23
Virtual Memory map

• Paged
• Memory mapped files
• Shared DLLs
• File Server
• IIS (html, jpg, gif)
• Page tables
• nonPaged
• I/O buffers used by device drivers
• TCP Session data
• Kernel threads (win32k.sys per process)

24
Virtual Memory map (Win2K)

• Virtual memory map features overflow areas for
  the file cache, PTEs, and Paged/nonPaged Pools
• One reserved area of virtual memory can fill up
  faster than the others
• No hard limits on the sizes of the pools

25
Virtual Memory map (Win2K)

• No hard limits on the sizes of the pools
• One reserved area of virtual memory can fill up
  faster than the others

26
Virtual Memory map (Win2003)

• Virtual memory map is dynamic, subject to
  adjustment by the OS
• One reserved area of virtual memory can fill up
  faster than the others
• When any one area fills, VMM routines can empty
  the file cache to acquire more vm for the
  paged/nonpaged pools

Paged Pool
Cache
PTEs
nonPaged Pool
27
Nonpaged pool size (ref. Q126402)

• Obsolete in Windows 2003
• Dynamic adjustment of system virtual memory when
  one pool is exhausted.
• NonPagedPoolSize, NonPagedPoolSize, and
  SystemPages can also be set explicitly.
• But how can you tell when you are running out of
  space in one of these pools?

28
Default Paged and Nonpaged pool size can be
overridden
29
Default Paged and Nonpaged pool size can be
overridden (Win 2003)

• System Pages xffff ffff16 or -1
• Maximizes the number of PTEs that can be built
• Potentially useful with /PAE or Terminal Services
• PagedPoolSize xffff ffff16 or -1
• Allows the OS maximum flexibility to determine
  the size of the Paged Pool
• Or PagedPoolSize and NonPagedPoolSize can be set
  explicitly
• LargeSystemCache controls the maximum virtual
  address range of the file cache
• 512 MB 1 GB

30

• !vm Debugger command runs
• the Page Frame Number (PFN) database
• Available Pages 4920 ( 19680 Kb)
• ResAvail Pages 358 ( 1432 Kb)
• Locked IO Pages 251 ( 1004 Kb)
• Free System PTEs 204387 ( 817548 Kb)
• Free NP PTEs 28645 ( 114580 Kb)
• Free Special NP 0 ( 0 Kb)
• Modified Pages 596 ( 2384 Kb)
• Modified PF Pages 660 ( 2640 Kb)
• NonPagedPool Usage 2750 ( 11000 Kb)
• NonPagedPool Max 33768 ( 135072 Kb)
• PagedPool 0 Usage 3544 ( 14176 Kb)
• PagedPool 1 Usage 1359 ( 5436 Kb)
• PagedPool 2 Usage 1340 ( 5360 Kb)
• PagedPool Usage 6243 ( 24972 Kb)
• PagedPool Maximum 138240 ( 552960 Kb)
• Shared Commit 6842 ( 27368 Kb)

31

• !poolused Debugger command accounts
• for all system pool allocations
• (plus pooltags.txt documentation)
• lkdgt !poolused 2
• Sorting by NonPaged Pool Consumed
• Pool Used
• NonPaged Paged
• Tag Allocs Used Allocs Used
• LSwi 1 2576384 0 0
• NV 287 1379120 14 55272
• File 2983 504920 0 0
• MmCm 16 435248 0 0
• LSwr 128 406528 0 0
• Devi 267 377472 0 0
• Thre 452 296512 0 0

32
Poolmon utility (from DDK)
33
Virtual Memory constraints

• Virtual Memory constraints in 32-bit Windows tend
  to appear when sizeof (RAM) ? 4 GB
• 2 GB private area is not enough virtual memory
  for some applications
• e.g., SQL Server, Exchange database (store.exe)
• Due to fragmentation, it is typically not
  possible to allocate all 2 GB
• 2 GB system area is not enough virtual memory for
  some applications
• File cache for a conventional IIS-managed web
  site with many static .htm, jpg, gif, etc., files
  to retrieve

34
Virtual Memory constraints

• Virtual Memory constraints in 32-bit Windows tend
  to appear when sizeof (RAM) ? 4 GB
• Ample RAM exists, but it is not possible for your
  applications to access it due to virtual memory
  addressing limitations
• Large number of Available Bytes

35
Virtual Memory constraints

• So try the /3 GB switch
• 1 GB system area is not enough virtual memory for
  some applications
• Possible shortage of Free System Page Table
  Entries
• Possible shortage of Nonpaged Pool
• Where Session data from TCP connections is stored
• Due to fragmentation, it may not be possible to
  failover a 2 GB private address space (e.g., SQL
  Server, MS Exchange database store.exe) using
  Microsoft Cluster Server (MCS)
• During address space recovery on the standby
  node, the entire virtual memory allocation is
  acquired at one time

36
Virtual Memory constraints

• So try the /3 GB switch
• 1 GB system area is not enough virtual memory for
  some applications
• PagedPoolSize and NonPagedPoolSize defaults are
  cut in ½
• Possible shortage of Free System Page Table
  Entries
• Possible shortage of Nonpaged Pool
• Where Session data from TCP connections is stored
• Where kernel threads are created per process for
  calls to win32k.sys (especially impacts large
  Terminal Server environments)

37
Exchange 2000 memory tuning

• Exchange default memory allocation parameters are
  self-tuning, but may not be optimal on servers
  with gt 1 GB RAM
• Adjust HKLM\SYSTEM\CurrentControlSet\Services\SMTP
  SVC\Queuing
• MsgHandleThreshold MsgHandleAsyncThreshold
• HKLM\SYSTEM\CurrentControlSet\Services\Inetinfo\P
  arameters
• FileCacheMaxHandles

38
Exchange 2000 memory tuning

• Store DB cache
• Store cache normally expands to fill RAM
• But, maximum cache 900 MB
• This value can be adjusted using ADSI Edit tool
• msExchESEParamCacheSizeMax
• msExchESEParamCacheSizeMin
• Also, consider adjusting msExchESEParamLogBuffers
  attribute for active, back-end servers

39
Detecting memory leaks

• Processes that allocate virtual memory, but later
  forget to free it.
• MS says leaks wont happen in .Net managed code
  due to automatic garbage collection
• But, meanwhile,
• Where to look depends on whether process or
  system addresses are being allocated
• Per Process Virtual Bytes, Private Bytes, Pool
  Paged Bytes, Handle Count
• System level Memory Pool Paged Bytes, Pool
  Nonpaged Bytes and the Objects Object

40
Detecting memory leaks

• Look for a steady increase or a sharp spike in
  process Virtual Bytes, or the Systems Pool Paged
  Bytes.
• If RAM is not full, the leak may also be manifest
  in the Memory allocation counters and result in
  increased paging, if RAM fills up.
• For example

41
Detecting memory leaks (1)
42
Detecting memory leaks (2)
43
Detecting memory leaks (2)
44
Detecting memory leaks (2)
45
Detecting memory leaks (2)
46
Detecting memory leaks (2)
47
Detecting memory leaks (2)
48
Detecting memory leaks (2)
49
Detecting memory leaks (2)
50
Detecting Memory Leaks (3)
51
Detecting Memory Leaks (3)
52
Detecting Memory Leaks (3)
53
Detecting Memory Leaks (3)
54
Questions

• ?