Handling Priority Inversion in ORB Middleware

1
Handling Priority Inversion in ORB Middleware

• Bryan Thrall Venkita Subramonian
• Presented towards partial fulfillment of
• CS520 Advanced Real-Time Systems, Fall 2003
• 30 Sept 2003

2
Sources of Priority Inversion

• Competition for resources
• Processor Resources
• Causes synchronization overhead
• Communication Resources
• Context switching
• Connection demultiplexing
• Connection cache
• Memory resources
• Memory accesses
• Cache misses

3
Priority Inversion in the CORBA ORB

• Connection Architecture (1)
• Connection multiplexing
• Dynamic connection management
• Memory Freestore (2)
• Serialized access
• Concurrency Architecture (3)
• Thread Pools
• POA
• Layered demultiplexing (4)
• Active Object Table synchronization (5)

4
Real-time CORBA 1.0

• Key Features to handle priority preservation
• Portable Priorities
• End-to-end priority propagation
• Protocol properties
• Key Features to handle priority inversion
• Thread pools and buffering
• Explicit binding
• Standard Synchronizers

5
Portable Priorities
Pluggable Mapping to convert RTCORBA priority to
native priority

• How can we map global priorities onto
  heterogeneous native OS host thread priorities
  consistently end-to-end?
• Use Standard RT CORBA priority mapping interfaces

OS-independent design supports heterogeneous
real-time platforms
CORBA priorities are globally unique values
that range from 0 to 32767
6
End-to-end Priority Preservation

• How can we ensure requests dont run at the wrong
  priority on the server?
• Use RT CORBA priority model policies

Server handles requests at the priority declared
when object was created
Request is executed at the priority requested by
client. Priority is encoded as part of client
request
7
Client Propagated Priorities

• Localized priority determination on each end
  system.

8
Priority Inversion - Request Processing

• Server concurrency limited during request
  processing
• High priority clients may not get sufficient
  resources held by low priority requests

Client
2
LO
1
Server
obj1.foo()
Client
3
ME
obj2.boo()
Client
5
4
HI
No thread to serve high priority client
Priority Inversion
9
Solution - Thread Pools

• How can we pre-allocate threading resources on
  the server portably efficiently?
• Use RT CORBA thread pools
• Pre-allocation of threads
• Partitioning of threads
• Bounding of thread usage
• Buffering of additional requests

10
Thread Pools - Implementation Strategies

• Use the Half-Sync/Half-Async pattern to have I/O
  thread(s) buffer client requests in a queue
  then have worker threads in the pool process the
  requests
• Use the Leader/Followers pattern to demultiplex
  I/O events into threads in the pool without
  requiring additional I/O threads

11
Thread Pools - Queue-per-Lane Design

• Single acceptor endpoint
• One reactor for each priority level
• Each lane has a queue
• I/O application-level request processing are in
  different threads

12
Thread Pools - Reactor-per-Lane Design

• Each lane has its own set of
• reactor
• acceptor
• endpoint
• I/O application-level request processing are
  done in the same thread

13
Thread-Pool designs - comparison
14
Priority Inversion Resource Usage
1
Server
LO
Connection Cache
2
ME
3
HI
Client

• Each lane has its own set of resources
• Memory pool
• Connection pool

15
Explicit Binding

• How can we achieve the following?
• Control jitter due to connection setup
• Minimize thread-level priority inversions
• Avoid request-level (head-of-line) priority
  inversions
• Use RT CORBA explicit binding mechanisms
• Connection pre-allocation
• Priority Banded Connection Policy
• Private Connection Policy

Note the priority inversion above since the
stop(), turn(), and query_state() requests all
share the same connection
16
Explicit Binding - Connection Banding

• How can we minimize priority inversions, so that
  high-priority operations are not queued behind
  low-priority operations?
• Program the client to use different connections
  for different priority ranges via RT CORBA
  priority bands

Note how the stop() and turn() requests no longer
share the same connection as query_state()
requests
17
Explicit Binding - Preallocate Connections

• How can we prevent connection establishment
  between the client station and server from
  resulting in unacceptable jitter?
• Jitter is detrimental to time-critical
  applications
• Pre-allocate one or more connections using the
  Object_validate_connection() operation

The _validate_connection() operation must be
invoked before making any other operation calls
18
Explicit Binding Private Connections

• How can we minimize priority inversions by
  ensuring applications dont share a connection
  between multiple objects running at different
  priorities?
• e.g., sending a stop() request should use
  exclusive, pre-allocated resources
• Use the RT CORBA PrivateConnectionPolicy to
  guarantee non-multiplexed connections

Note how the stop() and turn() requests no longer
share the same connection from client to server
19
POA Active Demultiplexing
20
POA - Active Demultiplexing (continued)
Work at Nuclear Plant
Run Nuclear Plant
Play PoohSticks
Play Saxophone
Make Money
Impress Girls
Sell Propane
Drink Beer
Eat Honey
Sell Beer
Bounce
Doh!
Skeleton Layer
Tigger
Pooh
Homer
Lisa
Mr. Burns
Moe
Hank
Bobby
Servant Layer
Fox / Simpsons
Disney / Winnie the Pooh
Fox / Simpsons / Family
Fox / Simpsons / Townspeople
Fox
Disney
Fox / King of the Hill
Root POA
POA Layer
ORB I/O
ORB Core Layer
21
Priority Inversion - POA dispatching

• An object cannot be destroyed while it is still
  registered in the dispatching table
• Monitor lock used both for dispatching and
  modifying the table
• Other threads cannot delete an object that is in
  the midst of being dispatched
• Unbounded priority inversion because upcall time
  could be unbounded

POA
2
1
3
obj1
LO
obj2
4
Block
HI
Active Object Map
22
Solution - Reference Counting

• Reference Count objects in Active Object Map
• Bounded blocking factor for high priority thread
• time to search the object entry time to
  increment the reference count locking overhead

Acquire
Release
POA
2
5
1
7
obj1
4
LO
6
obj2
3
HI
23
Experiments Baseline
Increasing work
Establishing system capacity
Increasing invocation rate
24
Experiments - Classic CORBA
Client
Server
HI 75Hz
ME 50Hz
LO 25Hz
System capacity at 150 (755025) invocations/sec
25
Experiments RTCORBA with TP lanes 1/2
Higher rate thread assigned higher priority
Server
Client
HI 75Hz
HI
ME 50Hz
ME
LO 25Hz
LO
26
Experiments RTCORBA with TP lanes 2/2
Higher rate thread assigned lower priority
Server
Client
LO 75Hz
LO
ME 50Hz
ME
HI 25Hz
HI
27
Best Effort with classic CORBA
Client
Server
HI 75Hz
ME 50Hz
LO 25Hz
Best effort
Continuous workers
28
Experiments Best Effort with RTCORBA 1/3
Server
Client
HI 75Hz
HI
ME 50Hz
ME
LO 25Hz
LO
Best effort
Best effort
29
Experiments Best Effort with RTCORBA 2/3
Server running at capacity (work30)
30
Experiments Best Effort with RTCORBA 3/3
Server running under capacity (work28)
31
Conclusion
32
Critiques Research ideas

• It would be nice to compare the performance of
  buffered request handling vs non-buffered request
  handling at the server
• In real-time embedded systems, footprint could be
  a big issue. With isolation of resources, the
  runtime footprint could be high
• Isolation of resources lead to more areas for
  replication for supporting fault-tolerance, which
  means more overhead and hence would be a concern
  for real-time
• Convergence of aspects like real-time,
  fault-tolerance and security and how each one
  affects the other is an interesting area of
  research
• Experimental baseline does not specify whether
  collocated or remote

33
Request Buffering

• How can we support real-time applications that
  need more buffering than is provided by the OS
  I/O subsystem
• Buffer client requests in ORB

34
Standard Synchronizers

• An ORB application may need to use the same
  type of mutex to avoid priority inversions
• e.g., using priority ceiling or priority
  inheritance protocols
• Use the RTCORBAMutex synchronizer

35
Protocol properties

• Selecting communication protocol(s) is crucial to
  obtaining QoS
• TCP/IP is inadequate to provide end-to-end
  real-time response
• Use RT-CORBA Protocol Policies to select and/or
  configure communication protocols

36
Network Priority Enforcement

• How will the RTCORBA priority be enforced by the
  network?
• Using RTCORBA protocol properties to enable
  priority enforcement mechanisms in the network
• e.g., DiffServ

• Pluggable Network Priority Mapping to convert
  RTCORBA priority to DiffServ TOS

IP Datagram Header
37
Experiments RTCORBA with TP lanes - remote
Higher rate thread assigned higher priority -
Remote
Collocated
Remote
38
Experiments - RTCORBA
Thread Pools with no lanes LO priority
Client
HI 75Hz
LO
ME 50Hz
LO 25Hz
Best effort
39
Thread Pools with no lanes ME priority
Client
HI 75Hz
ME
ME 50Hz
LO 25Hz
Best effort
40
Thread Pools with no lanes HI priority
Client
HI 75Hz
ME
ME 50Hz
LO 25Hz
Best effort
41
Locking Mechanisms A comparison