Time and Event Based System Software Construction

1
Time and Event Based System Software
Construction

• Christopher D. Gill
• cdgill_at_cse.wustl.edu
• Department of Computer Science and Engineering
• Washington University, St. Louis

EECS Lecture Series Vanderbilt University,
Nashville, TN Thursday October 13th, 2005
Research supported in part by NSF CAREER award
CCF-0448562
Thanks Venkita Subramonian, Huang-Ming Huang,
Jacob Perkins, Robert Pless (WUSTL) César
Sánchez, Henny Sipma, Zohar Manna (Stanford)
2
My Research Focus
Boeing/DARPA F33615-01-C-1898 (NEST)

• Middleware system software spanning networked
  hosts
• Hides complexity of underlying operating system
  and network
• Location/language transparency, event and message
  scheduling
• Significant potential to reduce system
  development time/cost
• Research challenges advancing the state of the
  art
• Fine-grain customization for space/time
  constrained applications
• Adaptation for real-time assurances in variable
  environments
• Formal specification and checking for
  high-assurance systems

3
Motivation Model/Software Chasm
Boeing/AFRL F33615-97-D-1155/0005 (WSOA)

• Increasing distance for system software between
• What development tools can produce
• What analysis tools can check for correctness
• Distributed real-time embedded systems also face
• Memory and power resource constraints
• Stringent timing requirements end-to-end
• Distribution across heterogeneous
  networks/platforms

4
Expert Opinions
ORB configurations have a substantial impact on
performance, e.g., TAO's fastest concurrency
model is thread-per-connection, rather than
thread pool
what's ultimately most important is to determine
how things perform for applications that are
representative of actual user configurations,
rather than micro-benchmarks alone
Dr. Douglas Schmidt ICE / CORBA newsgroup
discussion 3/29/05
a main obstacle to the application of rigorous
development techniques is the lack of methodology
for relating application software and functional
design to physical architecture and
implementation
Dr. Joseph Sifakis, Modeling real-time
systems-challenges and work directions, EMSOFT
2001
5
Its All About Abstraction

• Abstraction Can Help
• Location Transparent Remote Invocation
• Abstraction Can Hurt
• Semantic Mis-matches (especially real-time)

6
The Issue Complexity at Each Level

• CIAO QoS Aware Component Model
• Component, package, assembly, deployment
• TAO/nORB Object Request Brokers
• Reference, stub, skeleton, ORB core, POA
• ACE Portable C Framework
• Reactor, handlers, thread pools, timers
• POSIX OS interface
• Threads, sockets

Component
facet upcall
servant
Client
lookup
thread pool
stub
POA
ORB core
timers
ORB core
reactor
recv
send
IIOP message
socket
7
Key Questions

• (1) Does it really have to be so complicated ?
• But, I just need real-time remote
  invocations...
• I n many cases, complexity cant be eliminated
• (2) What problems have been addressed already ?
• A lot of progress on functional properties
• Frameworks, components, design patterns, static
  analysis, types
• A lot of progress on individual QoS properties
• Real-time theory, OS/middleware scheduling, RTSJ,
  RT-CORBA
• Some promising steps towards bringing these
  together
• DSLs, modeling environments, hybrid control, QoS
  aspects
• (3) What problems still need to be addressed ?
• General, viable foundation for software
  construction
• Detailed models of software abstractions used by
  developers
• Composition of models and software side-by-side
• Tractable analysis techniques, tools for
  round-trip engineering

8
Goal Low-Level Software/Model Bridge

• Model primitive building blocks of well-designed
  software
• Compose model elements software elements
  side-by-side
• Check and/or enforce properties in a tractable
  way

9
Research Problem I Level of Abstraction

• Higher levels of abstraction better for
  developers
• Promote design consistency and quality
• Hide tedious and error prone details
• Lower levels of abstraction better for analysis
• Sufficient detail to evaluate properties
  rigorously
• Can model concurrency, timeliness explicitly

10
Solution Approach

• Hypothesis ACE/POSA2 level strikes good balance
• Developers work with higher level abstractions
• Models of those abstractions sufficient for
  real-time
• Key abstractions
• Reactor, Event Handler, IPC Service Access Point,
  ...
• Run model and empirical evaluations side-by-side

11
Timed Automata Models (UPPAAL)
Communication (Event)
Guard (Value)
Invariant (Value)
Action (Value)
State Variable
Clock Variable
12
Selection of Modeling Tools

• SPIN
• Widely known, automata are untimed
• UPPAAL
• Nice graphical interface, timed automata
• No message parameters, instances defined
  statically
• IF (what weve chosen to work with)
• Timed automata, program models in an input
  language
• Connects to a variety of back-end toolsets (e.g.
  CADP)
• ADT support to abstract details (C/C
  implementation)
• Bogor (what we plan to help extend)
• Highly extensible model checker, open-source
• Explicit representations of objects, concurrency
• Automata are untimed (were interested in adding
  time)

13
Example Service Access Point Model in IF
14
Research Problem II Composition

• Developers must preserve QoS properties
• As various software elements are integrated
• Could preclude undesired semantics for a property
• Time triggered architecture, timed languages
• Can address a restricted domain very well
• Could enforce desired semantics
• Model integrated computing, real-time scheduling
• Can address a wider domain at a cost of some
  complexity

15
Solution Approach

• Hypothesis enforcement is possible and
  sufficient
• Extend from base of low-level mechanism models
• Study issues that arise when theyre composed
• Use model checking to unify different analyses

16
Example Reactor Thread Management
Application
Middleware
Environment
De-multiplexing Dispatching
Application logic
Event Handlers
Event sources
Reactor
Synchronous wait-enable
Serial dispatching
Asynchronous event arrival
17
Two Different Reply Wait Strategies

• Wait on connection strategy
• Gives non-interfering calls
• But may lead to deadlock
• Wait on reactor strategy
• Re-use thread while waiting
• Introduces new blocking factors
• Analysis depends on low-level middleware
  implementation
• E.g., blocking strategy, kind of reactor, method
  continuation

DARPA F33615-03-C-4111 (PCES)
18
Multi-Agent Deadlock Call Sequence

• Wait on connection strategy TP reactor RMI
  graph
• Can create deadly embrace scenarios
• Even if single threaded case wont deadlock

19
Wait-on-Connection Static Dependency DAG
reactor R2

• Model needs abstractions from multiple layers
• Application layer method invocation dependency
  DAG
• ACE layer reactors, threads, reply wait strategy
• Deadlock can occur w/ WOC
• Iff ?i ?k gt Ki calls into reactor Ri on a path
  in dependency DAG
• Where Ki is the number of threads in reactor Ri
• E.g., ltm0,m2,m4,m6sgt
• Graph node colors designate to which reactor they
  belong
• Count color changes along a path
• Models wait-on-connection, but not
  wait-on-reactor (yet)

k2 1
dependency
m6
m2
threads
m0
methods
m3
m4
m5
m1
m7
k1 2
reactor R1
20
Wait-on-Reactor Dynamic Call Tree
m0
m1
may interleave

• Reformulate dependency graph
• As a dynamic method call tree
• Edges denote method calls
• Nodes represent stack frames
• Tree traversal is ordered
• E.g., left to right, post-order
• E.g., deadline blocking analysis
• Child order may be permutable
• Subject to additional constraints
• Can now analyze wait-on-reactor
• Distinct trees method calls may interleave
• Upcalls serialized, nested per reactor thread
  (blocking factors)
• Reactor threads concurrent

m7
m2
m3
m5
m3
m4
m6
m5
may be permutable
21
Recent Collaborations On Composition

• WUGLE User Guided Lexical Engine
• Jacob Perkins and Robert Pless
• Expression DAG transformation tool
• Distributive expansion/reduction
• Maintain functional equivalence
• Remove deadlocks
• Reactor Thread Allocation Protocols
• César Sánchez, Henny Sipma, Zohar Manna
• Decorate call graph with thread height
• Protocol allocate thread iff safe
• Blocking factor trade-offs (again)

m2
m3
m4
m6
m5
Distributive reduction across m4
23
m4
12
12
m2
m2
1
1
1
1
m3
m6
m5
m3
22
Timed Automata Compose Unify Analysis
f5 reply processed
f2 returns
23
Research Problem III Tractability
right away
in a minute
get coffee
go for an espresso
maybe tomorrow?

• Model checking can produce a state space
  explosion
• State space reduction, live variable analysis can
  help
• But even good model checkers dont fully solve
  this
• Need to think of modeling as a design issue, too
• Does the model represent what it needs to
  represent?
• Can the model be re-factored to help the checker?

24
Solution Approach
Theorem 3 (Annotation). Given a system S and
annotation a, if the global call graph GS does
not contain any cyclic dependencies, then
Efficient-P used with a guarantees absence of
deadlock for S.
From Sánchez, Sipma, Subramonian, Gill, and
Manna, Thread Allocation Protocols for
Distributed Real-Time and Embedded Systems,
FORTE 05

• Hypothesis
• Some properties can be proven invariant
• Other properties can be enforced at run-time
• Remaining properties can be checked tractably
• Model/software co-design
• Add functionality to model to improve checking
• Add functionality to software to enforce
  properties
• But watch for whether this makes things better or
  worse!

25
Helping the Model Checker Idle Catcher
26
Reactor with Deadlock Avoidance

• Co-design
• DA Reactor
• IF Model
• Can execute the model
• Can run the software

27
Deadlock Avoidance Protocol Overhead
28
A Sampling of Other Related Work

• Vanderbilt University
• GME, CIAO, CoSMIC, PICML
• UC Berkeley
• E-machine, Giotto, Ptolemy
• Washington University
• Temporal types, schedulability using C type
  system
• MIT
• Concurrency type systems
• Utah
• Task/scheduling first-order logic

29
Concluding Remarks 5 Year Research Agenda

• Model broad substrate of primitives
• Via checkable formal models (timed automata)
• Empirical evaluation (esp. of timing,
  concurrency)
• Possibly extend beyond ACE to other domains
• Define composition algebras, type systems
• Minimize model checking to the extent possible
• Capture/exploit domain knowledge/abstractions
• Extend approach beyond real-time concerns
• Model mechanism faults and failure modes
• Power usage (yet another real valued clock?)

30
Questions ?
Christopher D. GillTime and Event Based System
Software Construction
31
Thanks!
Christopher D. GillTime and Event Based System
Software Construction
32
Backup Slides
33
IF Deadline Miss Extract
34
IF Priority Rules Extract
35
Unidirectional IPC Model in IF
36
IF Service Handler Extract
37
IF Select Extract
38
Single Agent Deadlock Scenario
39
Simple Blocking Factors Composition
40
Multi-Agent Blocking Factors Composition