Eagle Rule Based Runtime Verification

1
EagleRule Based Runtime Verification

• Koushik Sen

Joint Work with Howard Barringer, Allen Goldberg,
Klaus Havelund at NASA Ames Research Center.
2
Increasing Software Reliability

• Model checking and Theorem Proving are rigorous
• Not scalable
• Complex
• Testing is scalable and widely used
• Ad hoc
• Lack of coverage
• Can we combine Formal Methods and Testing ?
• To get the benefits of both the approaches.
• To avoid the pitfalls of ad hoc testing.
• To avoid the complexity of full-blown theorem
  proving and model checking.

3
Runtime Verification

• Merge testing and temporal logic specification
• Specify safety properties in some temporal logic.
• Instrument program to generate events.
• Monitor safety properties against a trace of
  event emitted by the running program.
• Pros Scalable
• Cons Lack of Coverage

4
Existing Systems

• MaC Tool (UPenn) uses past-time interval logic
• Temporal Rover commercial tool
• Java PathExplorer (NASA Ames) past-time logic
  or future time logic
• Java MultiPathExplorer past-time logic with
  extended coverage.
• Statistics Collection by Finkbeiner et al.
• Debugging Distributed Autonomous Systems by
  Simmons et al. (CMU)

5
Motivation

• Do we really need different logics?
• What is the most basic, yet, general
  specification language suitable for monitoring?
• Eagle Rule Based Verification Framework is an
  attempt to unify the existing logics into a
  single framework.

6
Features of Eagle

• Parameterized recursive equations.
• Minimal/maximal fix-point semantics.
• Three primitive temporal operators
• Next
• Yesterday
• Concatenation
• Can encode future time temporal logic, past-time
  logic, µ-calculus, real-time, data-binding, state
  machines, alternating automata over finite
  traces, statistics.
• Even suitable for Aspect-Oriented Programming.

7
Recursive Formulas

• One can state a property whenever x is positive
  then eventually y becomes positive.
• This can be written as
• ?(x gt 0 ? ?(y gt 0))
• The formulas ?F and ?F satisfies the equations
• ?F F ? ?(? F) maximal solution
• ?F F ? ?(? F) minimal solution

8
What does Eagle look like?

• max Always(Formula F) F /\ ? Always(F) .
• min Eventually(Formula F) F \/ ? Eventually(F)
  .
• min Eventually_(Formula F) F \/ ?
  Eventually_(F) .
• The formula ?((xgt0) ? ? (y3)) can be written as
• monitor M1 Always((x gt 0) -gt Eventually_(y3))
  .

9
Generic Rules for Past-time Temporal Logic

• max Always_(Formula F) F /\ ? Always_(F) .
• min Eventually_(Formula F) F \/ ?
  Eventually_(F) .
• min Since(Formula F1, Formula F2)
• F2 \/ (F1 /\ ? Since(F1,F2)) .
• max Zince(Formula F1, Formula F2)
• F2 \/ (F1 /\ ? Zince(F1,F2)) .

10
Generic Rules for Future-time Temporal Logic

• max Always(Formula F) F /\ ? Always_(F) .
• min Eventually(Formula F) F \/ ? Eventually(F)
  .
• min Until(Formula F1, Formula F2)
• F2 \/ (F1 /\ ? Until(F1,F2)) .
• max Unless(Formula F1, Formula F2)
• F2 \/ (F1 /\ ? Unless(F1,F2)) .

11
Mixture of Past-time and Future-time

• Using primitive operators
• min R1(Formula F1, Formula F2)
• F1 /\ ? R1(F1,F2) /\ ? F2.
• Using already defined Rules
• min R2(Formula F1, Formula F2, Formula F3)
• Always(F1 -gt Eventually_(F2 /\
  Eventually(F3))) .

12
Data Binding

• ?((x gt 0) ? k x ? (y k ))
• ?((login userid) ? k userid ? (logout k
  ))
• monitor M1
• Always((xgt0) -gt let k x in
  Eventually((yk))).
• which is rewritten into the following two rules
• min R(int k) Eventually(yk) .
• monitor M2 Always((xgt0) -gt R(x)) .

13
Real-time Special Case of Data Binding

• Want to define ?t1,t2 F
• min TEventually(Formula F, long t1, long t2)
• TEventuallyAbs(F,(t1time()),(t2s.time())) .
• min TEventuallyAbs(Formula F, long t1, long t2)
• (time() lt t2) /\
• (F -gt (t1 lt time())) /\
• ( F -gt ? TEventuallyAbs(F,t1,t2)) .

14
Extended Regular Expressions (ERE)

• Tr(ø) False
• Tr(a) a ? ? Empty()
• Tr(R1 R2) Tr(R1) ? Tr(R2)
• Tr(R1) Tr(R1)

Tr(e) Empty() Tr(R1.R2) Tr(R1).Tr(R2) Tr(R1
n R2) Tr(R1) ? Tr(R2)
Tr(R) X(R) where max X(Formula R) Empty() ?
(Tr(R) . X() ) max Empty() ? True
15
µTL Example

• The fix-point formula
• ?x.p ? ??x ? µy.q ? ?x ? ?y
• can be written as
• max X() p ? ??X() ? Y()
• min Y() q ? ?X() ? ? Y()

16
Collecting Statistics

• min Stat(Formula F,int count)
• (F -gt Stat(F,(count1))) /\ ( F -gt
  Stat(F,count)) .

F
F
F
F
F
F
F
F
F
Count 5
17
Algorithm for Monitoring Functional Style

• Evaluation of a formula F on a state s ?(i)
  results in another formula eval(F,s)
• ?,i ? F iff ?,i1 ? eval(F,s)
• The function update(F,s) updates a formula
  properly in presence of previous operator
• The value of a formula at the end of a trace is
  given by value(F)

18
Monitoring a Formula
?((xgt0) ? k x ?((z2) ? (yk)))
(x0,y4,z2)
?((xgt0) ? k x ((4k) ? ?((z2) ? (yk))))
(x3,y5,z0)
X
((3gt0) ? (43)) ? ?((xgt0) ? k x ((4k) ?
?((z2) ? (yk))))
False
19
Eagle Calculus Internals of Rules

• eval(exp,s) value of exp in s
• eval(F1 op F2,s) eval(F1,s) op eval(F2,s) where
  op is ?,?,?
• eval(F,s) eval(F,s)
• eval(?F,s) update(F,s)
• update(exp,s) exp
• update(F1 op F2,s) update(F1,s) op update(F2,s)
  
• update(F,s) update(F,s)
• update(?F,s) ?update(F,s)

20
Eagle calculus

• value(exp) False
• value(F1 op F2) value(F1) op value(F2) where op
  is ?,?,?
• value(F) value(F)
• value(?F) False
• value(R(F1,,Fn)) True if R is max
• value(R(F1,,Fn)) False if R is min

21
Calculus for Rules Simple Future

• max Always(Formula F) F ? ?Always(F)
• eval(Always(F),s) eval(F ? ?Always(F),s)
• update(Always(F),s) update(F ? ?Always(F),s)
• update(F,s) ? ?update(Always(F),s)
• update(Always(F),s) Always(update(F,s))

22
Previous Operator

• In a rule if F is guarded by ?
• Evaluate F at every state of trace
• Store the evaluation F in some temporary
  storeholder
• Use it in the next state
• To allocate storeholder
• Introduce an argument in the rule for every F
  guarded by ?

23
Calculus with Previous Operator

• max Foo(Formula F) F ? ?Foo(F)
• Foo(Formula F) Foo(Formula F,Formula past1)
• Foo(F) Foo(F,value(Foo(F)))
• eval(Foo(F,past1),s) eval(F ? past1,s)
• update(Foo(F,past1),s)
  Foo(update(F,s),eval(Foo(F,past1),s))

24
Rules on Right Hand Side

• max Bar(Formula F) Foo(F) ? ?Bar(F)
• eval(Bar(F),s) eval(Foo(F) ? ?Bar(F),s)
• update(Bar(F),s) Bar(update(F,s))
• update(Bar(F),s) update(Foo(F) ? ?Bar(F),s)
• Introduce additional argument as in case of
  previous
• Bar(F) Bar(F,Foo(F))
• eval(Bar(F,inner1),s) eval(inner1 ?
  ?Bar(F,inner1),s)
• update(Bar(F,inner1),s) Bar(update(F,s),update
  (inner1,s))

25
Data Values Partial Evaluation

• ?((x gt 0) ? k x ? (y k ))
• (x 5,y3)
• ? (y 5 ) ? ?((x gt 0) ? k x ? (y k ))

26
Calculus with Data Values

• max Pub(int k) (x k) ? ?Pub(yk)
• eval(Pub(arg1),s) let k eval(arg1,s) in
• eval((xk) ? ?Pub(yk),s)
• update(Pub(arg1),s) Pub(arg1)
• Note eval(arg1,s) may be partially evaluated if
  it refers to some variable in arg1 have partial
  value.
• Expression gets fully evaluated once all
  expressions referred by variables are fully
  evaluated.

27
Implementation

• We have an implementation in Java
• Each Rule gets converted into a Java class
• Partial evaluation of Java expressions is
  implemented by wrapping each possible expression
  in a Java object
• The package is available on request

28
Observer and State
class State int x,y update(Event e)
x e.x y e.y
class Observer Monitors mons State state
eventHandler(Event e) state.update(e)
mons.apply(state)
e1 e2 e3
class Monitor Formula M1, M2 apply(State
s) M1.apply(s) M2.apply(s)
class Event int x,y
29
Case Study

• Real-time properties have been monitored on
  Spacecraft Controller K9-Rover.
• 35,000 lines of C code,
• executes action plans.
• The test-case generator automatically generates
  input plans for the controller.
• For each input plan a set of temporal formulas is
  generated that the plan execution should satisfy.
  
• A previously unknown error was detected in the
  first run, demonstrating that a certain task did
  not recognize the too early termination of some
  other task.

30
Contributions

• A generic framework for defining temporal logics
  modularly.
• Algorithm to monitor mixed past-time and
  future-time temporal formulas
• Algorithm to handle Data Values
• An efficient Implementation of the monitoring
  framework in Java

31
Future Work

• Encode other temporal logics in Eagle.
• Improve Eagle by adding necessary primitive
  operators.
• Port it to C to increase speed.
• Integrate with Aspect Oriented Programming
  paradigm
• Temporal Aspects
• Extend Eagle to Partial Orders in Distributed
  Computations