Concurrent Games in Verification

1
Concurrent Games in Verification

• Rupak Majumdar
• University of California, Los Angeles

2
Games on Components

• Model synchronous interactions between components
  in an open system
• Games as models of interaction
• Reactive systems repeated game
• Controller synthesis
• Does the controller have a strategy to ensure the
  composition obeys good properties?

3
Component Composition

• Inputs and outputs behave in different ways
• Related to co- and contra-variance in type theory
• Interface automata

4
Robust Planning

• Suppose that environment actions are not known
  precisely
• Can formulate planning problems as two-player
  games
• the environment is allowed disturbance inputs
• Such plans oblivious to uncertainties

5
Games on Graphs

• Our games will be played on graphs.
• Moves correspond to moving from one vertex to a
  neighbor
• Games will be played for infinite number of
  rounds
• The outcome of a game is an infinite sequence of
  vertices of the graph
• Vertices States

6
Turn-based Games
a
b
c
d
c
e
Models asynchronous interaction, Full
Information

• Algorithm
• Start with P
• Iterate a Controllable Pre
• Until convergence

Reachability Ensure that some set P is reached
7
History Infinite Games

• Two person games studied in logic, automata
  theory, economics,
• Infinite games of perfect information are not
  determined GaleStewart53,Mazur?
• Open and closed games are determined GS53
• ?2 games determined Wolfe55
• ?3 games determined Davis64
• TheoremMartin75 Borel games are determined.
• Axiom of Determinacy

8
History Automata Theory

• Churchs Problem Church62 Synthesis Problem
  for S1S
• Solved by Buchi Landweber 69
• Rabins Tree Theorem (Decidability of S2S)
• Gurevich and Harringtons Proof using Games
  GH82
• Muchnik84, YY90, Zeitman94
• McNaughton Infinite Games on Finite Graphs
  McN93
• Synthesis Problem for LTL
• AbadiLamportWolper89,PnueliRosner89
  KupfermanVardi00
• Realizability Problem
• Receptiveness Dill89,AbadiLamport93
• Supervisory Control RamadgeWonham89

9
History Concurrent Games

• Incomplete Information
• Players simultaneously and independently choose
  moves
• Perfect recall
• Concurrent Games in verification
• Concurrent Games, ?-regular objectives
• Deterministic Strategies
• ATL AlurHenzingerKupferman97,AlurHenzingerKupfe
  rmanVardi98
• Applications to verification Mang03
• Theories of compatibility of interfaces
  deAlfaroHenzinger01,others

10
History Probabilistic Concurrent Games in
Verification

• Qualitative Winning Conditions win with
  probability 1
• deAlfaroHenzingerKupferman98, deAlfaroHenzinger00
  
• JurdzinskiKupfermanHenzinger02
• Quantitative Winning Conditions maximal
  probability
• deAlfaroM01 (This talk)
• Quantitative ? calculus characterization
• 2EXP algorithm
• Special case of Turn-based probabilistic
  ChatterjeeJurdzinskiHenzinger03,04

11
History Probabilistic Concurrent Games

• Minmax Theorem vN28,vNM44
• One shot zero sum game
• Randomized Strategies
• Markov Decision Processes (40s, 50s)
• Value exists for discounted stochastic games
  Shapley53
• Value for ?2 payoffs Blackwell67
• Value for limiting average criterion
  MertensNeyman81
• Value exists for payoff limsup f, f Borel
  MaitraSuddherth95
• Value for ?3 payoffs Vervoort00
• Martin98 Concurrent games with Borel payoff are
  determined.

12
Concurrent Games Example
01 10
01 10
00 11
00 11
Probability to win with deterministic strategies
is 0
Player 1 has a randomized strategy to win with
probability 1/2
Quantitative winning!
13
Concurrent Games

• Two players
• Finite set of states S
• Finite set of actions S
• Action assignments ?1,?2S! 2?n
• Probabilistic transition function
• d(s, a1, a2)(t) Pr t s, a1, a2

14
Overview of Types of Games
Deterministic
Probabilistic
Tic-tac-toe, Control of ?-automata
Control of probabilistic I/O automata
Turn based
Matching pennies, rock- Paper, scissors, Control
of synchronous components
Stochastic games Control of general Competitive
Markov Processes
Concurrent
15
Overview of Types of Games
Deterministic
Probabilistic
8 s2 S.?1(s)1or ?2(s)1 8 a2?1(s)8
b2?2(s)?(s,a,b)1
8 s2 S.?1(s)1or ?2(s)1
Turn based
8 a2?1(s)8 b2?2(s)?(s,a,b)1
Concurrent
16
Winning Conditions

• Outcome Sequence of states
• (or probability distribution over sequences of
  states)
• Winning Condition
• ?-regular language L
• Player 1s objective
• Ensure that the outcome is a member of L

17
Winning Conditions w-regular sets
Safety
Reachability
B
Always in B
Reach B
B
Büchi
coBüchi
Visit B infinitely often
Eventually forever B
B
B
1
2
3
0
Rabin chain
The highest index visited infinitely often is even
18
Strategies

• Deterministic Strategies
• Functions from histories to enabled moves given
  a play s0s1 sk,
• strategy ?i(s0s1...sk) a
• for some a 2 ?i(sk)
• Randomized strategies
• Functions from histories to lotteries over
  enabled moves given a play s0s1 sk,
• strategy ?i(s0s1sk) D
• for some distribution D over the enabled moves

19
Level 1

• Algorithms for Deterministic Games

20
Fundamental Question

• Given a deterministic turn based game and a
  winning
• condition, find the set of states from which
  player 1
• can win. Also find a (deterministic) winning
  strategy.

21
One-Step Game

• Regions are sets of states
• Let U be a set
• From where can we reach U surely in one step?
• CPre1(U)
• s9 a2?1(s).?(s,a)2 U s8 b
  2?2(s).?(s,b)2 U
• CPre1 is a transformer on sets
• Similarly, we can define CPre2 for player 2

22
Multistep Reachability

• On turn based deterministic games
• This is a least fixpoint
• ? x. P Ç CPre1(x)

P
.
CPre(P)
CPre2(P)
23
The Propositional ? calculus

• A general logic of fixpoint operatorsKozen83
• Basic modal logic fixpoints
• ? p p ?1Ç?2 ?1 Æ ?2 EX? AX ?
• x ? x. ? ? x. ?
• Semantics is given over sets of states
• The ?-calculus provides fixpoint
  characterizations for winning states in our games

24
History ? Calculus

• Introduced by Kozen 83 as basic modal logic
  fixpoints
• Very expressive, usually all program logics can
  be embedded
• Provides symbolic algorithm schemas
• Satisfiability of the ? calculus
• Reduce to the solution of Parity Games
  EmersonJutla91
• Solution of Parity Games expressed in the
  ?-calculus
• Memoryless determinacy, NPÅ coNP
• Model Checking Problem for the ? calculus
• Equivalent to solving Parity Games
  EmersonJutlaSistla93
• Efficient algorithms Jurdzinski99,JurdzinskiVoge0
  0

25
Multistep Reachability

• The proof is not yet complete.
• We can win from ? x. P Ç CPre1(x), to finish the
  proof we must show we cannot win from the
  complement

P
.
?
CPre(P)
CPre2(P)
26
Complementation and Correctness

• At this point there are two ways to finish the
  proof
• Find spoiling strategies of player 2 in the
  complement of the fixpoint
• Trouble We have to construct a player 2
  strategy, but the formula has CPre1
• Exploit the syntactic complementation of
  m-calculus
• For a formula f, there is a formula f such
  that
• f 1 f
• Construct a player 2 strategy from this
  complement
• When possible, this often allows easier arguments

27
Proof Strategy
Strategy for Player 1 that ensures f
Proving h 1 iY f
Objective Y
negate Y
negate f
Strategy for Player 2 that ensures f
Proving h 1 iY f
Objective Y
28
Winning Conditions w-regular sets
Safety
Reachability
B
Always in B
Reach B
B
Büchi
coBüchi
Visit B infinitely often
Eventually forever B
B
B
1
2
3
0
Rabin chain
The highest index visited infinitely often is even
self dual
29
Lets try Safety

• Complement of ? x. UÇ CPre1(x) is
• ? x. U Æ CPre1( x)
• What is CPre1( x) ? CPre2(x)!!
• So we have
• ? x. U Æ CPre2(x)
• We show this is the solution of the safety game
  always U for player 2

30
Lets try Safety

• ? x. U Æ CPre1(x)
• Extract a strategy for player 1
• Let X U Æ CPre1(X)
• From a state in X, play according to CPre1(X)
• Repeat
• Thats it!
• The game is determined

31
Büchi and co-Büchi Games

• Büchi visiting a set U infinitely often
• coBüchi eventually always staying in a set U

n y. m x. (( U Æ Cpre(x)) Ç (U Æ Cpre(y)))
m x. n y. (( U Æ Cpre(x)) Ç (U Æ Cpre(y)))
32
Relationship Deterministic Case

• A transition system is a game where only one
  player makes a move
• It is player 1 if player 1 makes all choices
• It is player 2 if player 2 makes all choices
• i-Verification Problem
• Given an objective ? and a player i transition
  system, does ? hold?
• Special cases of the game problem
• A solution ? to the game question also solves
  the i-verification questions

33
Relationship Deterministic Case

• What is the relationship between ? calculus
  formulas that solve games with ? calculus
  formulas that solve the verification problem?
• For reachability ? x. P Ç Pre(x) solves the
  verification problem
• But there are objectives ? and ? calculus
  formulas ? such that ? solves the 1-verification
  problem, but not the control problem

34
Extremal Model Theorem

• Theorem deAlfaroHenzingerM01 A ?-calculus
  formula ?(Cpre) solves the game ? iff ?(Pre)
  solves the 1-verification problem for ?, and
  ?(DPre) solves the 2-verification problem for ?.
• Essentially a restatement of finite memory
  determinacy

35
Entering Level 2

• Concurrent Games

36
Winning Conditions Concurrent Games

• Value of a game is the maximal probability of
  ensuring the outcome is in Y
• h 1 iY(s) supx 1infx 2 Prsx 1x 2 Y
• (where Y Index set for Y)
• Fundamental Question Given a concurrent game and
  a winning condition, find at each state the
  maximal probability with which player 1 can
  ensure the winning condition holds

37
Turn-based vs Concurrent
38
Algorithms
Turn-based
Concurrent
Qualitative
Quantitative
Safety Reachability Büchi coBüchi Rabin-chain
Win with probability 1 or limit probability
1 dAHK98 dAH00
Classical GH82 EJ91
Maximal probability of winning
39
One-Step Game

• Regions are functions f S ! 0,1
• Suppose f is a payoff function on states
• From state s, players choose actions a1, a2
  (simultaneously and independently)
• The next state Q is chosen according to the
  distribution d, and player 1 gets payoff f(Q)

40
One-Step Game

• Player 1s value
• Maximal expectation of f(Q)
• Define the value
• Ppre (f) (s) supx 1infx 2ESf(Q)

41
One-Step Game

• Monotone and continuous
• Equivalent to zero-sum matrix games
• Value and optimal randomized strategies exist for
  both players vonNeumann28
• Can be computed by linear programming

42
Reachability

• Maximal probability of reaching a set U of states
• Can be reduced to positive stochastic games
• Algorithm
• X0 0 Xn1 max(U, Ppre(Xn))
• X lim Xn
• Correctness is by induction on the n-step game

43
Reachability Example
01 10
01 10
S3
00 11
00 11
S1
S2
S4
Computing the least fixed point solution m x.
max (s4, Ppre(x))
44
Quantitative m calculus

• General theory of fixpoint operators
• f p x fÇf fÆf pre(f) m x.f n x.f

Normal m calculus
Quantitative m calculus
45
Guiding Principle

• For reachability, f Ppre / Cpre gave
  corresponding algorithm for concurrent games
• Conjecture that the same holds for all properties
  of interest

46
Proof Strategy I

• Proof Strategy for h 1iY f

For the objective Y, for any e, produce a
strategy for player 1 guaranteeing f e
showing h 1iY f
47
Proof Strategy II

• Proof Strategy for h 1iY f

For the objective Y, for any e, produce a
strategy for player 2 guaranteeing f e
showing h 2iY f or equivalently, h 1 i
Y f
48
Proof Strategy
Let f m x. U Ç Ppre(y)
Strategy for Player 1 that ensures f - e
Proving h 1 iY f
Objective Reach U
negate Y
negate f
Strategy for Player 1 that ensures f - e
Proving h 1 iY f
Objective Stay in U
49
Safety

• Maximal probability of staying forever in a set U
  of states
• m-calculus algorithm n x. UÆ Ppre(x)
• Complement of the reachability formula
• (m x. UÇ Ppre(x)) n x. U Æ Ppre(x)
• Iterative approximation
• X0 1 Xi1 U Æ Ppre(Xi)

50
Safety

• Let w U Æ Ppre (w)
• Strategy While in U, play to maximize the
  probability of going to w in one step
• Define a random process (submartingale)
• Show that the nth stage of the random process
  bounds the max probability of staying in U for n
  steps
• Finally, show that the limit of the process as n!
  1 converges to the value of the safety game

51
Safety Proof

• Let w n x. U Æ Ppre(x)
• Consider the following strategy p1 of player 1
• s2 U play optimally in Ppre(w)(s)
• sÏ U play arbitrary

52
Safety Proof

• Let w n x. U Æ Ppre(x)
• Consider the following strategy p1 of player 1
• s2 U play optimally in Ppre(w)(s)
• sÏ U play arbitrary
• Fix a state t and a strategy p2 of player 2

53
Safety Proof

• Define the process Hn as Hn w(Qn)
• For s2 U, we have w(s) Ppre(w)(s)
• From definition of p1 get for n 0
• Et Hn1 H0 Hn Hn
• So Et Hn H0 w(t)
• But Et Hn is bounded above by the event of
  staying in U for at least n steps
• Now take the limit as n! 1

54
Reachability and Safety

• For reachability optimal strategies may not
  exist, memoryless e-optimal strategies exist
• For safety memoryless optimal strategies exist
• Strategies may require randomization

55
No optimal strategy Example
01 10
00
11
Probability of winning is 1
Player 1 has a randomized strategy to win with
probability 1-e for all e
56
Winning Conditions w-regular sets
Safety
Reachability
B
Always in B
Reach B
B
Büchi
coBüchi
Visit B infinitely often
Eventually forever B
B
B
1
2
3
0
Rabin chain
The highest index visited infinitely often is even
self dual
57
Büchi and co-Büchi Games

• Büchi Maximal probability of visiting a set U
  infinitely often
• coBüchi Maximal probability of eventually
  always staying in a set U

n y. m x. (( U Æ Ppre(x)) Ç (U Æ Ppre(y)))
m x. n y. (( U Æ Ppre(x)) Ç (U Æ Ppre(y)))
58
Büchi and co-Büchi Games

• Strategy construction uses arguments similar to
  the safety case
• Reach U, then reach the Büchi state again
• For given e, first play an e/2 optimal strategy,
  then an e/4 optimal strategy, etc
• Optimal strategies may not exist
• e-optimal strategies for Büchi games may require
  infinite memory

59
Rabin-chain games

• m calculus algorithm
• lN-1 m x1 n x0. Çi0N-1 (Ci Æ Ppre (xi))
• The classical algorithm EJ91 for boolean
  turn-based game has an identical syntactic form
• But the proof is different
• Infinite memory e-optimal strategies exist

60
Rabin-chain games

• Winning condition
• Let C S ! 0, , N-1 be a coloring of the
  states
• A trace satisfies the Rabin-chain condition if
  the maximum color appearing infinitely often is
  even.
• All LTL games can be reduced to a Rabin-chain
  game on a product structure

61
Algorithms for Concurrent games

• The m calculus expressions give fixpoint
  characterizations
• Problem Games do not have order field property
• A game with all rational constants can be
  irrational
• So no straightforward reduction to LP (as for
  MDP)

62
Reachability Game
a,b
a,b
s
t
u
Reach u (t) (-32p 5)/5
63
Algorithms for Concurrent games

• However solution sets are semi-algebraic!!
• Ppre is semialgebraic in its arguments
• Fixpoint expressions can be expressed in (R, ,
  .)
• Using Tarskis Theorem, this gives an algorithm
  to check if solution is within e of some value
• Doubly exponential algorithm
• Theorem ChatterjeeJurdzinskiM03 Concurrent
  reachability games can be ?-approximated in NPÅ
  coNP

64
Entering Level 3

• Non Zero Sum Games

65
Non Zero Sum Games

• So far, our games had two players
• Player 1s goal was ?
• Player 2s goal was ?
• Strictly competitive!
• But systems are not (always) malicious
• Usually player 1 has a goal ?, player 2 has a
  goal ?
• Each is happy to ensure his own goal
• These games are naturally studied as non zero sum
  games
• Look for equilibrium solutions

66
Simple Example
(s,s), (ns,ns)
(n,s)
(s,n)
(n,s)
(s,n)
(n,s)
(s,n)
67
History Non Zero Sum Games

• Every finite n-player game has an equilibrium
  Nash50
• Complexity of finding a Nash equilibrium is open
  Pap94,Pap01
• Discounted stochastic n player games have a Nash
  equilibrium Fick64,MertensParthasarathy86
• 2-player nonzero sum stochastic games with
  limiting average payoff Vieille00
• Closed sets SuddherthSecchi02
• Open Sets (Reachability) ChatterjeeJurdzinskiM03
  
• (This talk)

68
Nash Equilibrium in Reachability Games

• In the rest of the talk we sketch a proof for 2
  players.
• First some definitions.
• A non zero sum reachability game consists of
• A concurrent game G
• Two sets of states S1 and S2 of G
• Player 1s goal is to get to S1
• Player 2s goal is to get to S2
• Given strategies ?1 and ?2, Valuei(?1,?2) is the
  probability with which the stochastic process
  visits Si

69
Nash Equilibrium in Reachability Games

• A pair of strategies (?1, ?2) is a Nash
  equilibrium if
• For all ?1, ?2
• Value2(?1, ?2) Value2(?1, ?2)
• Value1(?1, ?2) Value1(?1, ?2)
• That is, neither player has any advantage in
  deviating from the equilibrium strategy
• Note Existence of Nash equilibrium is not
  trivial, as the game is not finite stage

70
Nash Equilibrium in Reachability Games

• A pair of strategies (?1, ?2) is an ?-Nash
  equilibrium if
• For all ?1, ?2
• Value2(?1, ?2) Value2(?1, ?2) ?
• Value1(?1, ?2) Value1(?1, ?2) ?
• That is, neither player has more than ? advantage
  in deviating from the equilibrium strategy.

71
Total Reward Games

• In a total reward game, each player gets some
  reward at each state, and the total reward is the
  sum of all rewards obtained at each stage in the
  game
• From a reachability game, we can construct a
  total reward game as follows.
• Take 4 copies of the game.
• The game starts in copy 1.
• When player 1 reaches any state in his goal, he
  gets reward 1 and the game moves to copy 2.
• When player 2 reaches any state in his goal, he
  gets reward 1 and the game moves to copy 3.
• If they reach their goals simultaneously, the
  game moves to copy 4 (each get reward 1).
• Player 1 gets reward 0 in copies 2 and 4, player
  2 gets reward 0 in copies 3 and 4.

72
From Reachability to Discounted Games

• A ?-discounted reachability game is played as
  follows.
• At each stage, the game stops with probability ?,
  and continues with probability 1- ?.
• Theorem A ?-discounted reachability game has a
  Nash equilibrium in memoryless strategies.
• The proof is a standard application of Kakutanis
  fixpoint theorem.

73
Markov Decision Processes

• A Markov decision process (MDP) is a one player
  game.
• Reachability, discounted reachability is defined
  on MDPs by restriction from games.

74
Main Theorem

• Theorem A non zero sum reachability game has an
  ? Nash equilibrium in memoryless strategies for
  all ?.
• Idea of proof
• Consider a Nash equilibrium in the ?-discounted
  reachability game for suitable ?. This
  equilibrium can be approximated by strategies of
  a simple form (k-uniform)
• This strategy profile is an ?-Nash equilibrium in
  the original game.
• This is because if I fix the strategy of player
  2, in the resulting MDP, the value is close
  to the discounted value
• Similarly for player 1

75
Open Question

• Is there a nonzero sum version of Martins
  Theorem?
• For turn based games, the answer is yes.
• In fact, there is a general construction to
  construct Nash equilibria if corresponding two
  player zero sum games admit deterministic winning
  strategies ThuijsmanRaghavan97.
• A careful study of Martins determinacy proof
  shows that we can construct ?-optimal pure
  strategies
• So turn based probabilistic games with Borel
  payoffs have ?-Nash equilibria
• From a result by ChatterjeeJurdzinskiHenzinger04
  , it follows that turn based probabilistic games
  with Rabin-chain objectives have Nash equilibria.
• This is the best we can do there are turn-based
  games with no (exact) Nash equilibria

76
Credits

• Work done in collaboration with
• Luca de Alfaro
• Krishnendu Chatterjee
• Tom Henzinger
• Marcin Jurdzinski

77
Thank You!

• http//www.eecs.berkeley.edu/rupak