Introduction to Soar

1
Introduction to Soar

• John E. Laird
• University of Michigan
• September 2005

http//sitemaker.umich.edu/soar laird_at_umich.edu
Supported in part by ONR DARPA
2
Requirements for Human-Level AI

• Behave flexibly as a function of the environment
• Exhibit adaptive (rational, goal-oriented)
  behavior
• Operate in real time
• Operate in a rich, complex, detailed environment
• Perceive an immense amount of changing detail
• Use vast amounts of knowledge
• Control a motor system of many degrees of freedom
• Use symbols and abstractions
• Use language, both natural and artificial
• Learn from the environment and from experience
• Live autonomously within a social community
• Exhibit self-awareness and a sense of self
• All of these can apply to almost any task

(Adapted from Newell, Rosenbloom, Laird, 1989)
3
Cognitive Architecture

• Fixed mechanisms underlying cognition
• Memories, processing elements, control
• Purpose
• Bring all relevant knowledge to bear in select
  actions to achieve goals
• Examples
• Soar, ACT-R, EPIC, ICARUS, 4D/RCS, 3T, CLARION,

Knowledge
Goals
Architecture
Body
Task Environment
4
Generic Architecture
Long-Term Memories
Episodic
Procedural/Skill
Semantic/Concept
Episodic Learning
Rule Learning
Semantic Learning
Short-Term Memory
Control Procedure
Perception
Action
Body
5
Core Soar
Agent in real or virtual world
Agent in new situation/state
Agent in new state
6
Core Soar
Long-term memory Possible actions, effects of
actions, facts, episodes, expected rewards,
preferences, Short-term memory Current
situation, goals, intentions, hypothetical
states,
?
7
Soar Basics

• Operators Deliberate changes to
  internal/external state
• Activity is a series of operators controlled by
  knowledge
• Input from environment
• Elaborate current situation rules
• Propose and compare operators via preferences
  rules
• Select operator
• Apply operator Modify internal data structures
  rules
• Output to motor system
• Basic human cycle 50 msec

8
Soar Structure
Long-term Procedural Memory Production Rules
Decision Procedure Rule Matcher GUI
Short-term Declarative Memory
Preference Memory
Working Memory
Perception
-, ,
Action
9
Example Working Memory
A
B
Working memory is a graph. All working memory
elements must be linked directly or indirectly
to a state.
10
Water Jug Problem

• You are given two empty jugs. One holds five
  gallons of water and the other holds three
  gallons.
• There is a well that has unlimited water that you
  can use to completely fill the jugs. You can also
  empty a jug or pour water from one jug to
  another.
• There are no marks for intermediate levels on the
  jugs.
• The goal is to fill the three-gallon jug with one
  gallon of water.

11
Operators and States

• Operators
• Fill a jug from the well.
• Empty a jug into the well.
• Pour water from a jug to a jug.
• States
• Jug-a
• Volume 5 gallons
• Contents X gallons
• Empty Y gallons
• Jug-b
• Volume 3 gallons
• Contents M gallons
• Empty N gallons

12
Soar 101Internal Problem Solving
Propose Operator
Compare Operators
Apply Operator
Select Operator
Propose Operator
Compare Operators
Select Operator
Apply Operator
Elaborate State
Elaborate State
If selected operator fills jug --
contents gets volume
If no jugs, -- propose operator initialize
state
If selected operator is initialize state --
contents 0 volume 5 contents 0
volume 3
If jug empty 0, -- propose operator to
fill jug
If operator empties a jug -- operator If jug has content , volume , --
empty -
Operator initialize state
Operator fill j1
Operator proposal fill j1, fill j2
Operator proposal fill j2
j1 volume 5 contents 0
j1 volume 5 contents 5
empty 5
empty 0
j2 volume 3 contents 0
empty 3
13
Water Jug Operators

• Initialize-water-jug
• Fill a jug from the well
• Empty a jug into the well
• Pour water from a jug to a jug
• For every operator, must define at least two
  rules
• Proposal creates operator structure in working
  memory
• Usually includes name and parameters (fill-jug,
  empty-jug)
• Application tests for selected operator
• Makes changes to the state (based on parameters)
• Can also create selection rules, but not always
  necessary

14
Fill Jug

• Proposal
• If there is a jug that is not full, then propose
  the fill operator.
• Application
• If the fill operator is selected for a jug,
• then change the contents of that jug to its
  volume.

sp water-jugproposefill-water-jug (state
name water-jug jug )
( empty 0) -- ( operator
) ( name fill fill-jug )
sp water-jugapplyfill-water-jug (state
name water-jug operator
jug ) ( name fill
fill-jug ) ( volume
contents ) -- ( contents
contents -)
Only match if value 0
Causes WME to be removed
means indifferent (a random selection will be
made)
15
Elaboration of empty

• If a jug has volume v and contents c, then it has
  empty v c.
• empty is instantiation-supported i-support
• When this instantiation retracts, the working
  memory element is removed.
• The rule may match new values and produce a new
  working memory element.

sp water-jugelaborateempty (state name
water-jug jug ) ( volume
contents ) -- ( empty
(- ))
Subtraction of from
16
Persistence!

• Actions of non-operator application rules retract
  when rule no longer matches
• No longer relevant to current situation
• Operator proposals and state elaboration
• Instantiation-support i-support
• Rule doesnt test operator and modify state.
• Elaborate state
• Propose operator
• Create operator preferences
• Actions of operator application rules persists
  indefinitely
• Otherwise actions retract as soon as operator
  isnt selected
• Operators perform non-monotonic changes to state
• Operator-support o-support
• Rule tests a selected operator and modifies the
  state
• Operator application

17
Pour two implementations

• If the source jug holds less than or equal to the
  jug being filled
• If the source jug holds more than the jug being
  filled

No single rule can do both because two different
stopping criteria Stop when jug is filled or
when source jug runs out of water.
18
Goal Detection

• If there is a jug with volume three and contents
  one,
• then write that the problem has been solved and
  halt.

sp water-jugdetectgoalachieved (state
name water-jug jug ) (
volume 3 contents 1) -- (write
(crlf) The problem has been solved.) (halt)
19
Water Jug Problem Space
20
Simple Control Knowledge

• Rules that influence operator selection using
  preferences

sp water-jugselectemptyworst (state
name water-jug operator )
( name empty) -- ( operator

name water-jug operator )
( name empty) -- ( operator 30)
21
Summary of Preferences
Acceptable Reject - Better
Worse Best
Worst
Indifferent Indifferent
0-100
22
Impasses and Subgoals/Substates

• Problem
• What to do when inconsistent of incomplete
  knowledge?
• Approach
• Detect impasses in decision procedure tie,
  conflict, no-change
• Create substate with augmentations that define
  impasse
• Superstate
• Impasse no-change, tie, conflict,
• Item tied or conflicted operators
• Impasse resolved when decision can be made

23
Implications

• Substate is really meta-state that allows system
  to reflect
• Substate goal to resolve impasse
• Generate operator
• Select operator (deliberate control)
• Apply operator (task decomposition)
• All basic problem solving functions open to
  reflection
• Operator creation, selection, application, state
  elaboration
• Substate is where knowledge to resolve impasse
  can be found
• Hierarchy of substate/subgoals arise through
  recursive impasses

24
Operator No-change

• What happens if we dont have application
  knowledge?
• No apply rule

superstate
Pickup
Fill-jug
Putdown
S2
(j1 picked-up yes)
(s2 superstate s1 impasse no-change
attribute operator)
(j1 filled-up yes)
(j1 picked-up yes filled-up yes
contents - contents )
(s2 jug j1) Created by elaboration
25
Multi-Column Addition Decomposition
Process Column
Initialize
Next Column
Process Column
Compute result
Carry
Get-digit1
Get-digit2
Write Result
Semantic Memory Retrieval
Create new column
Compute Result
Write digit1
Semantic Memory Retrieval
26
TacAir-Soar Task Decomposition
Execute Mission
Fly-route
Fly-Wing
Intercept
Ground Attack
Achieve Proximity
Employ Weapons
Search
Execute Tactic
Scram
Get Missile LAR
Select Missile
Get Steering Circle
Sort Group
Launch Missile
Lock Radar
Lock IR
Fire-Missile
Wait-for Missile-Clear
250 goals, 600 operators, 8000 rules
27
Substate Results

• Problem
• What are the results of substates/subgoals?
• Dont want to have programmer determine via
  special syntax
• Results should be side-effect of processing
• Approach
• Results determined by structure of working memory
• Structure is maintained based on connectivity to
  state stack
• Result is
• Structure connected to superstate but created by
  rule that tests substate structure
• Structure created in substate that becomes
  connected to superstate
• Remove everything that isnt a result with
  impasse resolved
• Substate Approach Implications
• Results do not always resolve impasses
• One result can cause large substate structure to
  become result

28
Result Examples
29
Tie Impasses/SubgoalsInsufficient Control
Knowledge
Apply Operator
Elaborate State
Propose Operator
Compare Operators
Select Operator
Propose Operator
Compare Operators
Select Operator
Elaborate State
Apply Operator
If jug empty 0 -- propose operator to
fill jug
Impasse!
fill j1
fill j2
Evaluate-operator (fill j1)
Substate
Evaluate-operator (fill j2)
30
Combined Impasses/Subgoals
Apply Operator
Elaborate State
Propose Operator
Compare Operators
Select Operator
Propose Operator
Compare Operators
Select Operator
Elaborate State
Apply Operator
If jug empty 0 -- propose operator to
fill jug
Impasse!
fill j1
Substate
fill j2
Evaluate (fill j3)
Evaluate (fill j3)
fill j4
Evaluate (fill j4)
Copy
empty j3
fill j4
pour j3 j4
31
Persistence of Results

• Problem
• What should be the persistence of results?
• Based on persistence of structure in subgoal?
• Could have different persistence before and after
  chunking
• Operator in subgoal could create elaboration of
  superstate
• How maintain i-support after substate removed?
• Approach
• Build justification that captures processing
• Analyze justification
• Elaborate, propose, select, apply
• Assign o/i-support
• Maintain justification for i-support until result
  removed

32
Justification Example
33
Learning/Chunking

• Problem
• Subgoals discover knowledge to resolve impasses
  but it is lost after each problem solving episode
• Approach
• Automatically build rules that summarize
  processing
• Variablize justifications chunks
• Variablizes only identifiers no constants
• Loses ,
• Conditions include only those test required to
  produce result implicit generalization
• Chunks are built as soon as a result is produced
• Immediate transfer is possible
• One chunk for each result, where a result
  consists of connected WMEs that become results
  at the same time
• Different results can lead to very different
  conditions
• Improves generality of chunks

34
Chunk Example
35
Chunking Analysis

• Converts deliberate reasoning/planning to
  reaction
• Generality of learning based on generality of
  reasoning
• Leads to many different types learning
• If reasoning is inductive, so is learning
• Soar only learns what it thinks about
• All learning is impasse driven
• Learning arises from a lack of knowledge

36
Subgoal Structures
Impasse!
fill j1
fill j2
Substate
Evaluate (fill j3)
Evaluate (fill j3)
S2
fill j4
Evaluate (fill j4)
empty j3
fill j3
S3
pour j3 j4
(s1 jug j1 j2) (j1 volume 5 contents 0) (j2
volume 3 contents 0) (s2 superstate s1
impasse tie item o1 o2 operator
o4) (o4 name evaluate super-operator o1)
(s3 superstate s2 impasse no-change
jug j3 j4) (j3 volume 5 contents 0) (j4
volume 3 contents 0)
37
Soar 103 Eaters
Output
Input
Input
Propose Operator
Propose Operator
Compare Operators
Apply Operator
Select Operator
Compare Operators
Select Operator
Apply Operator
Output
If cell in direction is not a wall, --
propose operator move
If operator will move to a bonus food and
operator will move to a normal food, --
operator
If an operator is selected to move --
create output move-direction
If operator will move to a empty cell --
operator move-direction North
North East South North East South East North South
38
Tie Subgoals and Chunking
Output
Input
Input
Propose Operator
Compare Operators
Propose Operator
Compare Operators
Apply Operator
Select Operator
Select Operator
Chunking creates rules that create preferences
based on what was tested
North East South East North South
Evaluate-operator (North)
Evaluate-operator (South)
Evaluate-operator (East)
5
10
10
Chunking creates rule that applies
evaluate-operator
North
10
39
Soar 9
Long-Term Memories
Episodic
Procedural
Semantic
Chunking
Episodic Learning
Reinforcement Learning
Semantic Learning
Short-Term Memory
Appraisal Detector
Decision Procedure
Perception
Action
Body
40
Where do new pieces fit?

• Reinforcement learning
• Modifies numeric preferences for operators
• Specializes rules if no converging
• Episodic Memory and Semantic Memory
• Deliberate retrieval from long-term memory
• Provide additional knowledge sources in impasses
• Chunking eliminates need for deliberate retrieval
• Emotion
• How does cognition interact with body?
• I/O
• Need more realistic perception and action models
• Mental Imagery/Visualization
• Stub from perception with extra actions
• Will impact episodic and semantic memories

41
System Architecture
Soar Kernel
Soar 8.6 Kernel (C)
gSKI
Higher-level Interface (C)
Encodes/Decodes function calls and responses in
XML (C)
KernelSML
SML
Soar Markup Language
Encodes/Decodes function calls and responses in
XML (C)
ClientSML
Wrapper for Java/Tcl (Not needed if app is in C)
SWIG Language Layer
Application
Application (any language)
42
Features

• Embedded or socket connections
• Multiple language support for applications
• Combined debugging and I/O interface
• Dynamic connection/disconnection
• Tools less tightly tied to particular Soar
  version
• Faster than earlier Soar interfaces
• Logical structure to output from kernel