The Hybrid DeliberativeReactive Paradigm

1
The Hybrid Deliberative/Reactive Paradigm
The City College of New York
Department of Electrical Engineering
Group Member Jik Cheung Yongwen Zhu
Yayi Hu Xuezhou
Ma Junjun Li
2
Chapter Objectives

• Describe the hybrid paradigm in terms of SAP and
  sensing organization.
• Distinguish the responsibilities between the
  deliberative layer and reactive layer.
• List the basic components of a Hybrid
  architecture sequencer agent, resource manager,
  cartographer, mission planner, performance
  monitoring and problem solving agent.
• Identify the difference between managerial, state
  hierarchy and model-oriented styles of Hybrid
  architectures.
• Be able to describe the use of state to define
  behaviors and deliberative responsibilities in
  state hierarchy styles of Hybrid architectures.

3
Overview

• Reactive Paradigm is the major trend by the end
  of the 1980s.

• However, the robot could not
• Remember the state of the robot/world
• Plan optimal trajectories
• Make maps
• Monitor its own performance
• Select the best behaviors for a task
• Reactivity more art then science?
• Should planning be reintroduced?

4
Deliberative Vs. Planning

• Not all of these activities involve Planning
• Make maps
• Monitor its own performance
• Select the best behaviors for a task
• To differentiate this from path planning, the
  term deliberative was coined.

5
Hybrids

• How can slow planning be intergraded with fast
  reactivity?
• Five examples of architectures will be
  illustrated AuRA, SFX, 3T, Saphira and TCA.
• First Opinion The worst of both worlds!
• Reactive systems for unstructured worlds
• Hierarchical systems for knowledge-rich worlds
• Nowadays The best of both worlds!
• Reactive functions for low level control
• Deliberation for higher level tasks

6
Hybrid Paradigm
Organization Plan, Sense-Act
7
Motivation of Hybrids

• Cohesion (object oriented programming)
• Reactivity
• Short time horizon (Present)
• No global knowledge
• Work with sensors and actuators
• Deliberation
• Long time horizon (Pass, Future)
• Global knowledge
• Work with symbols
• Multi-tasking
• Deliberative functions execute in parallel with
  reactive functions.

8
Sensing Organization

• The Map (World Model)
• Can have its own sensors
• Can eavesdrop on other sensors
• Can act as virtual sensor

9
Skill Vs. Behaviors

• Not purely reflexive
• Reflexive (response to stimulus)
• Innate (virtual sensor turns behavior on or off)
• If power is low, charge
• Learned
• Retain feedback to determine best behavior
  sequence to instantiate next time
• More complex emergent behaviors
• Behavior sequences

10
Connotations of Global

• Global isnt always truly global in Hybrids.
• Behavioral Management
• Planning which behaviors to use requires
  knowledge about current and future world state
• Performance monitoring
• Detecting task progress and sensor confliction
  require knowledge about the robot hardware and
  the overall goals.

Nonetheless

• "Globle" - "Deliberative"

• "Local" - "Reactive"

11
Common Components

• Sequencer
• Generates a sequence of behaviors
• Resource Manager
• Allocates resources to behaviors
• Cartographer
• Creates, stores, maintains, accesses map
  information
• Mission Planner
• Interact with human and create a plan to achieve
  a goal
• Performance Monitor/problem solver
• Determines whether the robot is making progress
  toward its goal

12
Architecture Styles

• Managerial (division of responsibility as in
  business)
• AuRA
• SFX
• State Hierarchies (strictly by time scope)
• 3T
• Model-Oriented (Model serve as virtual sensors)
• Saphira
• TCA

13
Styles of hybrid architectures? Managerial
styles? State hierarchies styles?
Model-oriented styles
14
Managerial Architectures Description
-- top agents high level planning
?
subordinate agents refine plan, gather
resources
? lowest level
agents ? AuRA
Architectures ? SFX Architectures
15
? Autonomous Robot Architecture (AuRA)It
consists of five subsystems -- planner
responsible for mission and task planning
-- cartographer all map making, reading
functions -- motor motor schema --
sensor -- homeostatic control
modify the relationship between behaviors by
changing the gain as a function of robot or
other constraints
16
AuRA Architectural Layout
17
The table below summarizes AuRA in term of the
common components and style of emergent behavior
18
? Sensor Fusion Effects (SFX)description It
is an extension to AuRA. The extension was to add
modules to specify how sensing and handling
sensor failure.
19
Deliberative layers -- Mission planner
acts as a CEO giving a directions --
effector -- Task -- Sensor All
of three of above determine the best allocation
of effect, sensing resource and perceptual
schema. -- Cartographer map making, path
planning
20
SFX (Sensor Fusion Effects)
21
Reactive layersAll these layers reflect to
------- strategic behaviors and
tactical
behaviorsTactical behavior serves as filter on
strategic commands to ensure to robot acts in a
safe manner in as close accordance with the
strategic intent as possiblethe interaction of
strategic and tactical behaviors is still
considered emergent behavior
22
Tactical Behaviors
23
The table below summarizes SFX in term of the
common components and style of emergent behavior
24
State-hierarchy Architectures(3 layers)
25
? 3 tiered (3T)Used for planetary rovers
underwater vehicles
robot assistants for astronauts
26
Structure -- planner setting goal and
strategic plans -- sequencer select a
set of primetive behaviors develop a task
network -- skill manager in this layer
the skills have associated events to verify
explicitly that an action has had to correct
effect
27
3T Architecture
28
The table below summarizes 3T in term of the
common components and style of emergent behavior
29
Model-oriented Architectures two of
best-known model-oriented architecture?Saphira
architecture?Task Control Architecture
30
?Saphira Architecture -- PRS-Lite it is
capable of taking natural language voice commands
from humans and then operationalizing that into
navigation tasks and perceptual recognition
routines. -- virtual sensor -- navigation
tasks manage the behaviors -- LPS (Local
Perceptual Space) determine the planning
and execution improve the quality of the
robots overall behavior
31
Saphira Architecture
32
The table below summarizes Saphira in term of
thecommon components and style of emergent
behavior
33
?Task Control Architecture (TCA) -- Task
Scheduling (Mission Planner) determine the
goal and order of execution -- Path Planning
(Cartographer) -- Navigation (Sequencer)
to determine what the robot should be looking
for, where it is, where it has been. --
Obstacle Avoidance To factor in not only
obstacle but how to respond with a smooth
trajectory for the robots current velocity.
34
TCA
35
The table below summarizes TCA in term of the
common components and style of emergent behavior
36
Basic Important concept

• Paradigm
• Paradigm is both a way of looking at the world
  and an implied set of tools for solving problems.
• Sense, Plan, Act.
• Commonly accepted robotic primitives.
• Robotics have to go through these three, or at
  least two process to complete a mission.
• Local Processing and Global World Model
• Local sensor data used in specific for each
  function.
• Global all sensor data is processed to single
  model.

37
Hierarchical Paradigm

• What are the two main features?
• Robot operates in a top-down fashion.
• All sensor data tends to be gathered to one
  global world model. A single representation that
  planner can use to rout the action.

38
Reactive Paradigm

• What are the two main features?
• Throw out planning all together.
• The inputs to an act are the direct output of a
  sensors.
• examine living example of intelligence.

• ACT

39
Hybrid Paradigm

• Features of Hybrid Deliberative/Reactive Paradigm
• It is reactive planning, Planning to subtask is
  done at one step.
• Deliberative planning take a long time comparing
  to the time of reactive execution
• Sensor data go directly to each behavior but is
  also available to the planner for construction of
  task-oriented global world model.
• Model-based Architecture focuses on the creation
  and maintenance of a global world model.

40
Hybrid Paradigm

• The basic models of Hybrid Paradigm
• Sequencer generate a set of behaviors for
  subtasks.
• Resource manger allocate resources to behavior
• Cartographer for creating, storing, maintaining
  map or spatial information.
• Mission Planner interact with man, construct a
  mission plan.
• Performance Monitoring monitor the process of
  the executing, Its self-awareness.

41
Hybrid Paradigm
42
Hybrid Paradigm
Robot Primitive
output
Input
Information( sensed and
cognitive )
PLAN
Directives
BEHAVIOR
Actuator command
Sensed data
43
Other Hybrid Paradigm

• DARPA UGV Demo II and Demo III.
• Outdoor ground vehicle control and navigation.
  given a map and a set of directions find enemy
  location.
• Reach in automating highway vehicles by European
  Community ESPRIT agency and some United States
  agency
• Autonomous planetary rovers by NASA. Mapping
  planetary surface, planning path.

44
Advantages of Hybrid

• Architecture is highly modular
• Architecture is highly modular of the
  deliberative with object-oriented programming.
• Full knowledge of environment
• Software agents can use agent-specific
  abstractions to exploit the structure of an
  environment in order to fulfill their particular
  role in deliberation.
• Use of Global models
• Global models are only for symbolic functions and
  Planners( sequencers) often produce partial
  plans.

45
Advantages of Hybrid

• Execution is reactive.
• No frame problems.
• In the Hybrid Paradigm almost no the frame
  problems resulted by the Hierarchical.
• Self-consciousness.
• Ensure robustness by monitoring the performance
  of the robot and self-diagnosing, this is called
  self-consciousness.

46
Examples For Good of The Reactive

• Example1
• we dont need to turn all sensed data to global
  model to use in order to accuracy, convince,
  reliability, and saving time.
• Example 2
• in Hierarchical Paradigm it is unwise in a lot of
  practical problems to block out the sensed data
  to Behaviors( Actuator).

47
(No Transcript)
48
(No Transcript)
49
Interleaving Deliberation and Reactive Control

• For navigation
• Deliberation Cartographer( planner) generates a
  complete optimal route, decompose the route to
  segments-waypoints.
• Reactive Control Waypoint can be accomplished by
  behaviors.
• Top-down method
• Deliberative layers decompose the missions to
  finer steps. Reactive layers accomplish the first
  sub-goal.

50
Interleaving Deliberation and Reactive Control

• Bottom-up method.
• Deliberative layers act as virtual sensors. The
  analyzed information as a sensed data input into
  behaviors( reactive layers)-Bottom-up
• Other functions of Deliberations
• In the deliberative layers, sequencer must know
  why a failure and know the need to change the
  behaviors and alert the human supervisor.-self-con
  sciousness.

51
Summary of AI Robotics
52
Ch.1 From Teleoperation to Autonomy

• What is intelligent robots?
• What is the difference between AI and Engineering
  approaches to robotics?
• What is the difference between telepresence and
  semi-autonomous control?

53
What is intelligent robots?

• Mechanical creatures that can function
  autonomously, which means it can sense, act,
  maybe even reason doesnt just do the same thing
  over and over like automation.
• The intelligent robots arose by the development
  of AI since the 1990s.

54
Teleoperation

• Teleoperation is that a human operator controls a
  robot from a distance.
• It is a ideal solution for controlling remotes
  because AI technology is nowhere near human
  levels of competence, especially in terms of
  perception and decision making.
• Cons Cognitive fatigues communications dropout
  communications bandwidth communications lag

55
Add more intelligence to the early teleoperation

• Telepresence
• providing sensory feedback to the point that
  teleoperator feels they are present in robots
  environment by adding more cameras.
• Semi-autonomous control
• human is involved, but routine or safe portions
  of the task are handled autonomously by the robot
• It is really a type of mixed-initiative

56
The Seven Areas of AI

• knowledge representation
• How does the robot represent its world, task, and
  itself.
• understanding natural language
• Natural language is usually challenging, it is
  not only talking about looking up words from a
  dictionary by understanding.
• Learning
• A robot could be programmed by just watching a
  humans behaviors.

57
The Seven Areas of AI

• planning and problem solving
• The ability to plan actions and solve problems
  with those plans
• Inference
• Inference is generating an answer when there is
  no complete information
• Search
• Search means efficiently examining a knowledge
  representation of a problem to find the answer.
• Vision
• The robot can simulate the effects of actions in
  its head

58
Robotics Paradigms

• What are robotic paradigms?
• A paradigm is a philosophy or set of assumption
  and/or techniques which characterize an approach
  to a class of problems.
• There paradigms
• Hierarchical paradigm (Ch. 2)
• Reactive paradigm (Ch. 4)
• Hybrid paradigm (Ch. 7)

59
Ch. 2 Hierarchical paradigm

• The oldest paradigm, and was prevalent from
  1967-1990.
• Under this paradigm, the robot senses the world,
  plans the next action, and then acts.

60
Strips means-ends analysis

• Strips is a variant of the general problem solver
  method, it uses an approach of means-ends
  analysis, where if the robot cant accomplish the
  task in one movement, it picks a action which
  will reduce the difference between what the now
  state versus the goal state.
• To implement Strips, Designer must set up
• World model representation
• Difference table with operators, preconditions,
  add delete lists
• Difference evaluator

61
Strips means-ends analysis

• Strips assumes closed world
• Closed world world model contains everything
  needed for robot (implication is that it doesnt
  change)
• Open world world is dynamic and world model may
  not be complete
• Strips suffers from frame problem
• Frame problem representation grows too large to
  reasonably operate over

62
Representative Architecture

• An architecture is a method of implementing a
  paradigm, of embodying the principles in some
  concrete way.
• The two best known architectures are the Nested
  Hierarchical Controller (NHC) developed by
  Meystel and the NIST Realtime Control System
  (RCS) originally developed by Albus.

63
Evaluating the Two Architectures

• support for modularity
• decomposition by functionality
• niche targetability
• good, both have been used for apps like vehicle
  guidance, mining equipment
• ease of portability to other domains
• unclear, not sure if code could be reusedlots of
  rewriting on previous apps
• robustness
• RCA simulates plans in advance, but not sure what
  it would do with sensor or mechanical failures,
  etc.

64
Advantages and Disadvantages

• Advantages
• It provides an ordering of the relationship
  between sensing, planning, and acting.
• Disadvantages
• Planning for every update cycle, robots had to
  do some type of planning.
• Dependence on a global world model
• Uncertainty did the robots actually finish the
  action? We dont know for sure.

65
Ch. 3 Biological Foundations of the Reactive
Paradigm

• Why explore the biological sciences?
• What are the three levels in a computational
  theory?
• What are animal behaviors?
• Coordination of behaviors, perception, schema
  theory, and more

66
Why do we need to explore the biological sciences?

• Animals and man provide existence proofs of
  different aspects of intelligence.
• The principles of animal intelligence are
  extremely important.
• For examples roboticists may overcome the closed
  world assumption that presented problems with
  shakey by observing the animals behaviors in an
  open world.

67
Marrs Computational Theory

• The levels in the computation theory can be
  stated as
• Level 1 What is the phenomena were trying to
  represent?
• Level 2 How it be represented as a process with
  inputs/outputs?
• Level 3 How is it implemented?

68
Animal Behaviors

• A behavior is a mapping of sensory inputs to a
  pattern of motor actions which then are used to
  finish a task
• Three catagories
• Reflexive
• stimulus-response, often abbreviated S-R
• Reactive
• learned or muscle memory
• Conscious
• deliberately stringing together

69
Coordination and Control of Behaviors

• There are four ways to acquire a behavior, which
  are
• To be born with a behavior (innate)
• Examples Arctic terns.
• To be born with a sequence of innate behaviors.
• Examples mating cycle in digger wasps.
• To be born with behaviors that need some
  initialization (innate with memory).
• Examples bees, which are born with in hives.
• To learn a set of behaviors
• Examples Lions, who are nor born with any
  hunting behaviors.

70
How behaviors are coordinated and controlled--
innate releasing mechanisms (IRM)
Releaser
Pattern of Motor Actions
Sensory Input
BEHAVIOR

• The Releaser acts as a control signal to activate
  a behavior. If a behavior is not released, it
  does not respond to sensory inputs.

71
Perception

• Two functions of perception (can be the same
  percept)
• Release a behavior
• Guide a behavior
• Action-oriented perception (Neisser)
• Planning is not needed to act
• Perception is selective

72
Schema Theory

• Schema theory provides a helpful way of casting
  some of the insights from above into an OOP
  format.
• is generic, equivalent to an object in OOP
• schema specific knowledge (local data)
• procedural knowledge (methods)
• schema intiantation is specific to a situation,
  equivalent to an instance in OOP
• a behavior is a schema, consists of
• perceptual schema
• motor schema

73
Ch. 3 Summary

• A behavior is the fundamental element of
  biological intelligence, and will server as the
  fundamental component of intelligence in most
  robot systems.
• Innate Releasing Mechanisms (IRM) are one model
  of how intelligence is organized.
• Perception in behaviors serves two roles,
  including a releaser for a behavior and a precept
  which guides the behavior.
• Schema theory is an object-oriented way of
  representing and thinking about behaviors.

74
Ch. 4 The Reactive Paradigm

• The Reactive Paradigm was a reaction to the
  Hierarchical Paradigm, and it was heavily used
  between 1988-1992.
• The fast execution time can be achieved by
  throwing away Planning.

RELEASER
behavior
SENSE
ACT
75
Reactive Robots
RELEASER
behavior
SENSE
ACT

• Most apps are programmed with this paradigm
• Biologically based
• Behaviors (independent processes), released by
  perceptual or internal events (state)
• No world models or long term memory
• Highly modular, generic
• Overall behavior emerges

76
Hierarchical Organization isHorizontal

• Horizontal decomposition of tasks into the S, P,
  A organization of the Hierarchical Paradigm.

77
More Biological is Vertical

• The right figure shows that a vertical
  decomposition of tasks into an S-A orgrnization.

78
Architectures

• Historically, there are two main styles of
  creating a reactive system
• Subsumption architecture
• Layers of behavioral competence
• How to control relationships
• Potential fields
• Concurrent behaviors
• How to navigate

79
Subsumption Architecture

• Subsumption has a loose definition of behavior as
  a tight coupling of sensing and acting.
• Higher layes may subsume and inhibit behaviors in
  lower layers.
• The design of layers and their behaviors is
  usually difficult.
• Behaviors are released by the presence of
  stimulus.
• Subsumption solves the frame problem by
  eliminating the need to model the world because
  the behaviors just simply respond to whatever
  stimulus is in the environment.
• Perception is largely direct, using affordances.
• Perception is ego-centric and distributed.

80
Potential Fields

• Potential field styles of behaviors always use
  vectors to represent behaviors and vector
  summation to combine vectors from different
  behaviors to produce an emergent behavior.
• Behaviors are defined as consisting of one or
  more of both motor and perceptual schemas and
  (or) behaviors.
• All behaviors operate concurrently and output
  vectors are summed.
• Behaviors may make varying contributions to the
  overall action of the robot, although they are
  treated equally.
• Perception is usually handled by direct
  perception or affordances.
• Perception can be shared by multiple behaviors.

81
Evaluation of Reactive Architectures

• Support for modularity
• Both decompose the actions and perceptions.
  Subsumption favors a composition suited for a
  hardware implementation, whereas potential fields
  methods for a software-oriented system.
• Niche targetability
• Both have hign targetabilities.
• Ease of portability to other domains
• Subsumption depends on low layers heavily, while
  potential fields usually have no implicit
  reliance on a low layer.
• Robustness
• Neither can be called genuinely robust.

82
Ch. 4 Summary

• The organization of the Reactive Poradigm is
  SENSE-ACT, No PLAN component.
• Under reactive paradigm, behaviors serve as the
  basic building blocks for robot actions.
• Reactive systems also exhibit good software
  engineering principles due to the programming by
  behavior approach.
• At last, two representative architectures are
  subsumption and potential fields. However,
  despite the differences in theory, these two
  systems appear to be largely equivalent
  practically.

83
The key points to understand what is main
characters of AI robotics?

• OOP (Object-Oriented Programming)
• Model of sensing
• Hybrid deliberative/Reactive Paradigm
• Example of our homework3
• Future of Robot

84
What is OOP?
Object-Oriented Concepts tap into this
natural human tendency resulting in an easy to
understand and use language. An automobile
is a very good example of the Object-Oriented
Concept. As humans, it is our natural tendency to
think of an automobile as a single "thing", and
not as a large group of several thousand small
"things". Thinking of the automobile as a single
"thing" helps us deal with the overwhelming
complexity of the whole machine. We would say
simple statements like "Fill her up. or "How
fast are we going?" or "I have a Blue car. "
..... and everyone would understand how those
statements apply to our car.
85
1. Example for OOP Programming

• Using an automobile as an example of an
  Object, the following program shows an example of
  Object Oriented programming
• BobsCar.Speed 50
• If BobsCar.SpeedgtCurrentRoad.SpeedLimit
  Then
• PoliceCar.Mode Chase
• PoliceCar.Target BobsCar
• PoliceCar.Speed
  BobsCar.Speed 10
• End If
• Is it very simple and easy to understand?
• Here, please imagine that if we do not use
  OOP, what should our
• program look like?

86
2. How behaviors can be implemented using OOP
constructs such as classes?

• Recall from software engineering that an
  object consists of data and method, also called
  attributes and operations. And as noted before,
  schemas contain specific knowledge and local data
  structures and other schemas. So, a schema as a
  programming object will be a class. Its defined
  as below

87
3. Example move-to-go behavior

• 1) We put a robot in an empty arena with
  Coca-cola cans in random location and a blue
  recycling bin in a corner.
• 2) The behaviors needed is picking up a red can
  and moving to a blue bin. But we write a single
  generic behavior move_to_goal (color) to deal
  with both behaviors.
• 3)The behavior move_to_goal consist of a
  perceptual schema, which will be called
  extract-goal and a motor schema, which used an
  attractive field. extract-goal uses the
  affordance of color to extract where the goal is
  in the image, and then computer the angle to the
  center of the colored region and size of the
  region.
• The table below implies some important
  points about programming with behaviors

88
4) The attraction motor schema takes that
percept and is responsible for using it to
turn the robot to center on the region and
move forward.
5) Two schemas are both independent. The
perceptual schema doesnt know the existence
of motor schema.
89
1. Model of sensing
Sensor/transducer----------gtBehavior--
-----------gtAction
90
2. Behavioral Sensor Fusion
Perception in a reactive robot system has
two roles 1)to release a behavior
2)to support or guide the action of the
behavior All sensing is behavior-specific,
where behaviors map tap into the same
sensors, but use the data independently of
each other.
91
The Hybrid Deliberative /Reactive Paradigm
1. It can be thought as PLAN, then SENSEACT. 2.
The SENCEACT portion is always done with
reactive behaviors, where PLAN includes a
broader range of intelligent activities. 3.
Planning can be interviewed with execution. 4.
Architecture usually encapsulate functionality
into modules. The basic modules are mission
planner, behavior manager, performance
monitor. 5. State-hierarchies divide deliberation
and reaction by the state, available to the
modules or agents operating that layer. Three
states are Past, Present, Future.
92
Example
Do we use PLANSENSEACT concept? Modules
concept? State-hierarchies? Planning can be
interviewed with execution?
93
Future of Robot
Enabling technologies Enabling technologies
ranging from sensors to radio communications
and navigation aids are all accelerating
logarithmically. The ubiquitous acceptance of
wireless LAN systems, the plunging costs of
video cameras and processors, the availability
of affordable laser navigation systems, and the
ever-increasing accuracy and dropping cost of
GPS navigation receivers are all combining to
make autonomous robots potentially cheaper and
ever more capable. At least as important, we
now have enormous resources
94

• in human experience. Countless software
  engineers and
• academics have spent endless hours
  developing concepts
• of modeling and control that are just as
  much part of the existing robotics
• toolbox as any sensor or processor. As a
  result, only the integration of
• these elements is required for new robotic
  configurations to burst onto the
• scene with blinding speed.
• Social forces
• The social issues already discussed are
  pushing customers to look for new
• solutions to performing many of the tasks
  that now require manual labor.
• These are tasks which autonomous robots can
  easily provide. Slowly but
• surely, a few venture capitalists (real
  ones) are beginning to make
• investments in companies like iRobot, and
  the industry is beginning to
• gain a little attention.

95

• Thank you for your time!