Controlling Mobile Robots with Distributed Neuro-Biological Systems

1
Controlling Mobile Robots with Distributed
Neuro-Biological Systems

• Sebastian Gutierrez-Nolasco (UCI)
• Nalini Venkatasubramanian (UCI)
• Alfredo Weitzenfeld (ITAM)

Contact info seguti_at_ics.uci.edu
2
Biologically Inspired Robotic Systems

• Nature has always been a source of inspiration in
  the development of autonomous robotic systems
• Ethology
• Animal behavior-based simulation
• Interaction with the environment is usually
  oversimplified
• Lack of strong biological basis for their working
  assumptions
• Lack of any formal underpinnings for the
  simulation results
• Neuroethology
• behavior related to neurobiological structure
• Replicate brain models to provide credible and
  general animal behavior
• Provide inspiration for further robotics
  architectures
• More complex and accurate than ethology systems
• Enable experimentation
• Experimentation requires real-time performance

3
Neuroethological robotic systems

• Super Robots
• Incorporate extensive processing capabilities
• Bulky
• Expensive
• Inexpensive Robots
• Smaller and inexpensive robots connected to a
  network of processing nodes
• Concerns
• Real-time performance
• Unpredictable communication environment affects
  robot performance

4
Challenges of Biologically Inspired Autonomous
Robots

• A neural model may take hours of processing time
• Simulation of multiple neural networks require a
  distributed processing environment
• A typical retina model may consist of more than
  100,000 neurons and 500,000 interconnections
• Biologically inspired robotics demand
  sophisticated image processing techniques
• Communication intensive tasks are required
• Autonomous robotic agents have real-time and
  processing restrictions, as well as power
  awareness requirements
• Battery usage is a major concern in mobile robots

5
Developing Biologically Inspired Robot
Architectures

• Developing software for autonomous mobile robots
  is complex
• Highly heterogeneous methods for capturing and
  processing sensor information
• Multiple sensory input devices
• Sensory input multi-granularity
• Communication is error-prone due to unpredictable
  interference and failures
• partial and complete failures
• Unreliability and disconnection
• Varying available bandwidth

6
Our Approach

• Develop an embedded architecture capable of
  conducting neuroethological robotic
  experimentation
• Inexpensive small robots communicate (wireless)
  with distributed computational resources
• Neural models are distributed in multiple
  processing nodes
• Adaptive robotic middleware optimizes robot
  communication in response to varying network
  conditions

7
Structure of the Talk

• Neuroethological Modeling
• Study animal behavior and corresponding neural
  structure as inspiration to robotic architectures
• Embedded Mobile Robots
• Develop distributed wireless robot architectures
  capable of efficient neural processing
• Adaptive Middleware
• Achieve real-time computation and adapt embedded
  architecture to varying network conditions
• Internet Based Robotics
• Enable remote robot task development and
  experimentation

8

• Neuroethological Modeling
• Study animal behavior and corresponding neural
  structure as inspiration to robotic architectures
• Embedded Mobile Robots
• Develop distributed wireless robot architectures
  capable of efficient neural processing
• Adaptive Middleware
• Achieve real-time computation and adapt embedded
  architecture to varying network conditions
• Internet Based Robotics
• Enable remote robot task development and
  experimentation

9
BehaviorPraying Mantis - Chantlitaxia Arkin,
Ali, Weitzenfeld and Cervantes, 2000
10
BehaviorFrog and Toad - Rana Computatrix
Arbib 1987, Cervantes 1990
-
PS NMO
PS MO



-
-
PS Mate
PS Prey
PS Pred
Mate-Pair S
Prey-Acq S
Pred-Av S
Find-Loc S
Moving-Object S
Non-Moving-Object S
Perceptual Schema (PS)
Main Schema (S)
11
BehaviorToad Prey Acquisition Cervantes 1985
Stimulus
Response
Mobile visual stimulus in lateral visual
field (monocular perception)
Orientation
Mobile visual stimulus in binocular visual
field (short distance)
Binocular fixation
Attack
Mechanic stimulus in mouth and pharynx receptors
Snap
Clean
12
BehaviorToad Prey Acquisition with Detour
Behavior Before and After Learning Corbacho and
Arbib 1995
10cm Barrier
20cm Barrier Before learning
20cm Barrier After learning
13
Schema Computational Model
dout
din
1
1
...
...
...
...
Schema
din
dout
n
m
data in
Schema Level 1
data out
Schema Level 2
Schema Level
Neural Level
Other Processes
Neural
14
Neural based Behavior Toad Prey Acquisitions
and Predator Avoidance
Prey Recognizer
Prey Approach
Moving Stimulus Selector
Predator Recognizer
Forward
Predator Avoid
Depth
Visual
Orient
Backward
Sidestep
Static Object Recognizer
Static Object Avoidance
Tactile
Schema Level
Tectum
Neural Level
T5_2
R1-R2
R1-R2
R1-R2
R3
R3
Motor Heading Map
R3
R4
R4
TH10
R4
Retina
Stereo
MaxSelector
PreTectum/Thalamus
15
Toad Prey Acquisition with DetourSimulation
Results
10cm barrier
20cm barrier Before learning
20cm barrier After learning
16

• Neuroethological Modeling
• Study animal behavior and corresponding neural
  structure as inspiration to robotic architectures
• Embedded Mobile Robots
• Develop distributed wireless robot architectures
  capable of efficient neural processing
• Adaptive Middleware
• Achieve real-time computation and adapt embedded
  architecture to varying network conditions
• Internet Based Robotics
• Enable remote robot task development and
  experimentation

17
Embedded Mobile RobotsRobot Hardware
LEGO
OOPIC
18
Embedded Mobile RobotsDistributed Embedded
Architecture
Wireless
Remote Computaional System
Instance 1
Autonomous Robot 1
Internet Server
...
...
...
Autonomous Robot N
Remote Computational System
Instance N
19
Embedded Mobile RobotsDistributed Embedded
Architecture

• Time consuming processes are carried out in the
  (neural) computational system
• Neural processing
• Image processing
• Limited task are carried out in the robot
  hardware
• Sensory input
• Motor output
• Default behavior
• Communication and data transformation is managed
  by the adaptive middleware

20
Embedded Mobile RobotsDistributed Embedded
Architecture
camera
Remote Computaional System
servo
Frame Grabber
Robot
Wireless
CPU (OOPic)
Transceiver
Sensors (tact)
Transceiver
PC
Power stage
motor
motor
21
Embedded Mobile RobotsDistributed Embedded
Architecture
Remote Computational System
Wireless
NSL
NSL
NSL
NSL
Robot
Video/ Image
Video Server
Processing
camera
ASL
ASL
transceiver
tactile
transceiver
Tactile Server
ASL
ASL
motor
Motor Server
NSL
NSL
NSL
NSL
NSL Neural Simulation Language ASL Abstract
Schema Language
22
Embedded Mobile RobotsProcessing cycle
Video capture
Video processing
Model simulation
Model output
(d , ?r , ?c)
Navigation control
23

• Neuroethological Modeling
• Study animal behavior and corresponding neural
  structure as inspiration to robotic architectures
• Embedded Mobile Robots
• Develop distributed wireless robot architectures
  capable of efficient neural processing
• Adaptive Middleware
• Achieve real-time computation and adapt embedded
  architecture to varying network conditions
• Internet Based Robotics
• Enable remote robot task development and
  experimentation

24
Distributed Systems Middleware

• Enables the modular interconnection of
  distributed software
• Abstract over low level mechanisms used to
  implement resource management services
• Concurrent Object Oriented Model
• Separation of concerns and reuse of services
• Customizable, Composable Middleware Frameworks
• Provide for dynamic network and system
  customizations, dynamic invocation/revocation/inst
  allation of services
• Concurrent execution of multiple resource
  management policies

25
Core Resource Management Services

• Core Services - basic services where interactions
  between the application and system can occur.
• Building blocks for other services
• Reduce interactions among many services to
  interactions between a few simple services
• Choosing core services - commonly observed
  patterns
• Recreation of data/services at a remote site
• Capturing approximation of distributed state at
  multiple sites
• Interactions with a global repository

26
TLAM The Two Level Meta-architecture
27
Adaptive Robotic Middleware (ARM)

• Extends the TLAM to
• Optimize information flow between robots and the
  computational system
• Determine how, when and what information should
  be modified in order to match fluctuations in the
  communication environment
• Compose communication protocols to obtain the
  combined benefits - conflicting requirements
• Explicit knowledge of how communication protocols
  compose and interact is required
• Adapt protocols and mechanisms to changing
  communication and power constraints

28
ARM Distributed Embedded Architecture
Remote Computational System
Wireless
ARM
Robot
NSL
NSL
NSL
NSL
Video/ Image
Video Server
ARM
processing
camera
ASL
ASL
transceiver
tactile
transceiver
Tactile Server
ASL
ASL
motor
Motor Server
NSL
NSL
NSL
NSL
NSL Neural Simulation Language ASL Abstract
Schema Language
29
ARM Components

• Communication manager
• Provide and enforce application level
  requirements
• Components
• Oracle
• determine most suitable protocol implementation
  in terms of coverage and efficiency
• Set of communication protocols
• Protocol installer/uninstaller
• Resident ARM module running in the robot
  (resident evil)
• Adaptation manager
• Provide adaptation and monitor mechanisms
  operating at different levels of abstraction
• Reactive
• Triggered when failure to achieve intended
  communication goal is detected
• Proactive
• Triggered when a more efficient communication can
  be achieved under the current environment
  conditions
• Adaptation Repository
• Determine most suitable adaptation strategy to be
  applied

30
ARM Example
31

• Neuroethological Modeling
• Study animal behavior and corresponding neural
  structure as inspiration to robotic architectures
• Embedded Mobile Robots
• Develop distributed wireless robot architectures
  capable of efficient neural processing
• Adaptive Middleware
• Achieve real-time computation and adapt embedded
  architecture to varying network conditions
• Internet Based Robotics
• Enable remote robot task development and
  experimentation

32
Interned based Robotics Web Access
33
Experimental Results 2 Preys
34
Experimental Results 2 Preys and Predator
35
Embedded Mobile RobotsExperimental Results
Prey Acquisition with 10 cm Barrier
36
Embedded Mobile RobotsExperimental Results
Prey Acquisition with 20 cm Barrier
(B)
(C)
(D)
(A)
(F)
(G)
(H)
(E)
37
Neural based Behavior Prey Acquisition (10cm
barrier)

• Barrier (PreTectum)

• Prey (Tectum)

• Integrated (MHM)

• Heading (MHM)

• Tactile

Visual Fields
38
Neural based Behavior Prey Acquisition (20cm
barrier before bumping)

• Barrier (PreTectum)

• Prey (Tectum)

• Integrated (MHM)

• Heading (MHM)

• Tactile

Visual Fields
39
Neural based Behavior Prey Acquisition (20cm
barrier after bumping)

• Barrier (PreTectum)

• Prey (Tectum)

• Integrated (MHM)

• Heading (MHM)

• Tactile

Visual Fields
40
Neural based Behavior Prey Acquisition (20cm
barrier after learning)

• Barrier (PreTectum)

• Prey (Tectum)

• Integrated (MHM)

• Heading (MHM)

• Tactile

Visual Fields
41
Future Work

• Complete Internet based System
• Develop middleware adaptation capabilities
• Build smaller robotic systems
• Extend to multiple robot tasks
• Extend vision system to true moving forms
• Extend biological models

42
Video
43
Bonus Section
44
Research Cycle
Formal Models
Data, Hypotheses
Brain Theory
Neuroscience
Robotics
(Experiments)
(Modeling)
New Ideas
New Hypotheses
(Results from
Experiments with
Physical Devices)
New Hypothesis
Gaps in Knowledge

45
Neural Maps
T - Temporal, D - Dorsal, N - Nasal, V -
Ventral O - Optic Tectum, B - Nucleus of
Belonci C - Lateral Geniculate Nucleus, P -
Thalamic Pretectal Neuropil X - Basal Optic
Root Scalia and Fite 1974
46
Neuron Model

• mp - membrane potential dmp(t)/dt f(s,mp,t)
• mf - firing rate mf(t) s(mp(t))
• Leaky Integrator t dm(t)/dt -m(t) s

47
Retina-Thalamus-Tectum


Retina
TP
R4
Input
R3
R2



TP ThalamusPreTectum GL Glomerelus SN
Stellate Neurons SP Small Pear LP Large
Pear PY - Pyramidal

GL
SN
-


-
-
SP



-
LP
Synapsis





Excitation

-
Inhibition
-
PY
Output
48
Max Selector Didday 1976
49
Max Selector Model
MaxSelectorModel
MaxSelector
out
MaxSelector
s_in
uf
Stimulus
Vlayer
Ulayer
in
MaxSelector
s
_out
in
s
_
Output
vf
uf
in
u
_
in
v
_
nslModel MaxSelectorModel () extends
NslModel() private MaxSelector
maxselector(10) private MaxSelectorStimulus
stimulus(10) private MaxSelectorOutput
output() public void initSys()
system.setRunTime(10.0) system.setRunDelta(
0.1) public void makeConn()
nslConnect(stimulus.s_out,maxselector.s_in)
nslConnect(stimulus.s_out,output.s_in) nslConn
ect(maxselector.out, output.uf)
50
Max Selector Module
MaxSelector
out
Ulayer
Vlayer
in
s
_in
uf
vf
u
_in
v
_in
nslModule MaxSelector (int size) extends
NslModule() public Ulayer u1(size) public
Vlayer v1(size) public NslDinDouble1
in(size) public NslDoutDouble1
out(size) public void makeConn() nslRelabel(
in,u1.s_in) nslConnect(v1.vf,u1.v_in) nslCon
nect(u1.uf,v1.u_in) nslRelabel(u1.uf,out)
51
Ulayer and Vlayer Modules
nslModule Vlayer(int size) extends NslModule ()
public NslDinDouble1 u_in(size) public
NslDoutDouble0 vf() private NslDouble0
vp() private NslDouble0 hv() private double
tau public void initRun() vp
0 vf 0 hv0.5
tau1.0 public void simRun()
vp nslDiff(vp,tau,-vpnslSum(u_in)
hv) vf nslRamp(vp)

nslModule Ulayer(int size) extends NslModule ()
public NslDinDouble1 s_in(size) public
NslDinDouble0 v_in() public NslDoutDouble1
uf(size) private NslDouble1 up(size) private
NslDouble0 hu() private double tau public
void simRun() up 0
uf 0 hu 0.1 tau
1.0 public void simRun()
up nslDiff(up,tau, -up uf - v_in hu
s_in) uf nslStep(up,0.1,0.1.0)

52
Neuron (detailed)
53
Neural based Behavior Prey Acquisition (without
barrier)

• Visual Fields
• Predator (PreTectum)
• Prey (Tectum)
• Integrated (MHM)
• Heading (MHM)

54
Neuroscience Autonomous Biological Agents
55
Robotics Autonomous Robotic Agents