Software Architecture: Issues and Illustrations

1
Software Architecture Issues and Illustrations

• Larry Matthies
• Supervisor, Machine Vision Group
• Jet Propulsion Laboratory

2
Notional Software Architecture
Global map
Mode manager
Diagnostic display
Global map update
Global route planner
(Potentially everything)
Obstacle list
Steering votes
Obstacle avoidance planner
Steering votes
Obstacle pruning
Sensor map generation
Local map update
Obstacle list
Steering votes
Range image
Geometric obstacle detection
Vehicle state
Labelled image
Arbiter
Road finder
Chosen trajectory
Range image
IR
Range data
Stereo matching
Velocity planner
Image classification
Stereo
Ladar
RGB
RGB IR
Stereo pairs
State estimator
Motion compensation
Actuator control
Preprocessing
Exposure control feedback
GPS
IMU
Etc
Ladar
FLIR
Stereo
3
Stereo, RGB Classification, Obstacle Detection,
and Obstacle Pruning
Original image
Range map
Classified obstacles
Terrain classification
4
Fort Knox Terrain Classification Example
Left color CCD image
Terrain Classification image
Dry Vegetation
Green vegetation
Unknown
Rock
Soil
5
Passive Perception Legacy System Ft Knox
Elevation/Obstacle Map Example
Passive Subsys
0.2x0.2x0.2m cell resolution
Negative obstacle
Positive obstacle
25 m
Frame clst026-005
6
Passive Perception Legacy System Ft Knox
Elevation/Classification Map Example
Passive Subsys
0.2x0.2x0.2m cell resolution
Green vegetation
Dry Vegetation
Soil
25 m
Rock
Unknown
Frame clst026-005
7
Passive Perception Legacy System Map Format
(continued)
4 bytes/cell
Green Vegetation Dry Vegetation Soil Rock Unknown
Stereo vision map
Positive Obstacle Negative Obstacle Cover
8
Notional Software Architecture
Global map
Mode manager
Diagnostic display
Global map update
Global route planner
(Potentially everything)
Obstacle list
Steering votes
Obstacle avoidance planner
Steering votes
Obstacle pruning
Sensor map generation
Local map update
Obstacle list
Steering votes
Range image
Geometric obstacle detection
Vehicle state
Labelled image
Arbiter
Road finder
Chosen trajectory
Range image
IR
Range data
Stereo matching
Velocity planner
Image classification
Stereo
Ladar
RGB
RGB IR
Stereo pairs
State estimator
Motion compensation
Actuator control
Preprocessing
Exposure control feedback
GPS
IMU
Etc
Ladar
FLIR
Stereo
9
Passive Perception Legacy System Map Format

• Size/Throughput
• 50m x 50m, 125 cells x 125 cells
• 26.7 KB max (at 4 bytes/cell) w/o white spaces
• 133.4 KB/s max at 5Hz
• Orientation
• North oriented
• Location
• origin determined by sensor pointing
• Resolution
• 0.4 x 0.4 x 0.2m
• Range (45º HFOV sensor)
• 46-54m on periphery
• 50-60m straight ahead
• Elevation
• -20.3 m to 20.3 m

North
East
50m
50m

• Pantilt is always directly above the circle.
• Pan pointing angle passes through the map center

map origin
10
Notional Software Architecture
Global map
Mode manager
Diagnostic display
Global map update
Global route planner
(Potentially everything)
Obstacle list
Steering votes
Obstacle avoidance planner
Steering votes
Obstacle pruning
Sensor map generation
Local map update
Obstacle list
Steering votes
Range image
Geometric obstacle detection
Vehicle state
Labelled image
Arbiter
Road finder
Chosen trajectory
Range image
IR
Range data
Stereo matching
Velocity planner
Image classification
Stereo
Ladar
RGB
RGB IR
Stereo pairs
State estimator
Motion compensation
Actuator control
Preprocessing
Exposure control feedback
GPS
IMU
Etc
Ladar
FLIR
Stereo
11
TMR Behaviors Obstacle Detection Approach
Left camera image
Right camera image
Obstacle
Free space
Vehicle center
No data
Low-level stereo processing
2D Occupancy grid
Disparity image
Obstacle image
12
TMR Behaviors ODOA Obstacle Regions
2.5 m
hand rail (unclimbable)
outside the FOV
cement curb (climbable)
2.5 m
Occupancy grid

• No Obstacle 0-9 cm
• Climbable obstacle 9-20 cm
• Unclimbable obstacle 20 cm

Left camera image
13
TMR Behaviors ODOA Candidate Paths
To be used in next revision where the sensor map
is merged into the world map
Current Implementation
2.5 m x 2.5 m
98.7º FOV
v
11 candidate arcs
23 candidate arcs conform to previously verified
free space

• There are two additional arcs for turn-in-place.
• Arcs are evaluated out 1.5 meters for
  intersection with grown obstacles.
• Arcs that intersect unclimbable obstacles w/in
  1.0 m of the vehicle are eliminated.
• Arcs that intersect unclimbable obstacles beyond
  1.0 m are penalized.

14
TMR Behaviors ODOA Obstacle Growing
outside the FOV
Configuration Space
Occupancy grid
cardboard box
Obstacle Growing
cement wall
Stereo image pair
Some candidate paths
15
TMR Behaviors ODOA Map Merging
FOV Polygon
World map
Sensor map
5.0 m x 5.0 m
Display map
3.5 m x 3.5 m

• FOV polygon merged into world
• map
• Scrolling map
• North oriented
• Global coordinates
• Morphin planning performed here

• Vehicle centered map
• North oriented
• Global coordinates
• Obstacle growing performed here

3.5 m x 3.5 m

• Vehicle centered submap

16
TMR Behaviors Obstacle Avoidance Results
Frame 01
Frame 05
Frame 09
Frame 13
Left Images
Sensor maps (ungrown obstacles)
Display maps (grown obstacles)
17
Notional Software Architecture
Global map
Mode manager
Diagnostic display
Global map update
Global route planner
(Potentially everything)
Obstacle list
Steering votes
Obstacle avoidance planner
Steering votes
Obstacle pruning
Sensor map generation
Local map update
Obstacle list
Steering votes
Range image
Geometric obstacle detection
Vehicle state
Labelled image
Arbiter
Road finder
Chosen trajectory
Range image
IR
Range data
Stereo matching
Velocity planner
Image classification
Stereo
Ladar
RGB
RGB IR
Stereo pairs
State estimator
Motion compensation
Actuator control
Preprocessing
Exposure control feedback
GPS
IMU
Etc
Ladar
FLIR
Stereo
18
Autonomous NavigationPath Following

• Leader transmits safe path segment to follower
  (Set of subgoals separated by 20cm)
• Update a carrot goal in real-time estimate track
  speeds to run through the carrot
• Can be used for path backtracking
• Useful to recover from a comm dropout
• Position estimation used for robot position
  robust through GPS dropouts

Leader
Path segment
carrot
L
normal
Follower
19
Path Following Control Algorithm Combined Pure
Pursuit and PID Control

• Pure Pursuit Controller
• PID Controller
• Combination of controllers are used

Normal approach to path can be unstable
Accurate around curves
Inherent error around curves
Stable when approaching paths
20
Path Following Architecture
3-Axis Accels
High-level Processor (Pentium - VxWorks)
3-Axis Gyros
Robot Pose Estimator with Kalman Filter
GPS
Low-level Motor Control Processor (M 68332)
Path Following Recording Behavior
Robot Motion Arbiter
Digital Compass
Object Detection Avoidance Behavior
Stereo Cameras
Ladar
21
Path Following with ODOA
22
Path Following with ODOA
23
Notional Software Architecture
Global map
Mode manager
Diagnostic display
Global map update
Global route planner
(Potentially everything)
Obstacle list
Steering votes
Obstacle avoidance planner
Steering votes
Obstacle pruning
Sensor map generation
Local map update
Obstacle list
Steering votes
Range image
Geometric obstacle detection
Vehicle state
Labelled image
Arbiter
Road finder
Chosen trajectory
Range image
IR
Range data
Stereo matching
Velocity planner
Image classification
Stereo
Ladar
RGB
RGB IR
Stereo pairs
State estimator
Motion compensation
Actuator control
Preprocessing
Exposure control feedback
GPS
IMU
Etc
Ladar
FLIR
Stereo
24
Dynamics-based Velocity Controlfor Obstacle
Negotiation
25
Compressible Terrain Modeling
QUARTER CAR MODEL

• Create terrain compliance model
• Vehicle modeled by a dual spring-mass system
  (vehicle body and tire)
• Fast ODE solver for given horizontal velocity
• Terrain modeled by spring and damper system
• Rigid terrain (stone) high stiffness
• Green vegetation, grass low stiffness
• Twigs/branches partially compressible
• Predict vertical accelerations along a path for
  different horizontal velocity
• Determine maximum velocity such that the maximum
  vertical acceleration is below a given threshold

26
Vehicle/Terrain Modeling
VEHICLE BODY
KV, Cv
Hv
TIRE KT, CT
HT
TERRAIN KR, CR
EG(x)
For faster computationCouple tire and terrain
spring constants1/Keff 1/KT 1/KResulting
in variable spring constant and damper
coefficient (depending on terrain material)

• Couple tire and terrain models 1/Keff 1/KT
  1/KR
• d2Hv/dt2 -Kv/Mv ( Hv - HT - LV) - Cv/Mv (
  dHv/dt - dHT/dt) - g
• d2HT/dt2 -Kv/MT ( Hv - HT - LV) CV/MT (
  dHv/dt - dHT/dt) - Keff/MT (HT - EG(x) - LT) -
  Ceff/MT ( dHT/dt - dEG(x) /dt ) -g

27
Notional Software Architecture
Global map
Mode manager
Diagnostic display
Global map update
Global route planner
(Potentially everything)
Obstacle list
Steering votes
Obstacle avoidance planner
Steering votes
Obstacle pruning
Sensor map generation
Local map update
Obstacle list
Steering votes
Range image
Geometric obstacle detection
Vehicle state
Labelled image
Arbiter
Road finder
Chosen trajectory
Range image
IR
Range data
Stereo matching
Velocity planner
Image classification
Stereo
Ladar
RGB
RGB IR
Stereo pairs
State estimator
Motion compensation
Actuator control
Preprocessing
Exposure control feedback
GPS
IMU
Etc
Ladar
FLIR
Stereo
28
Example Diagnostic Display