DARPA Organic Air Vehicle Project

1
DARPA Organic Air Vehicle Project
OAV System Concept
Hand Held Controller
Ground Station
Air Vehicle
Integrated Guidance, Navigation, Control Imaging
2
UAV 3D Navigation and Mapping

• Presented AINS C3UV Kickoff MeetingRaja
  Sengupta (lead), Zu Kim, Jitendra MalikJoao
  Sousa, Sivakumar Rathinam, Marco Zennaro

3
Scaling up to Swarms

• Algorithmic Complexity
• Caused by Constrained Resource Allocation
• Appears in the MILP formulation
• How reference..
• Control complexity
• Caused by rate of change of information,
  resources, constraints, .

4
Scaling up to SwarmsAlgorithmic Complexity

• There are two hopes
• The phase transition literature shows one may
  encounter it rarely
• Deal with complexity by genetic programming or
  simple heuristics
• Create complex emergent behaviors by simple rules

5
Scaling up to SwarmsAlgorithmic Complexity

• There are two hopes
• The phase transition literature shows one may
  encounter it rarely
• Deal with complexity by genetic programming or
  simple heuristics
• Create complex emergent behaviors by simple rules
• Deal with it by statistical or fluid mechanical
  abstraction to reduce large numbers
• Solve smaller resource allocation problems on the
  abstraction

6
Scaling up to SwarmsHeuristic Research at UCB

• Collaborative Multi-Vehicle task completion

destination
task
obstacle
obstacle
7
Scaling up to SwarmsHeuristic Research at UCB

• Collaborative Multi-Vehicle task completion

8
Scaling up to SwarmsHeuristic Research at UCB

• Collaborative Multi-Vehicle task completion

9
Scaling up to SwarmsGenetic Programming at UCB

• Swarm navigation through obstacles

10
(No Transcript)
11
Scalability

• Incremental growth of service network
• add a program or add a vehicle
• SNP integrates it with the rest when required
• Asynchronous
• Incremental growth of management overhead
• search complexity of the Broker increases
  gradually with the number of registered services
• NP-complete in the worst case sense

Phase transitions have been found for
satistfiability problems
12
Scaling up to SwarmsControl Complexity

• Caused by
• Changing structure
• Resources
• Connections

13
Scaling up to SwarmsControl Complexity

• Caused by
• Changing information
• Tasks
• Obstacles
• Teammates,

14
Scaling up to SwarmsControl Complexity

• Our Innovations
• Service Networks
• Distributed Publish Subscribe
• in multi-vehicle control
• Aim is to control a system of changing dimension
• Keep operational properties invariant even as
  system dimension changes
• Examples of operational properties
• Agents may leave the system but information does
  not (remain invariant)
• Each user task will be serviced

15
Scalable Information ManagementSearching for
Unknown Threats
16
Scalable Information ManagementPublish and
Subscribe
Team Level

Operation decomposer
Information state
Resource allocation/scheduling
Target Distribution Map
Operation monitor
Risk Map
Dispatcher
Formation Navigation
17
Scalable Information ManagementTarget Map and
Risk Map

• Target distribution map
• P(A, N, t) probability of N targets of type t in
  area A
• Target distribution update
• Fuses measurements from different kinds of
  sensors (SAR and EO)
• Bayesian update
• Risk map computation
• Integral of threat model with respect to the
  measure P(A, N, t)
• Generates the value function for navigation

18
Scalable Information ManagementSearching for
Unknown Threats
19
Scalable Information ManagementDistributing the
Publisher Service

• Geographic Data Management Network

20
Scalable Information ManagementDistributing the
Publisher Service
User
21
Scalable Information ManagementDistributing the
Publisher Service
Movie of our Implementation 4 servers on 4
laptops over wireless
22
Controlling Systems with Changing Dimension

• Service Networking Based on Jini Middleware

23
What is Jini?

• Service and User arrive at different times
• Easy repair
• Proxy implementation invisible to user
• Easy system integration

Registration
Service
Lookup manager
Multicast Network
RMI
Proxy
ServiceItem
Client
24
Systems with Changing DimensionService Network
for Multi-Vehicle Search
Lookup Manager
Mission Control
Search Software
Host
Vehicle
Txn Manager
Search Server
Publisher
25
The SN Synthesis Theorem

• Theorem Let the set of concrete values be
  finite. Then there exists a synthesis procedure
  that is sound, complete, and decidable.

26
Systems with Changing DimensionMovie of Pursuit
Evasion

• Vehicle simulated on laptops over wireless LAN

27
Scalable Information ManagementDistributed
Estimation over the GDMN
Theorem 2 If observations stop coming in at time
m and reach all the map-builders by time n,
then If communication between servers is faster
than the target then the Distributed computations
converge to the centralized ones.
Movie of our Implementation 4 servers on 4
laptops over wireless
28
Summary

• Our work in the AINS program has attacked the
  problem of collaborative vehicle control from two
  ends
• We first attacked the large-scale problem in
  simulation
• Then the small scale problem with real vehicles
• Future
• Attack the large scale multi-vehicle
  collaborative control problem with real vehicles

29
Large Numbers of Clients Generating Tasks
30
Research Objectives

• Real time 3D mapping of complex urban
  environments
• Coordinated team control
• Mapping, Surveillance, and Tracking Missions

31
Performance Requirements

• Map the 3-D region
• in minimum time
• to a specified precision, with information
  completeness, and safety of UAV
• Information completeness
• Acquire all the information that could possibly
  be obtained
• Safety Fly with controlled risk of damage or
  destruction

32
Technical Challenges..

• Mapping in the face of imprecise sensor location
  information
• 3D map construction based on
• Multiple images from single camera
• Multiple images from multiple cameras
• Path Planning with incomplete maps in a 3D
  environment
• Real time Map should evolve fast enough for
  control, control should evolve fast enough for
  mapping
• Should ensure information completeness
• UAV control for safe path execution in the face
  of non-holonomy

33
UAVs and Machine Vision

• Zu Kim
• Computer Science Division / California PATH
• University of California, Berkeley

34
Machine Vision Challenges

• Computation
• Input 320x240 pixels (minimum), 1030 frames per
  second
• Cannot apply complex algorithms to real-time
  applications
• Human vision uses massive parallel computation
• Moving Camera
• All the pixels move
• Not many real-time algorithms dealing with moving
  cameras
• Limited Bandwidth

35
Machine Vision Research with UAVs

• Mapping (mostly off-line)
• Aerial map generation (2-D and 3-D)
• Man-made feature detection and description
• 3-D detailed map generation (from ground-view)
• Camera calibration / accurate ego-motion
  estimation
• Target Detection / Recognition / Tracking
• Navigation (on-board processing)
• Stereo-based range map (depth map) generation
• Real-time ego-motion estimation / moving object
  detection

36
2-D/3-D Aerial Map Building

• Stereo range map building (off-line)
• Commercial s/w available (eg. SocetSet fro BAE
  Systems, Inc.)
• 2-D map building
• Real-time image mosaic algorithm availableKang
  Medioni 99,

37
Man-made ObjectDetection and Description

• Building Road Detection and Description

Price 99
Kim 2000
38
3-D Map Building fromGround-View Frueh Zakhor
39
3-D Map Building fromGround-View Frueh Zakhor
40
Interactive 3-D Map Building

• Façade Devebec Malik 96

41
Target Detection and Tracking

• Challenges
• Target shapes and orientation vary
• Illumination condition varies
• Non-uniform background
• Most algorithms are based on pixel comparison,
  hence require significant computation

42
Target Detection and Tracking
43
Navigation Stereo-based Range Map Generation

• Needs two forward-looking cameras
• Real-time algorithms available for PCs
• Intel OpenCV library
• Low quality
• Relatively simple but massive operation
• Specialized h/w (parallel processor) can improve
  the quality

44
Navigation Real-time Ego-motion Estimation and
Moving Object Detection

• Challenges
• Accurate ego-motion estimation require accurate
  optical flows (that require massive computation)
  and multi-frame optimization
• Intrinsic ambiguity between speed and distance
  (require sensor fusion)

45
Machine Vision Research Focus

• 3-D map generation (off-line)
• Target detection and recognition (on-line and
  off-line)
• Navigation
• Real-time ego-motion estimation (use multiple
  cameras)
• Real-time moving object detection and tracking

46
Planned Approach Hierarchical Design to Beat
Complexity

• Mapping Manager
• Team control and operator interface
• Adaptive 3-D Path Planner
• Handle
• Optimality
• Information completeness
• Map Information Structure
• Object independent grid decomposition
• Path computation
• Fast approximations for Traveling Salesman
  computations
• Adaptive space filling

47
Planned Approach Hierarchical Design to Beat
Complexity

• Safe UAV flight controller
• Handles safety
• Refines the nominal 3D path into a safe but
  informationally adequate path
• Continuous weave of half ellipses
• Non-holonomic constraints
• Stick model (Dubins, Boissonnat)
• Extend to aggressive maneuvers
• 3D Map Builder
• Image mosaicing
• Ground robot 3D SLAM algorithms based on EM and
  Maximum likelihood for (Thrun et al)

48
Past Work Threat Search Simulation in the 2D
Setting
Launcher On Complete Architecture On
49
UAV Sensor and Flight ControlChallenge

• Use imaging sensor to see stationary launchers
• If UAV flies straight with sensors on at its
  minimum airspeed or more it gets shot by the
  launchers
• Imaging and object interpretation times are large
  (seconds)

Launcher On UAV Sensor On
50
Past Work Threat Search Simulation in the 2D
Setting
Launcher On Complete Architecture On
51

UAV Safe Controller Design Elliptical flight
path

• Algorithm
• Notations ? - imaging box side length, RL -
  range of the launcher, (z,w) major and minor
  axis of the ellipse

52
Coordinated Team Control for Autonomous Mapping,
Surveillance, and Tracking Missions

• Design of distributed control protocol for
  adaptive relocation of sensor UAVs for optimal
  sensing task performance
• Estimate the geographical distribution of demand
  for sensors
• Complementary task division amongst heterogeneous
  sensors
• Proxy design for individual task monitoring

53
Schedule

• Year 1
• - Off-line mapping in a complex 3D environment
  with UGV image data feeds
• Year 2
• - 2-UAV image fusion for navigation and mapping
  in a 3D environment
• Year 3
• - Collaborative mapping with 3 UAVs with dynamic
  team control and task decomposition
• Year 4
• - Collaborative mapping to navigate reconfiguring
  formations
• Year 5
• - Integration with project demonstration to
  show integrated execution
• of mapping, surveillance, or tracking
  missions

54
Scalable Information ManagementDistributing the
Publisher Service

• Geographic Data Management Network

User