Project Review UAVUGV System ARO VisionBased Traffic Monitoring Hills' County

1
Project ReviewUAV-UGV System (ARO)Vision-Based
Traffic Monitoring (Hills. County)

• K. Valavanis, M. Labrador, W. Moreno. P.-S. Lin
• A. Weitzenfeld, N. Tsourveloudis

2

• Other USF Faculty ARL
• Lee Stefanakos Dr. Stephen (Drew) Wilkerson
• Paris Wiley Dr. MarryAnne Fields
• Graduate Students
• W. Alvis L. Barnes C. Castillo M.
  Castillo-Effen
• K. Dalamagkidis D. Ernst R. Garcia S.
  Ioannou
• D. Jabba M. Kontitsis I. Moura S. Murthy
• A. Puri I. Raptis N. Saigal A. Tsalatsanis
• P. Wightman
• Undergraduate Students
• M. Wallace M. Michael

3
Meeting Objectives

• Unmanned Systems Lab Infrastructure
• Research Capabilities
• Expertise
• Project Review
• UAV-UGV System (ARO / ARL / US SOCOM / CTC)
• Traffic Monitoring (Hillsborough County)
• Results and Demos
• and
• Convince sponsors that we can do the job and
  deliver final results/products in short- and
  long-term continue funding.

4

• Main Objective
• Development, testing and use of fully autonomous
  (small, miniature) unmanned aerial/ground
  vehicle systems in military and civilian
  applications.
• From conceptual design, to modeling, validation,
  integration, testing and implementation
• Design, model, build, validate, test, implement
  fully functional and autonomous unmanned
  systems, easily reconfigurable, with modular
  components, using very cost effective and
  reliable technology.

Ready to use system
5

• Our approach and how different we are
• Component plug in plug out concept
• General / generic designs for both aerial and
  ground vehicles
• Designs fit small unmanned vehicles regardless of
  type
• Designs suitable for systems with very strict
  payload limitations
• Miniature configuration systems
• Energy efficient designs with enhanced endurance
  and range
• Very cost effective technology - at least one
  order of magnitude cheaper compared to
  competition and at least twice as powerful!
• All systems built and integrated in house!

6
Partners / Sponsors
US SOCOM

• ARL, Aberdeen MD
• Georgia Tech
• Hillsborough County Traffic Department
• Universita Politecnica Delle Marche, Ancona,
  Italy
• Instituto Tecnologico Autonomo de Mexico
• Technical University of Crete, Greece
• QTSI

7
Find out about who we are

• http//www.cse.usf.edu ? Research ?
• Unmanned Systems Lab
• http//www.cse.usf.edu/USL/uslindex.htm

8

• Our strengths
• Control Systems
• Real-time control systems, computer-controlled
  systems
• Controller design and synthesis, testing,
  implementation
• PD/PID LQG / LQR Fuzzy Logic
• Neuro-Fuzzy Genetic
• Guidance and Navigation
• Sensors and sensor fusion
• Integrated control and diagnostics
• Swarm formation control
• Machine / Robot Vision
• Hardware and Software Design
• Networks and Communications
• Energy Efficient Systems
• Computational Intelligence

..and most important of all
Ability to build and test completely operational
and integrated systems in house!
9

• Dedicated resources / assets
• Unmanned Ground Vehicles
• 1 ATRV-Jr, customized with GPS, IMU, Sick
  Laser, Stereo Vision System
• 5 RC-trucks, custom built with GPS, IMU, Stereo
  Vision System
• VTOL vehicles
• 3 RAPTORs, type 30, (70 converted to a ) 90,
  90SE
• 2 BERGEN Twin built by Rotomotion
• 1 YAMAHA Rmax (1)
• 2 Electric Helicopters
• Maxi-Joker
• Trex (miniature)
• Fixed wing UAVs
• 3 fixed wing UAVs, donated by ARL

10
Equipment

• Raptor 90 SE/Generation I Controller Box
• Needed
• Safety Switch for Autonomous Operation
• 5 Hz GPS
• IMU
• Stabilized Camera Platform
• Higher Performance Computer System
• Better Vision Capabilities
• Cleaner, More Efficient Operation
• Removable, Easy to Reconfigure Boot Device

11
Equipment

• Raptor 90 SE/Generation I Controller Box
  (continued)

12
Equipment

• ARL Lynchbot/Oregon State Autopilot
• Needed
• Higher Processing Power for Advanced
  Functionality
• Faster GPS
• Better Interoperability Between Components
• Less Weight
• Fewer Custom Fabricated Parts
• Cleaner Data
• Better Ground Clearance
• 802.11 Comms

13
Equipment

• Emaxx/Generation I Controller Box
• Needed
• IMU
• Faster Processing
• Safety Switch
• Better Ground Clearance
• Better Vision
• Pan/Tilt Unit
• 5 Hz GPS

14
Updated Equipment

• Generation II Controller Box
• Includes
• 2 Ghz Intel Pentium M Processor
• 2 GB Memory
• 5 Hz Superstar II GPS
• Microstrain 3DM-GX1 IMU
• Microbotics Servo Controller with Safety Switch
• Pico Power Supply Unit
• Four Port Video Capture Card
• USB Boot

15
Updated Equipment

• Maxi Joker 2/Generation II Controller Box
• Includes
• Fully Electric Helicopter
• Quiet Operation
• No Mess
• Easy and Fast Set-up
• Custom Shock Mount Skids
• Shock Mounted Pan/Tilt
• Double Shock Mounted IMU
• Sony Block High Resolution Camera with Zoom
  Capabilities
• Separate Power for Safety Switch
• Full Autonomous Capabilities
• Wireless Video Transmission

16
Updated Equipment

• Maxi Joker 2/Generation II Controller Box
  (continued)

17
Updated Equipment

• Maxi Joker 2/Generation II Controller Box
  (continued)

18
Updated Equipment

• Maxi Joker 2/Generation II Controller Box
  (continued)

19
Updated Equipment

• Emaxx UGVs/Generation II Controller Box
• Includes
• Fully Electric Ground Vehicles
• Special Oil Filled Shocks
• Upgraded Springs
• Brushless Motor
• Two Sony Block High Resolution Cameras with Zoom
  Capabilities
• Custom Pan/Tilt Units
• Full Autonomous Capabilities
• Wireless Video Transmission

20
(No Transcript)
21
Updated Equipment

• Emaxx UGVs/Generation II Controller Box
  (continued)

22
Updated Equipment

• Emaxx UGVs/Generation II Controller Box
  (continued)

23
For Preliminary Traffic Data

• Camera Poles
• Necessary Low Cost System for
• Testing Vision Algorithms
• Pan/Tilt Capabilities
• Fully Adjustable to 24ft
• Provides Movement to Simulate
• Helicopter Flight
• Does Not Need Campus Wide
• Approval for Traffic Monitoring

24
Equipment (more)

• Use of Bergen Industrial Twin to Provide High
  Endurance Flights with Greater Stability

25
VTOL Model Comparison

• Maxi Joker 2 Electric
• Quiet
• Easy Setup
• No Mess Operation
• Less Helicopter Vibration
• Short Endurance (20 minutes)
• More Responsive
• Payload (up to 10 lbs)
• Cost 14,000.00

• Bergen Industrial Twin
• Economical to Operate
• Withstand Higher Winds
• High Endurance (up to 1.5 hours with Optional
  Tanks)
• Large Payload (Up to 20 lbs)
• Cost 20,000.00

26
(No Transcript)
27
One of three fixed wing, donated by ARL
28
(No Transcript)
29
TREX Micro Electric
30
(No Transcript)
31
(No Transcript)
32
The Maverick
33
Compare size and cost (THIS IS OUR DIFFERENCE!!!!)
34
Computer Controller Comparison
35
Application Domains

• Reconnaissance and surveillance
• Mapping
• Demining
• Terrain identification
• Troop Monitoring
• Threat identification
• Convoy support / scouting
• Traffic monitoring and management
• Inspection
• Border patrol
• Harbor security
• Search and Rescue.
• Etc.

36
Project Review

• Vision-Based traffic Monitoring
• UAV-UGV Cooperation

37
Common Concept Objective

• UGV / VTOL Communication

Representation of the communication network
between a small UGV/VTOL fleet and two manned
command centers.
38
Common Concept Objective
39

• Border Patrol patrol.mpg
• Latching Mechanism latch-pdemo.mpeg
• The concept may be extended to include small
  sea-surface vehicles that launch / recover VTOLs
  in near beach areas for inspection, surveillance,
  reconnaissance, demining, etc.

40
Idea of launching and recovering on the move
41
Traffic Monitoring
Traffic monitoring Framework for incorporating
real-time data in simulation models
Sample traffic visual data DOT_Heli_Traffic_11_29
_06.wmv
42
The overall design process
Stabilization
Feature Extraction
Images from camera
Motion Extraction
Feature Grouping
Environment Setup Selection
Traffic Statistics
Vehicle Tracking
43
Image Processing

• Selected Modules
• Noise reduction
• Low pass filter
• Motion Segmentation
• Temporal differencing
• Vehicle localization

44
Image Processing

• Selected Modules
• Background extraction
• Moving Time averaged accumulator

45
Image Processing

• Selected Modules
• Road extraction
• Selection of low color saturation pixels

road is black
46
Output Video (helicopter camera)

• Blue and Green boxes denote counting zones
• Red rectangles flash momentarily when the
  program counts the car
• Video part1a.m1v

47
Output counts

• Output follows format

Time is in frames
It can be converted to real time using the
systems clock.
48
Next Steps

• Collect real-time traffic video data over an
  intersection, road segment, highway segment,
  specific traffic network using single / multiple
  unmanned helicopters.
• Store data on-board for evaluation, analysis,
  etc.
• Transmit data to the traffic control centers
  (ground control stations) for on-the-spot /
  immediate decision making when necessary, as well
  as traffic signal timing modifications,
  re-routing, emergency response, etc.
• Convert real-time collected data to statistical
  profiles, to be used as inputs to traffic
  simulation models, aiming at improving their
  accuracy, predictability, parameter calibration,
  etc..

49
Goals

• Real-time, dynamic
• traffic monitoring
• traffic network management
• optimal traffic signal management
• optimized traffic flow and rerouting
• minimized emergency response time
• improved resource/asset allocation in emergencies
• Improve
• traffic simulation models
• model accuracy
• calibration
• predictability

50
The Proposed Solution Considers

• Real-time eye-in-the-sky detailed video data
• Every traffic network (segment of traffic
  network) has its unique characteristics (for
  example downtown peak-hours differ from campus
  peak-hours)
• Ability to update simulation model in real-time
  (especially important in case of incidents or
  events)
• Performance measures can be easily observed
• Ability to predict traffic patterns using
  real-time data.

51
Statistics - Parameters

• Speed
• Flow
• Occupancy
• Density (Spatial-temporal)
• Turning Movement
• Queue Length
• Delay
• Origin-Destination
• Efficiency Parameters (LOS, VMT)

52

• Car-following behavior
• The following car maintains acceptable gap from
  the leading car
• For total length of link d, the equation
  becomes
• Thus, approximate capacity of link is
• Occupancy can be derived as

53
Speed

• Mean speed can be calculated by observing the
  travel time of individual vehicles through the
  link
• Flow is given by number of vehicles passing
  through a certain point in network in a given
  time period
• (L is number of lanes.)

54
Density

• Spatial

• Temporal

• (Pseudo) Spatial-Temporal

55
Turning Movement/ O-D

• Assign virtual detectors on start and end of
  links.
• Tag vehicle id with time of arrival and position
  at each VD each passes.
• Maintain a link list to record path of each
  individual vehicle.
• Vehicle Path VD1,VD2, , VDn

56

• Delay
• VMT
• VMTFlow x Distance

57
Efficiency Parameters

• LOS

58
Synchro Model

• Campus network simulated
• in Synchro
• accurate geometry
• speed limit
• storage lanes included.

59
(No Transcript)
60
SimTraffic Report
61
Another (not so distant) Goal

• Equip each car with a smart device that
  informs, alerts and warns the driver (after
  she/he has registered or defined the initial
  origin-destination route) about the state of
  the specific network segment she/is is driving
  through, suggesting alternative routes in case of
  emergencies or other unforeseen situations.
• Note that the technology is out there.

62

• Collaborative Autonomous Unmanned Aerial
  Ground Vehicle Systems for Field Operations

63
Objectives

• UAV UGV integrated system with autonomous /
  semi-autonomous capabilities and characteristics
  
• UGV loose formation control even in presence of
  failures
• Waypoint
• Follow the leader
• Random (survey an area)
• Several formations (circle, ellipse, line,
  flocking, etc)
• UGV navigation based on VTOL received commands
• Follow the leader (leader is helicopter)
• Go to navigation where location is dictated by
  the VTOL
• Formation reconfiguration based on communication
  constraints
• Each UGV is a repeater node
• Each UAV is a repeater node
• UGV-UAV comms
• UAV-UGV formation (3-D) even in presence of
  failures

64
Swarm Formation Control Objectives

• Derive a simple method for robot swarm formation
  control as a whole, with characteristics
• Scalability, applicable to different size swarms
• Computationally efficient
• Supporting different (not fixed) formations
• Supporting centralized and decentralized
  formation control
• Homogeneous and heterogeneous swarms
• Expandable to aerial ground vehicle swarms
• Basis
• Potential fields for obstacle avoidance, swarm
  orientation, and swarm movement

65
Specific ExampleAccompanying a Convoy

• Consider the case of a swarm of robots that
  needs to accompany a convoy of vehicles
• Only UGVs, overlook the shape of the terrain,
  this is the 2-D case!

66
Describing The Convoy

• Generally speaking enclose the convoy in some
  geometric scheme, define loosely dimensions,
  direction of travel and center of mass.

2A
2B
Direction of Travel q
67
Enclosing Convoy in Concentric Ellipses
Direction of Travel q Center (cx, cy) Major
Axis A Minor Axis B Axis Ratio g B/A
R(x, y) (x-cx)2 g(y cy)2
Design a field to attract swarm members to the
ring R DRi lt R(x, y) lt R DR0
68
Problem Formulation

• Design a vector field that attracts particles to
  the ellipse within the bands
• R ?Ri lt R(x, y) lt R ? Ro
• Where R is

Where ? is the ratio of the minor and major axes.
69
Final Vector Field
Color indicates the original vector field Red
attraction Green repulsion Blue orbiting
70
Simulations Real-Time Results

• In order to test the theory, simulation results
  are presented with particles and modeled
  RC-trucks. RC-trucks are considered heterogeneous
  with different modeling parameters.
• Multiple formations are shown with 3 and 10
  particles / RC-trucks .
• Real-time results are presented with 4 RC-cars
  showing ellipse and wedge formations including
  failures.

71
Tested Functionality

• Stationary Obstacle Avoidance
• Introduce additional vector fields that have
  limited influence
• Avoiding collisions with other swarm members.
  Implement one of two approaches (so far)
• Treat each swarm member as a quasi stationary
  obstacle
• Modify the speed of a swarm member based on
  near-by swarm members
• Following waypoints
• Model each waypoint as an attractor
• Tolerance to errors and uncertainties
• Realistic Vehicle Dynamics.

72
Information Requirements

• Each swarm member has information pertaining to
• Own Location
• Convoy Properties (xc, yc)
• Location of Nearby Swarm Members
• Location of Stationary Obstacles

Modifying formation In addition, parameters can
be modified to manipulate the vector field so
that robots can form multiple different
formations (e.g. line, different ellipses,
inverted V, etc..)
73
Robots Line Formation
In order to force the robots into a line
formation a must be very small in comparison to ß
so the surface of the ellipse function from the
main swarm function is long and skinny. All ten
robots were slightly different but used an
identical vector generation code.
Line formation with 10 robots at (a) t1. (b)
t25. (c) t50. (d) t100
74
Robots Ellipse Formation
In this case, ßgt?, so the formation follows a
narrower ellipse configuration along the path.
Ellipse formation with 10 robots at (a) t1.
(b) t50. (c) t100. (d) t200
75
10 Particles Circle/Ellipse
Circle
Ellipse
76
Real-time Results

• Custom-built (in house) 4 RC-cars equipped with
  computer control system utilizing GPS and IMU
  sensors, stereo vision and encoders. For this set
  of experiments only GPS and IMU have been used.

77
Software

• All software for the ground robots is written in
  C.
• All robots run simple TCP communication code to
  share data with the other robots.
• Each robot runs a server to send its own data
  out.
• Each robot also has to run a client for the other
  n-1 robots in the swarm enabling them to collect
  position information from the other swarm
  members.
• Since there are no obstacle avoidance sensors on
  the robots, the robots run in an obstacle free
  environment but avoid each other via their GPS
  coordinates.

78
Real-Time Results
Four robots-1 4bot-clip1.wmv Four robots-2
4bot-clip2.wmv Three robots-1 3bots-clip1.wmv Thr
ee robots-2 3bot-clip2.wmv Swarm with 3
helicopter swar-heli-3bots.wmv Follow the
leader Leader-Follow.wmv Swarm of three
following the helicopter swar-heli-3bots.wmv Swar
m with failure of one robot-1 robot-failure-SF-12
-2.wmv Swarm with failure of one robot-2
robot-failure-DF2-12-2.wmv Swarm with failure of
one robot-3 robot-failure-DF-12-2.wmv
79
Helicopter Control

• Hover Hover V2(2).wmv
• Waypoint Waypoint V2.wmv
• With failure-1 560 dps CCW Failure.wmv
• With failure-2 560 dps CW Failure(2).wmv
• With failure-3 Tail Failure V2(2).wmv

80
Robot ID from VTOL
Thresholding Hue, Saturation and Illumination
components
RGB to HSI conversion
Image acquisition
Extract background
Extract Robots
81
Robot ID from VTOL

• Green rectangles are areas for consideration
• The identification is complete only when another
  box with the corresponding color is placed

82
Videos
Tracking combined indoors / outdoors all
tracking videos.mov Indoors-1movie_indoors_blue.m
ov Indoors-2movie_indoors_red.mov Outdoors-1
movie_outdoor_blue.mov Outdoors-2
movie_outdoor_red1.mov Outdoors-3
movie_outdoor_red2.mov Cave Simulation Cave
simulation_2.wmv
83
Details About

• Communication Issues
• Auto pilot design
• WSN and localization
• Tracking