Courtney S' Sharp, Omid Shakernia and S' Shankar Sastry

1
A Vision System for Landing an Unmanned Aerial
Vehicle
Courtney S. Sharp, Omid Shakernia and S. Shankar
Sastry
Presentation by Efe Sirimoglu
2
Applications of Vision-based Control
MQ-9 Reaper
3
Goal Autonomous landing on a landing platform

• Challenges
• Hostile environments
• Ground effect
• Pitching deck
• High winds, etc
• There is already
• GPS
• INS
• Laser range finders
• Digital compass and sonar

• Why vision?
• Cheap, Passive sensor!
• Information reach!
• Observes relative motion!

4
Simulation Vision in the loop
5
Vision System Software Flow-chart
Involves systematic integration of hardware,
image processing in low-lewel, estimation and
synthesis of Real Time controllers in high-level.

6
Image Processing
Goal - Locate landing target - Extract
and label its feature points.
Process 1) Tresholding 2) Segmenting 3)
Detecting the corners 4) Labeling those
corners
Thresholding Threshold the image based on
fixed percantage between min max grey
levels.
Thresholded image
Camera view
Histogram
7
Image Processing
Segmentation Seperate the landing target
from background and return white squares.
Two-pass standard connected component labeling
algorithm on 4 connectivity.
8
Image Processing
Corner Detection
Convexity preserved For a line drawn vertically
through the interior of a convex polygon, the set
of points in the polygon with maximal distance
from each side of line contain at least two
distinct corners of that polygon.
9
Image Processing
Feature Labeling Idea For some plane P
containing points q1, q2, q3 ? R3 and ? is the
angle between vectors (q2-q1) and (q3-q1) where
?gt0 is CCW when viewing some surface side of P,
sign(?) sign(?)
q3
?
q1
q2
F
E
A
D
C
q3
B
T
q1
q2
10
Image Processing
Trying other corner detection algorithms
Some of the corners not properly detected. (No
advantage)
Advantage Will provide more accurate
results. Disadv Cost is more CPU time and not
feasible for low resolution and noisy images.
11
Feature Extraction

• Acquire Image
• Threshold Histogram
• Segmentation
• Target Detection
• Corner Detection
• Correspondence

12
Pose Estimation
Current pose
is an unknow scale
Homog. Coordinates of feature points on image
plane
Image plane
is the camera calibration matrix
is the projection matrix
Homogeneous Euclidean motion(landing pad frame
camera head frame)
Homogeous repr. of points in the world
Feature Points
Pinhole Camera
Landing target
13
Pose Estimation
Using a calibrated pinhole model for perspective
projection of the camera without loss of
generality, we set
Then,
where
yields to
Substituting the above equation in
where
Now we need to recover
is the rotation
is the translation from the landing target to the
camera head
14
Pose Estimation Linear Optimization
We may choose the inertial coordinate frame such
that
Taking
where
Since equation is linear in
we can solve linearly for
where
15
Pose Estimation Linear Optimization
Then F appers to be
for
where
for
More than 4 feature points detection, in
practice, F is full rank.
but due to the noise in corner
Apply standard SVD to compute singular value
vector
16
Pose Estimation Linear Optimization
Compute the least squares estimate of null space
of F
where
and
Last column of V gives the estimation of 9
parameters for a system of equations with 6
degrees of freedom.
is a scale
To ensure R is a rotation matrix, norm of a
column of R must be equal to 1.
gives us
Since camera is always in z direction of the
landing target,
gives us
17
My results
Snapshot from a camera mounted on a tripod.
The measured parameters are a 10 ß
-5 ? -90 Tx -50 mm Ty 100
mm Tz 1100 mm The computed results are
Error a 8.4167 ? 1.5833 ß
-4.6063 ? 0.3937 ? -90.3855 ?
0.3855 Tx -44.4043 mm ?
5.5957 mm Ty 96.2556 mm ? 3.7444 mm Tz
1179.8926 mm ? 79.8926 mm
x
y
18
My results
a -4.0896 ß 13.2417 ? 0.7939
Tx 20.0179 mm Ty -44.7538 mm Tz 1328.044
mm
a -9.3765 ß 1.7582 ? 88.6290
Tx 104.7199 mm Ty 12.5841 mm Tz
1262.6163 mm
a 25.9018 ß -2.3746 ? 1.1843
Tx -81.1413 mm Ty -24.0769 mm Tz 981.6103
mm
a 10.2972 ß -22.4489 ? 4.8908
Tx -28.9989 mm Ty -65.1032 mm Tz
1097.4525 mm
19
Vision-Based Landing of a UAV

• Motion estimation algorithms
• Linear, nonlinear
• Error 5cm translation, 4 rotation
• Real-time vision system
• Customized software
• Off-the-shelf hardware
• Vision in Control Loop
• Landing on stationary deck
• Tracking of pitching deck

• Pan/Tilt to keep features in image cetner
• Prevent features from leaving field of view
• Increased Field of View
• Increased range of motion of UAV

Nonlinear Optimization
Minimizing reprojection error
20
Vision-based Motion Estimation
Current pose
Image plane
Feature Points
Pinhole Camera
Landing target
21
Conclusions

• Contributions
• Vision-based motion estimation
• A good corner detection algorithm for a known
  image.
• Not a very efficient thresholding lack in
  different backgrounds
• What theyve done Demonstrated proof of concept
  prototype, first vision-based UAV landing
• Extensions
• Dynamic vision Filtering motion estimates
• Symmetry-based motion estimation
• Fixed-wing UAVs Vision-based landing on runways
• Modeling and prediction of ship deck motion
• Landing gear that grabs ship deck
• Unstructured environments Recognizing good
  landing spots (grassy field, roof top etc)

22
Thanks... Questions?