Robust Modeling of Datasets with unbounded Outliers

1
Robust Modeling of Datasets with unbounded
Outliers

• Derek King
• Vladimeros Vladimerou

ECE586YM Spr2006
2
Proposal Review
3
Proposal Review
4
Proposal Review
5

• Hough Transform
• developed in the 1960s by Paul V.C Hough to find
  lines in reactor images
• initially looked for slope and axis-crossings ?
  unstable
• better ? x cos(?) y sin(?)
• iterate on sample points, finding which ? and ?
  satisfy the above eq.
• sample points vote on the ?s and ?s.
• peak (?,?) points are parameters of best line
  fits

6
EXAMPLE
Grayscale image
7
EXAMPLE
Edge detection image
8
Hough intensity image
9
Lines with Highest votes
10
More Examples
Look for a scaled/rotated/relocated version of
in

• Scans around surface of object in edge image
• Iterates through parameters for each
  point
• number of possible locations per pixel
• places votes in each bin of parameters

11
More Examples
Edge detection
rotation and scaling
x, y (positioning)
12
More Examples
Detected object outlined
Edge detection
rotation and scaling
x, y (positioning)
13
More on Hough Transforms

• Problems
• Slow !
• Memory-demanding
• Designed for image processing
• Specific, with transformations
• Lines
• Circles
• Ellipses
• General curve detection
• General pattern detection
• Things that can make it FAST
• Computation Parallelizable
• real-time (specific modifications)
• hardware

14
Our Hough Transform

• Our version
• General parameter voting
• ANY dimension inputs
• ANY dimension parameters
• ANY function of the above
• NOT pixel-in-image-oriented
• which makes it slow
• Written in MatLab (no C / Mex)
• Ongoing Improvements
• Run on multiple machines
• C code
• Home - in on target parameters

• Problems
• Slow !
• Memory-demanding
• Designed for image processing
• Specific, with transformations
• Lines
• Circles
• Ellipses
• General curve detection
• General pattern detection
• Things that can make it FAST
• Computation Parallelizable
• real-time (specific modifications)
• hardware

15
Our Toy Applications

• Image Processing
• find patterns
• edge ? sketch / vectorization
• Economics
• find patterns in stock correlation trends
• use to optimize portfolio

16
Sketch lines/curves in Images
longest lines, not line segments are found
17
Sketch lines/curves in Images
longest lines, not line segments are found
pre-edging-processing
18
Stock Portfolios

• Calculate Statistics of Portfolio
• correlations between a number of stocks
• trends enter modes varying through time
• Use to predict performance / balance
• minimize risk for given mean return
• n stocks in portfolio, of each,
• with
• Maximize return
• at lowest risk

risk
19
Stock Portfolios
One point for each set of weights (many picked
at random)
Best weight set and return for given risk 0.1
risk (based on variances of stock combos)
20
Stock Portfolios
Identify different modes and use them to
adjust portfolio at each instant
time
21
Stock Portfolios
Identify different modes and use them to
adjust portfolio at each instant
Directly applying GPCA fails to see
time-continuity
time
22
Stock Portfolios
Identify different modes and use them to
adjust portfolio at each instant
Directly applying GPCA fails to see
time-continuity
time
Sounds like a GPCA (AR) problem!
23
Stock Portfolios

• Things to decide
• what covariance / means to use
• really old horizon
• recent values
• a combination
• which method to use?
• GPCA / AR seems more applicable but cant handle
  outliers
• GHT works w/ outliers well but.
• Its Slow
• time needs to be parameterized or windowed
• Data
• Yahoo! stock data into excel for covariance
• calculation

24
GPCA for finance

• Stock correlation important for portfolio
  selection
• Time frame for correlation?
• long
• short
• Independence (correlation 0)
• Correlation coefficient ignores any trends with
  stock data

25
GPCA for finance
2 modes alternating
stock2
stock1
stock2
stock1
stock2
time
stock1
but what if
26
GPCA for finance
Fit to a Hough function
trend is smooth
stock2
stock1
stock2
stock1
stock2
stock1
stock2
time
stock1
27
Stock Correlation data
2 different modes identified
28
More information

• Website
• http//legend.me.uiuc.edu/vladimer/projects/ece58
  6ym
• Information included
• References
• Hough Transform links to papers / websites
• Stock Portfolios
• Data
• Images
• Financial
• Code
• Our parameter-based generaluse Hough Transform
• Data processing