Abstract

1
Abstract
2
FPGA Implementation Of Non Linear Filters For
Image Processing

• Mr. Hirschl Boaz
• Boaz.hirschl_at_intel.com
• Guide Prof L. P. Yaroslavsky

3
Agenda

• Background
• Non Linear Filters
• Hardware and Flow
• Research
• Research goals
• Related work
• Algorithms
• Conclusion
• Results
• Demo
• Bibliography

4
The big picture

• Bio-Medical Imaging System require massive image
  processing
• Image processing solution
• Real time
• Implemented in hardware
• Focus on non linear filters.
• FPGA

5
Non Linear Filters

• Background
• Non Linear Filters
• Hardware and flow
• Research
• Research goals
• Related work
• Algorithms
• Conclusion
• Results
• Demo
• Bibliography

6
Non Linear Filters topics

• Unified approach - definitions
• What is a window
• Example of Sliding window
• Types of non linear filters
• Neighborhood Estimation
• Non linear filters examples
• Image enchantment
• Histogram equalization
• Other

7
Unification approach definition

• Filters work in a moving window.
• For each window a filter generate output value by
  means of a certain estimation operation ESTM
  applied to a certain set of values that we will
  call neighborhood NBH.

L.P. Yaroslavsky, Nonlinear Signal Processing
Filters A Unification Approach.
8
What is a window example

• We take an image
• Look at a small part on the left upper corner
• It is made of 7 x 5 pixels

9
Sliding Window

• A 3 x 3 sliding window example
• N n x n
• Number
• Of
• elements

• K nearest value (100,1)

NBH example
Sliding example
10
Unification approach pixel
L.P. Yaroslavsky, Nonlinear Signal Processing
Filters A Unification Approach.
11
Unification approach nbh estm
L.P. Yaroslavsky, Nonlinear Signal Processing
Filters A Unification Approach.
12
Window operations example

• Lets look a 3 x 3 window

• Vector it

• Sort it

• Min, Max, Median

Rank
13
Median example

• Example for 5x 5 window median filter.
• The images are before and after running in the
  hardware simulator

14
Window operations example

• Lets look a 3 x 3 window

• Vector it

• Get rank order statistics

• Min, Max, Median

• Create look up table

• Histogram equalization

Histogram
15
Unification approach hist eq
L.P. Yaroslavsky, Nonlinear Signal Processing
Filters A Unification Approach.
16
Unification approach hist eq
17
Unification approach -example
L.P. Yaroslavsky, Nonlinear Signal Processing
Filters A Unification Approach.
18
Hardware and flow

• Background
• Non Linear Filters
• Hardware and flow
• Research
• Research goals
• Related work
• Algorithms
• Conclusion
• Results
• Demo
• Bibliography

19
Hardware implementations topics

• FPGA
• VHDL
• Tools
• Flow
• Generation
• Implementation
• Verification
• Analysis
• VHDL Code generator
• Verification suite

20
FPGA - Architecture
Gate
Field
Array
Programmable

• CLB
• IOB
• OSCStartup
• JTAG
• Routes

CLB
IOB
ROT
LOG
Configuration Memory
Configure the FPGA to specific application
Configure the FPGA to specific application
Configure the FPGA to specific application
21
FPGA Building blocks CLB

• Look Up Table - LUT
• FF
• Routes

22
FPGA Building blocks IOB

• PAD
• BUFFER
• FF

23
FPGA Building blocks ROUTE

• PSM -
• Programmable
• Switching
• Matrix

24
VHDL

• Hardware Description Language
• Standard IEEE language for hardware generation
  simulation
• Top-Down design
• Design reuse
• Behavioral description
• RTL Register Transfer Logic

example
25
FLOW General

• Entering the design
• Synthesizing
• Func Simulation
• Implementation
• Time Simulation
• Programming file

26
TOOLS

• Matlab - modeling of a filter in HW writing
  style.
• Xilinx WebPACK synthesizer, mapper , place and
  route
• Model sim VHDL model simulation
• VHDL code generator

27
VHDL code generator

• One of the novelties in our work
• Creates the required VHDL code
• Support all window sizes
• Vendor independent
• Simple to use.

28
FPGA Design verification

• Take an image
• MATLAB Make it into a stream files
• Send it to simulator
• Receive the simulator output vector stream
• Verified in MATLAB environment VHDL model result
  Vs Matlab model result.

29
Research goals

• Background
• Non Linear Filters
• Hardware and flow
• Research
• Research goals
• Related work
• Algorithms
• Conclusion
• Results
• Demo
• Bibliography

30
Research goals topics

• Algorithms implementation study
• Create building blocks for real time image
  processing LEGO style
• Graphic Co-Processor
• Long term goals

31
Algorithms implementation study

• Compare different implementations for the same
  algorithms
• Compare variations of the same algorithms
• Area
• Speed - Performance
• Latency
• Power
• Other studies
• Silicon regularity
• Primitives usage
• Pipe lining and routing issues

32
Create Processing Blocks

• Serial / Parallel sorter
• Serial / Parallel Rank computer
• Serial / Parallel Occurrences computer
• Serial Histogrammer
• Histogram equalization
• Focus on the engine
• Intellectual Property (IP) philosophy

33
Create Processing Blocks

• A sorter in this example 3 input vector

34
Create Processing Blocks

• A median filter to denoise image

Noisy Image
Denoise Image
35
Graphic Co-Processor

• Advanced Bio medical imaging systems
• Accelerate graphic performance
• Concentrate on non linear filters
• Dedicated hardware
• Single Instruction Multiple Data SIMD
• Configurable processor.

36
Artificial retina

• Numerous works trying to progress in the field.

37
Related work

• Background
• Non Linear Filters
• Hardware and flow
• Research
• Research goals
• Related work
• Algorithms
• Conclusion
• Results
• Demo
• Bibliography

38
Related work topics

• Graphic processing hardware language
• Specific image processors
• Application Specific Integrated Circuit
• ASICs and boards
• Sorters
• Histogrammer

39
Image language- crooks

• In this works the group developed a high level
  language that is based on a set of image
  processing commands.
• This language can be synthesize a flexible HW
  solution
• Based on specific HW non generic
• Limited abilities

P. Donachy, Design and Implementation of a High
Level Image Processing Machine Using
reconfigurable Hardware. PhD thesis, The Queens
university of Belfast , Ireland 1996.D. Crookes,
K. Benkrid, J. Smith, A. Benkrid, High Level
Programming for Real Time FPGA-Based Video
Processing, Proceedings of ICASSP2000, Istanbul
2000.D. Crookes, K. Benkrid, A. Bourdane, K.
Alotaibi, A. Benkrid, Design and implementation
of high level programming environment for
FPGA-based image processing, IEEE Proc visual
image process, Vol. 147 No. 4 August 2000.
40
ASIC Image processor

• A full fixed image processor
• Implemented in ASIC
• Required large memory
• Parallel approach
• Off line processing
• 100 MHz 0.1Ghz 10 ns

S. Muller, A New Programmable VLSI Architecture
for Histogram and Statistics Computation In
Different Windows,IEEE08186-7310-9/95 Hamburg
Germany 1995.
41
Fixed Image processor

• A image processor that is able to do
• For a 3x3 window
• Median, Morphological , addition , subtraction ,
  mostly linear
• 100 MHz 0.1Ghz 10 ns

K.wiatr, Pipeline Architecture of specialized
reconfigurable processor in FPGA structures for
real time pre-processing,IEEE1089-6503/98
University of Krakow , Poland 1998.
42
Other

• Other sorters used specific cells
• Combination of HW and software solution

R. Lin, S.Olariu, Efficient VLSI Architecture
for column sort. IEEE Transactions on VLSI
system Vol 7, NO 1, March 1999.M. Bednara, O.
Beyer, J. Teich, R. Wanka, Tradeoff Analysis And
Architecture Design Of Hybrid Hardware/Software
Sorter, Application-Specific Systems,
Architectures, and Processors, 2000.
Proceedings., 10-12 July 2000 pg 299 308.
43
Algorithms

• Background
• Non Linear Filters
• Hardware and flow
• Research
• Research goals
• Related work
• Algorithms
• Conclusion
• Results
• Demo
• Bibliography

44
Algorithms topics

• Sorters
• Serial / Parallel
• Rank computer
• Serial / Parallel
• Histogrammer
• Serial / Parallel
• Histogram equalization

45
Sorter Serial - basic

• Cell
• Value
• Age
• Sorter
• Cells main shadow
• Full Sorter
• Not a First In First Out FIFO

46
Sorter Serial cells

• Main Cell
• Shadow cell

A 3 bit sorter
47
Parallel Sorter - basic

• Distributed Arithmetic's

Example
48
Parallel Sorter - pipeline

• Fully pipe lined sorter.
• Partly pipe lined sorter
• Interesting enough the partly pipe line sorter is
  faster in some cases.
• For example Adjustable parallel sorter works at
  15 faster then fully pipe lines sorter at 150
  MHz.

49
Parallel Rank computer

• .Compare each pair
• Sums up the comparisons
• Use of comparator primitives

SRC
OCC
HIST
Based on Prof Yaroslavsky work
50
Serial Rank computer - basic

• .Cell
• Value
• Rank
• Computer

51
Serial Rank computer - cells

• .First cell
• Rank cell
• FIFO

52
Serial Occurrences computer

• Based on Rank computer
• First occurrence cell
• Occurrence cell

53
Histogrammer

• A 5 pixel FIFO , 256 level example

108 leaves 103 enters
103 leaves 113 enters
FF HIST
54
Histogrammer - DPR

• A dual port RAM DPR histogrammer

Two port,enable The Access To two memory Cell on
the Same time.
55
Histogram equalization

• Mapping from a window gray scale
• 0-Max Pixel Value range to
• a full dynamic range
• 0-255

56
Histogram equalization

• Calculate the rank vector
• Create a Divider using a look up table
• integrate both to achieve this functionality

Histogram equalization
LUT
Rank Computer
slide
57
Results

• Background
• Non Linear Filters
• Hardware and flow
• Research
• Research goals
• Related work
• Algorithms
• Conclusion
• Results
• Demo
• Bibliography

58
Results topics

• Analysis of algorithms
• Area
• Speed - performance
• Power
• Latency
• Conclusions

59
Results Speed - basic

• The speed is for one operation on N elements and
  is defined in MHz
• For reference a 8 bit counter run at 300Mhz
• A 81 pixels sorter works at 147Mhz
• So 81 pixels will be sorted every 6.8 ns
• The speed is limited by the time it takes for a
  signal to propagate from one state element to the
  next state element.

60
Results Speed Speed

• The result are normalized to the slowest
  algorithm working at 96 MHz

61
Results Size

• For a N x N window

• Or a more General realization

N
N
62
Results Size

• For a N x N window

• Or a more General realization

N
N
63
Results Latency

• The latency is dependent on the architecture
  mainly the number of state elements

N
2
64
Results power

• The power is dependent on
• activity factor
• area used

N
N
65
Results Histograms

• Histogram using DPR is very inexpensive in terms
  of area of the FPGA.
• Each 256 DPR histogram takes about 1/32 of the
  available DPR

66
Results Histogram equalization

• Histogram equalization makes use of the rank
  computer
• The Look Up Table used to equalize the histogram
  is a ROM that is free of charge

67
Results Uniqueness

• Focus on non linear filtering
• Support any window size
• Pipe line adjustable sorter
• VHDL generator configurable processor
• HW oriented Matlab models
• Full verification suite
• IP approach
• Analysis based on implementaions

68
Conclusion

• Parallel algorithms are faster then serial
• Parallel algorithms are more costly then serial
• ADA is better then DA sorter
• FPGA are fit to process high volume data
• The usage of FPGA for NLF is feasible. Algorithms
  implementation study
• Create building blocks for real time image
• processing LEGO style
• Graphic Co-Processor
• Long term goals - After the engine is ready we
  need the body and interface.

69
Further work

• Graphic Co-Processor
• Long term goals - After the engine is ready we
  need the body and interface.
• Building more blocks like , neighborhood
  creation.
• Extending estimation operations

70
Demo

• VHDL code generator
• Implementation
• Simulation
• Image example

71
Thanks

• TO
• My wife Nava for her devoted support
• Prof Yaroslavsky for patient guidance
• Mr. Shalom Danny for helping in the GUI.

72
Bibliography

• Non linear filters
• Artificial retina
• Image processor
• Sorters
• Rank computer
• Histogramming

73
Bibliography

• 1     J. Astola, P. Kuosmanen, Fundamentals of
  Nonlinear Digital Processing, CRC Press, Boca
  Raton, N.Y., 1997
• 2     L. Yaroslavsky, Nonlinear Filters for
  Image Processing in Neuromorphic Parallel
  Networks, Optical Memory and Neural Networks,
  vol. 12, No. 1, 2003
• 3     L. Yaroslavsky, Digital Holography and
  Digital Image Processing, Kluwer scientific
  publications, Boston, 2003, ch.12.
• 4     A. Asano, K. Kazuyoshi, Y. Ichioka, The
  nearest neighbor median filter some
  deterministic properties and implementations.
  Pattern Recognition Vol23, No. 10, pp.1059-1066,
  Great Britain 1990.
• 5     P. Donachy, Design and Implementation of
  a High Level Image Processing Machine Using
  reconfigurable Hardware. PhD thesis, The Queens
  university of Belfast , Ireland 1996.
• 6     D. Crookes, K. Benkrid, J. Smith, A.
  Benkrid, High Level Programming for Real Time
  FPGA-Based Video Processing. Proceedings of
  ICASSP2000, Istanbul 2000.
• 7     D. Crookes, K. Benkrid, A. Bourdane, K.
  Alotaibi, A. Benkrid, Design and implementation
  of high level programming environment for
  FPGA-based image processing. IEEE Proc visual
  image process, Vol. 147 No. 4 August 2000.
• 8     R. Lin, S.Olariu, Efficient VLSI
  Architecture for column sort. IEEE Transactions
  on VLSI system Vol 7, NO 1, March 1999.
• 9     C. Hennind, T. G. Noll, Architecture And
  Implementation Of BitSerial Sorter For Weighted
  Median Filter. Custom Integrated Circuits
  Conference, Proceedings of the IEEE 1998, pg
  189192, University Of Technology RWTH Aachen,
  Germany.
• 10    L.Lin, G.B. Adams II, E.J. Coyle, Input
  Compression and Efficient Algorithms and
  Architectures for Stack filters. IEEE proc.
  Winter Workshop on non linear digital signal
  processing, Tempere Finland pp.5.2-5 Jan 1993
• 11    M. Bednara, O. Beyer, J. Teich, R. Wanka,
  Tradeoff Analysis And Architecture Design Of
  Hybrid Hardware/Software Sorter,
  Application-Specific Systems, Architectures, and
  Processors, 2000. Proceedings., 10-12 July 2000
  pg 299 308.
• 12    N. Woolfries, P. Lysaght, S. Marshall, G.
  McGregor, D. Robinson, Fast Implementations Of
  Non Linear Filters using FPGAs, Non-Linear
  Signal and Image Processing (Ref. No. 1998/284),
  IEE Colloquium on , 22 pg. 13/1-13/5 May 1998.
• 13    J. H. Koo, T. S. Kim, S. S. Dong, C. H.
  Lee, Development Of FPGA Based Adaptive Image
  Enhancement Filter System Using Genetic
  Algorithm , Evolutionary Computation, 2002. CEC
  '02. Proceedings of the 2002 Congress on ,
  Volume 2 , pg 1480-1485 12-17 May 2002.

74
Sorters

• 1 J. Wiseman, A Hardware architecture for
  efficient Implementation of Real-Time Weighted
  median filter .www.
•  
• 2 L.Lin, G.B. Adams II, E.J. Coyle, Input
  Compression and Efficient Algorithms and
  Architectures for Stack filters, IEEE proc.
  Winter Workshop on non linear digital signal
  processing, Tempere Finland pp.5.2-5 Jan 1993
•  
• 3 N. Woolfries, P Lysgat, S. Marshall, G.
  Mcgregor, D. Robinson, Fast implementation of
  Non-linear filters using FPGA.
•  
• 4 R. Lin, S.Olariu, Efficient VLSI Architecture
  for column sort, IEEE Transactions on VLSI system
  Vol 7, NO 1 ,March 1999
•  
• 5 I. Hatirans., Y. Leblebci, Scalable Binary
  Sorting Architecture based on Rank Ordering with
  Linaer Area Time Complexity IEEE 0-7803-6598-4/00
  2000
•  
• 6 M. Bednara,O .Beyer,J. Teich,R. Wanka,
  Tradeoff Analysis And Architecture Design Of
  Hybrid Hardware/Software Sorter, Paderborn
  University , Germany 2000.
•  
• 7 C. Hennind, T. G. Noll, Architecture And
  Implementation Of Bit Serial Sorter For Weighted
  Median Filter, RWTH Aachen, Germany 1998.

75
Sorters

• K.wiatr, Pipeline Architecture of specialized
  reconfigurable processor in FPGA structures for
  real time pre-processing,IEEE1089-6503/98
  University of Cracow , Poland 1998.
• S. Muller, A New Programmable VLSI Architecture
  for Histogram and Statistics Computation In
  Different Windows,IEEE 08186-7310-9/95 Hamburg
  Germany 1995.
• design Implementation And Evaluation of a VLSI
  High Speed array Processor for real time image
  processing morphology operations 1990 !!!
• A. Raghupathy,P. Hsu,K.J. Liu,N.
  Chandraxhoodan,VLSI Architecture and Design for
  High Performance Adaptive Video Scaling, IEEE
  0-7803-5471-0/99, University of Maryland, USA
  1999.
• M. Kelly, K. W. Kenneth, W. Hsu, A flexible
  pipelined image processor, IEEE 0-7803-4980-6/98
  NY,USA 1998
• G. Angelopoulos,I. Pitas, A Fast Implementation
  of 2-D Weighted Median Filter,IEEE 1051-4691/94
  University of Thessalonica Greece, 1994.
• P.S. Windyga, Fast Impulsive Noise Removal, IEEE
  10577149/01, University of central Florida
  Orlando 2001
• 2D median filter algorithm for parallel
  reconfigurable computers 1995

76
END

• Any questions

77
VHDL - example

• Counter example

78
Rank

• Number of the neighboring elements
• with values lower the a
• position of value a in a variational row (
  ordered, in ascending values order sequence of
  the neighborhood elements)

Original Vector
Rank Vector
variational Vector
79
Histogram

• Number of the neighboring elements with the same
  value as that of the element a.
• ( defined for quantized values).

Original Vector
Histogram
variational Vector
80
Non Linear filters
81
Histogram equalization

• Lets look a 3 x 3 window

• Get Pixels Ranks

• Create look up table

• Histogram equalization

82
Neighborhood example I

• For this 3 x 3 window
• Morphological cross/lower part
• Value -2
• Rank -1

83
Operation on Sliding window

• Running a window of n x n pixels
• N n x n
• N Number of pixels

84
FPGA Programmable Logic
Gate
Field
Array
Programmable

• logic functions
• AND OR etc, and
• Math functions ,
• Memory, FF, State Elements
• Flip Flop - FF
• Latch
• Random Access Memory - RAM
• Read Only Memory - ROM
• First In First Out - FIFO
• Dual Port Ram - DPR

85
FLOW General

• Functional specification
• Design specification
• MATLAB simulation
• Design and verification
• Implementation and analysis

86
TOOLS FPGA Design flow
87
FPGA Design - Synthesizer

• Translate VHDL into Physical components like
  Gates and FFs.
• Optimize Boolean Logic.
• Use constraints to define its goals.
• Use specific vendor primitives

88
FPGA Design - Simulator
89
Sorter Serial Main cell

• Main Cell

90
Sorter Serial Shadow cell
91
Sorter Serial 3 bit sorter
92
Parallel Sorter - array

• Distributed Arithmetic's

93
Histogrammer FF

• A dual port single state element cell
• This cell enables
• MUX on I/O
• Write enable
• Memory

94
Histogram equalization Divider

• The divider is a ROM a look up table LUT
• The input is the address of the memory cell
• The memory cell store the result of division
• The LUT will give the result for given constant
  coefficient

Divider
Address 8 bit Input Value
Output 8 bit Division Result
95
Results for Parallel Sorter

• Analysis

96
END

• END

97
Result for Serial Sorter

• Analysis of the Xilinx mapper and place and route
  reports

Parallel