Fundamental research and development

1
Fundamental research and development

• PoliMI LOA TUC - BU

2
Whats out in the market?

• Electronics/optoelectronics embedded in the
  surface
• Sensitive surfaces that are built specifically
  for that purpose and sold as part of the
  equipment
• Localization achieved or made robust through
• multi-modal sensing
• active vibration emission

3
Our phylosophy

• Make objects from the real world sensitive
• Work on objects that are not designed for that
  purpose
• Learn on the spot all you can about acoustic
  propagation and use it
• TDOA, inverse modeling, dispersion backtracking
• If you cant make sense of acoustic propagation,
  be ready to work in a black-box fashion
• Time reversal
• If geometry is too hard to handle but propagation
  isnt, then use a coherent signal-based approach
• in-solid acoustic holography
• Extend the objects sensitivity beyond its
  boundary
• in-air acoustic holography
• Infer how the interaction is taking place
• What is being used and how is it being handled?
• Learn all you can about who is interacting with
  the surface
• Stroke style and pace
• Users mood

4
Model-based approach
PoliMI Augusto Sarti, Diego Rovetta, Gabriele
Scarparo
5
Modeling propagation

• Semi-infinite solid half space
• Longitudinal (P-wave)
• Shear (S-wave)
• Lateral (Head wave)
• Rayleigh (Surface wave)

• Infinite solid plates
• Longitudinal (P-wave)
• Shear (S-wave)
• Lateral (Head wave)
• Rayleigh (Surface wave)
• Lamb (Guided wave modes)

Symmetric
Antisymmetric
In thinnest plates only Lamb wave arrivals are
visible
6
Modeling propagation in infinite plates

• The dominant wave propagation modes in thin
  plates are Lamb waves
• As they propagate, Lamb waves undergo heavy
  dispersion, which can be accurately modeled if we
  know the elastic parameters of the material
• Elastic parameters can be estimated through a
  simple acoustic procedure
• Interaction wavelet (touch signature) can be
  estimated
• Wavelet at sensor can be estimated
• Impulse response at any point can be estimated

7
Active dispersion modeling

• Reference approach ultrasound pulse-echo delay
  measurement (LOA)
• Simplified approach active dispersion curve
  measurement(PoliMI-LOA)
• Through sensors Rx1 and Rx2, we compute phase
  differences and measure phase velocity
• Phase velocity is then used for estimating the
  materials elastic properties

Phase velocities of the a0 mode in MDF measured
and computed data (range of frequencies 1.5 5
kHz).
8
Passive dispersion modeling

• Excitation signal is no longer generated and
  transfered through a piezo sensor but is the
  tapping itself

Dispersioncurve
va vß estimates
Touch signal
Phase alignment
Phase unwrapping
Phase velocity computation
Exhaustivesearch
windowing
FFT
Theoreticalcurve
simulation
9
Excitation estimation

• Now that we know how the wavefront propagates, if
  the contact point is known, then the transmitted
  wavelet can be estimated through inverse
  propagation

10
Inferring signals at sensors
Knowing the touch wavelet and the propagation
model allows us to estimate the direct arrivals
acquired by all the receivers
s1
s3
s2
s4
11
Accounting for boundaries

• Real-time ray tracing (based on beam tracing)

12
Accounting for boundaries
Direct arrival
13
Accounting for boundaries
Direct arrival 1 Reflected ray
14
Accounting for boundaries
Direct arrival 2 Reflected rays
15
Accounting for boundaries
Direct arrival 3 Reflected rays
16
Accounting for boundaries
Direct arrival 4 Reflected rays
17
Using the propagation model
Estimation of the difference of the distances
between a touch position and two different
receivers
The relative distances between the receivers are
useful to circumvent the problem to know when
the pulse left the transmitter for the
estimation of the finger touch position.

• A good estimation of the difference ?dref 1 can
  be achieved by inverse propagating s1 until the
  fitting with sref in the time window of the
  direct arrival is maximized
• The best fitting can be obtained with a grid
  search technique.

18
Using the propagation model
All the signals can be inverse propagated until
the fitting with sref is maximized.
real value
estimated value
19
Using the propagation model
All the signals can be inverse propagated until
the fitting with sref is maximized.
real value
estimated value
20
Using the propagation model
All the signals can be inverse propagated until
the fitting with sref is maximized.
real value
estimated value
21
Using the propagation model
All the signals can be inverse propagated until
the fitting with sref is maximized.
real value
estimated value
22
Localization as an inverse problem

• Traditional TDOA is not accurate enough because
  of the dispersion of a0 Lamb wave.
• Dispersion is a precious source of information
  for the estimation of the relative distances
  between receivers and touch location
• An inverse problem can be formulated through
  Tarantolas inversion theory
• Find the model that bestexplains the acquired
  data

actual
estimated
23
Touch tracking

• Determine initial location (inverse problem
  method)
• Update location through inversion of the acquired
  data
• start from previous finger location
• use Tarantolas iterative technique for nonlinear
  inverse problems
• If finger is still moving, then go back to step
  2, else stop
• Apply Extended Kalman Filtering for trajectory
  regularization
• Apply particle filtering/swarm intelligence (next
  step) to account for physical motions dynamics

24


25
A preview of the demos

• Tracking handwriting in real time
• Robust estimation occlusion management through
  self-reconfiguring sensors
• Inverse problem approach for rebalancing sensor
  importance

26
Tracking of multiple contact pts through blind
channel identification

• A two-stage process
• Channel Identification
• Based on a successive decomposition of covariance
  matrices of mic signals
• TDOA estimation
• TDOA estimation from measured channels
• Inversion of TDOAs for the localization of the
  sources

27
Channel Identification
Step 1 Signal acquisition
28
Channel Identification
Step 2 Covariance matrix computation
29
Channel Identification
Step 3 SVD analysis
30
Channel Identification
Step 4 Subspace decomposition
31
Channel Identification
Step 5 Channel Identification
32
Early results
33
Low-Level touch classification

• Method based on wavelet packet decomposition and
  energy tree comparison

34
Live concert with a Tai virtual drum set
35
Live concert with a Tai virtual drum set
36
Tai tables at Comos civic museum
37
Tai tables at Comos civic museum
38
They talk about us