High Transfer Rate, Real-time Brain-Computer Interface

1
High Transfer Rate, Real-time Brain-Computer
Interface

• Machine-based learning techniques towards a
  practical spelling device for the completely
  paralyzed

2
Agenda

• Brain Computer Interfaces brief intro.
• Our system
• Overview, technical details
• Machine learning Support Vector Machines
• Additional Bandwidth Word Prediction
• Results
• Future Improvements, QA
• Demonstration at Psychology Lab

3
BCIs the Need

• Locked-in patients
• Example J.D. Bauby, The Diving Bell and the
  Butterfly
• Persistence of life butterfly
• Extreme physical disability diving bell

4
BCIs the Need

• Amyotrophic Lateral Sclerosis (ALS), aka Lou
  Gherigs
• Degeneration of motor neurons, paralysis of
  voluntary muscles
• 120,000 diagnosed each year worldwide
• 2000 Canadians live with ALS right now
• Can leave patients locked-in
• Cognitive and sensory functions remain intact

5
BCI(1) Slow Cortical Potentials (SCPs)

• Extensive training 3 months using biofeedback
  mechanism
• Tested on ALS patients, learned to control SCPs

Ref N. Birbaumer et al., The thought
translation device (TTD) for completely paralyzed
patients, IEEE Trans. Rehab. Eng., Vol. 8, pp.
190-193,June 2000.
6
BCI(1) SCPs cont.

• Most successful subject artificially fed and
  respirated for 4 years
• After 3 months of training, wrote letter below
• Took 16 hours to write 2 letters/minute
• Expresses thanks, wants to have a party

7
BCI(2) Implants - Cyberkinetics Inc.

• BrainGate Neural Interface System Mkt. cap
  45mil.
• Control of cursor on PC using implant in motor
  cortex
• Undergoing limited clinical trials
• Limb movement possibilities

8
P300 Spelling Device the P300 Event Related
Potential

• Known as oddball or surprise paradigm
• Inherent

9
P300 Spelling Device the System

• Non-invasive
• Inherent Response

10
P300 Speller Terminology

• Epoch One flash of any row or column
• Trial 1 complete set of epochs - all rows and
  columns
• Symbol Alphanumeric characters or pictures

11
BCI Competition 2003

• Provided pre-collected data for competition
• P300 Spelling Paradigm
• Winners included Kaper et al.
• Used Support Vector Machines
• Achieved high transfer rate with real-time
  implementation possibilities

12
System Operation

• Steps
• Training (approximate 1hr)
• Provide visual stimuli (flashing of rows/columns)
• Record data with known classification label
• Run data through pattern recognition algorithm
  (SVM)
• Create customized models for each individual
• Spelling
• Load customized model for individual
• Provide visual stimuli (flashing of rows/columns)
• Record data with unknown classification label
• Run data through SVM classifier
• Sum up decision values
• Feedback most probable letter

13
Display

• Flexible matrix size
• Flexible matrix contents
• Alphanumeric Characters
• Words
• Symbols

14
Display cont

• Random and exhaustive flashing of all of the rows
  and columns on display
• Flashing cycle 300ms
• 100ms intensification period
• 200ms de-intensification period
• 10 second rest period at the end of each symbol

15
Data Collection

• Collect data from DAQ sampled at 240Hz
• 600ms after intensification
• Buffer overlap
• Flexible data collection delay
• Flexible data recording time

16
Data Collection cont.

• 10 channels collected simultaneously
• Data from each channel concatenated together
• Data stored into program memory
• Collected until end of a symbol
• Converted to array
• Memory cleared for next symbol
• System is timing critical

17
Timing Issue

• Purpose
• Process within 300ms window
• Bottleneck
• Online SVM processing
• Old design 340ms/Epoch
• New design 17.67ms/Epoch
• Requirement
• Pentium4 or equivalent is sufficient

18
Matlab Interface

• Why we use Matlab?
• VBMatlab interface using APIs
• Common functions
• Pass matrix array to Matlab workspace
• Get matrix array from Matlab workspace
• Execute command line or script

19
Support Vector Machines

• Pattern recognition Algorithm
• SVM used for
• Creating models for different individuals (train)
• Getting discriminant scores (spelling)
• Detailed information covered later

20
Score Matrix
21
Word Prediction

• Idea predict intended words based on previous
  spelling. Similar to cellular phone smart text
• Extract top ranked words
• SQL for fast searching
• Dynamic database
• Selection updated on
• the bottom of the display
• Words chosen same way

22
System Design

• Modular Design Approach

23
What is SVM?

• Developed by Vapnik in 1992 at Bell Labs
• Broad applications
• Based on concept of learn from examples
• Key concepts
• Linear Decision Boundary with Margin
• Nonlinear feature transformation

24
Basic Concept

• x1, ..., xn be our training data set
• yi Î 1,-1 be the class label of xi then,
• Find a decision boundary
• Make a decision on disjoint test data

25
Decision Boundary (linear)
Class 1

• Infinite possibility

Class -1
26
Bad Decision Boundary
Class 1
Class 1
Class -1
Class -1
27
Good Decision Boundary

• Want to maximize m
• Boundary found using constrained optimization
  problem

28
Optimization Problem

• Optimization Problem

29
After Training

• xis on the decision boundary are called SUPPORT
  VECTORS
• Support vectors and b defines the decision
  boundary

30
Geometrical Interpretation
31
Non-separable Samples

• Use of Soft Margin Separation
• Kernel Transformation

32
Soft Margin Separation
33
Soft Margin Separation

• Idea simultaneous maximization of margin and
  minimization of training error

34
Nonlinear Samples

• Some Samples are inherently nonlinear in input
  space
• No linear boundary is sufficiently accurate

35
Solution?
36
Kernel Transformation

• Idea map input space into feature space such
  that samples become linearly separable

37
Gaussian Kernel
38
SVM Implementation

• Matlab interface to libsvm
• Kernel RBF with ?? 6.6799e-4
• C parameter 20.007

39
SVM Implementation

• Average Method (61.538)
• Multi-Model Method (65.22)
• Concatenation Method (82.418)
• Weighted Concatenation Method
• (max. 86.264)

40
Possible Improvements

• Weighted concatenation method
• Customized Kernel Parameters

41
Measure of Performance

• Bit Rate
• N number of available symbols
• p prediction accuracy
• t number of seconds taken to choose one symbol
• Letters per minute

42
Cont

• Resulting Transfer Rates
• Without using dictionary
• With using dictionary

43
More Accurate Measure

• Resulting Transfer Rates
• Without using dictionary
• With using dictionary

44
Cont

• Mechanism
• Receives a chosen letter from control module
• Appends the letter to current letters in the word
• Searches SQL database
• Return list of most probable target words based
  on ranking

45
Result Analysis

1. Accuracy across subjects
2. Accuracy over time, same subject
3. Accuracy over number of trials
4. Accuracy versus model size

46
Accuracy Across Subjects
Date Subject of Trials Accuracy (letters) Percentage
March 26 Jack 15 12/14 86
March 28 Min 15 12/21 57
March 30 Brian 15 19/19 100
March 31st Jyh-Liang 15 26/26 100
April 2 Lucky 15 25/28 89
47
Accuracy Across Subjects
Date Subject of Trials Accuracy (letters) Percentage
March 31st Jyh-Liang 3 13/13 100
March 31st Jyh-Liang 2 18/20 90
March 31st Jyh-Liang 1 10/21 48
48
Accuracy Over Time, Same Subject

• Subject Jack

Date Time Change from Model Made of Trials Accuracy (letters) Percentage
March 26 0 15 12/14 86
March 28 2 days 15 19/19 100
March 31 5 days 15 19/22 86
49
Accuracy Over Number of Trials

• Subject Jyh-Liang

50
Accuracy Versus Model Size

• Subject Jyh-Liang

51
Questions?