Speeding%20up%20multi-task%20learning

1
Speeding upmulti-task learning

• Phong T Pham

2
Multi-task learning

• Combine data from various data sources
• Potentially exploit the inter-relation between
  different data

• M datasets D D1,.,DM for M tasks
• Each dataset Dm(xm,t,ym,t),t1..Tm is i.i.d

3
Maximum Entropy Discrimination

• Similar to Bayes, assume some prior p(T) and
  solve for p(TD)
• Instead of using Bayes rule, finds p(TD) that
  minimize KL(p(TD) p(T))
• Subject to classification constraints
• Has close form solution

4
Log-linear MED

• Assume log-linear model

• Prior p(T) factorizes, and all terms are white
  Gaussians

• Leads to support vector machines

5
Kernel selection
6
Feature selection

• Special case of kernel selection
• Kernels
• kd(x,y) x(d) y(d)

7
Speeding up

• Convex optimization problem
• Can be solved using standard convex optimization
  algorithms
• Impractically slow M x T variables
• Need speeding up

8
Method

• Optimize its lower bound instead

• This upper bound for f(x) is quadratic
• Can use fast quadratic optimization methods

9
Optimization procedure

1. Initialize ?
2. Set ? ?
3. Optimize the quadratic equation
4. Re-compute coefficients and return to step 2

10
Sequential Minimal Optimization

• The quadratic optimization is similar to standard
  SVM
• Implement a variant of SMO

11
Experiments

• Feature selection
• Landmine dataset
• 29 binary tasks
• 9 features
• 450-700 examples per task
• Parameters
• Training examples 20i, i1..5
• Trade-off constant 10(i/2), i1..5
• Alpha 5(i-1), i1..5
• Performance average over 5 runs with random
  choice of training examples

12
Result Running time (1)
13
Result Running time (2)
14
Result - Performance
15
Future work

• Further improve running time
• Evaluate on more datasets

16
Thank you