Bounded Independence Fools Halfspaces

1
Bounded IndependenceFools Halfspaces

• Emanuele Viola
• Northeastern University
• work partially done at Columbia University
• Joint work with
• Ilias Diakonikolas, Parikshit Gopalan,
• Ragesh Jaiswal, Rocco Servedio
• April 2009

2
Halfspaces

• Halfspace (a.k.a. Threshold)
• h -1,1n ? -1,1
• h(x) sign(wx t) sign(w1x1
  wnxn - t)
• weights w1, , wn, t ? R
• Studied in complexity (NP ?? halfspace of
  halfspaces)
• learning (Perceptron,
  Winnow, ...)
• social choice

3
Examples of halfspaces

• h -1,1n ? -1,1 h(x) sign(w1x1 wnxn
  - t)
• Majority(x) sign( x1 xn )
• AND(x) sign( x1 xn - n 1/2 )
• x gt y? sign( 2n(xn - yn) 2n-1(xn-1 - yn-1)
  21(x1 - y1) )
• weights w1, , wn can be taken integer need wi gt
  2n

4
Our results

• Def. Distribution D over -1,1n is k-wise
  independent if
• projection on any k coordinates is uniform
  over -1,1k
• Thm Any such D ?-fools any halfspace h -1,1n
  ? -1,1
• Ex?uniform h(x) - Ex?D h(x) ?
• where k (1/?)2 polylog(1/?)
• Optimal up to polylog(1/?)
• D (x1, x2, , xk, ?i?k xi, ) h(x)
  sign(?i?k1 xi )

5
Our results on generators

• k-wise independent distribution on -1,1n
• can be generated with s (log n)k random
  bits
• Alon Babai
  Itai Chor Goldreich '85
• Corollary Explicit generator G -1,1s ?
  -1,1n that
• ?-fools any halfspace h -1,1n ? -1,1
• Ex?uniform h(x) - EY h(G(Y)) ?
• where s (log n)(1/?)2 polylog(1/?)
• First generator for halfspaces

6
Our generator vs. others

• Our result Explicit generator G -1,1s ?
  -1,1n that
• ?-fools halfspaces h(x) sign(w1x1
  wnxn - t)
• with s (log n) poly(1/?)
• Nisan ('92) fools if weights wi ?
  1,...,poly(n), S gt log2n
• Rabani Shpilka ('08) G -1,1s ? -1,1n
• hits h-1(1) (when gt ?), does not fool, s
  O(log n/?)

7
Progress on generators 2005 - now

• Random walks Trevisan Vadhan Reingold
• Polynomials Bogdanov V., Lovett
• Constant-depth circuits Bazzi, Razborov,
  Braverman
• Halfspaces Rabani Shpilka, this talk
• Challenges (1) RL ? L
• (2) Fool width-3 read-once branching program,
  s O(log n)

8
Outline

• Overview and our results
• Proof

9
Recall our result

• Def. Distribution D over -1,1n is k-wise
  independent if
• projection on any k coordinates is uniform
  over -1,1k
• Thm Any such D ?-fools any halfspace h -1,1n
  ? -1,1
• Ex?uniform h(x) - Ex?D h(x) ?
• where k (1/?)2 polylog(1/?)

10
Proof overview

• Case analysis based on structure of halfspace
• Servedio 2007 Rabani Shpilka 2008
• Def. halfspace h(x) sign(wx - t)
  sign(w1x1wnxn- t)
• regular if every wi small w.r.t. (Si
  wi2)1/2 (at most e frac.)
• regular ? wx ? Normal(0, Si wi2)
• Berry-Esséen

11
Outline

• Overview and our results
• Proof
• Regular halfspaces
• Non-regular halfspaces

12
Sandwich approximation

• Lemma Bazzi, Benjamini Gurel-Gurevich Peled
• h -1,1n ? -1,1 is e-fooled by k-wise ind.
  distributions
• ?
• ? degree-k polynomials qu,ql -1,1n ? R
• (1) ql(x) ? h(x) ? qu(x) ? x ? -1,1n
• (2) EX ? -1,1nqu(X) h(X) ? e , Eh(X)
  ql(X) ? e
• Proof (?) If D k-wise independent, X uniform
• E h(D) E h(X)
• ? E qu(D) Eql(X) (1)
• ? E qu(X) Eql(X) because qu has
  degree k
• ? 2e (2)
  Q.e.d.

13
Construction of qu

• Build univariate P R?R approximator to sign
  R?-1,1
• qu(x) P(w1x1
  wnxn)
• (t 0 and ignore
  scaling)

1
0
wx
-1
14
Properties of P
P degree k (1/e)2 (a) ? x P(x) ? sign(x)
(b) ? x ? M P(x) - sign(x) lt e (c) M gt 1,
C lt e h(x) sign(wx) qu(x) P(wx)
1
0
wx
-1
C
M
M
Assuming P, we show how to fool regular h
15
Correctness of qu
P degree k (1/e)2 (a) ? x P(x) ? sign(x)
(b) ? x ? M P(x) - sign(x) lt e (c) M gt 1,
C lt e h(x) sign(wx) qu(x) P(wx)
1
0
wx
-1
C
M
M
Want (1) h(x) ? qu(x) ? x (2) EXqu(X) h(X) ?
e
16
Correctness of qu
P degree k (1/e)2 (a) ? x P(x) ? sign(x)
(b) ? x ? M P(x) - sign(x) lt e (c) M gt 1,
C lt e h(x) sign(wx) qu(x) P(wx)
1
0
wx
-1
C
M
M
Want (1) h(x) ? qu(x) ? x Given by (a)
Q.e.d.
17
Correctness of qu
P degree k (1/e)2 (a) ? x P(x) ? sign(x)
(b) ? x ? M P(x) - sign(x) lt e (c) M gt 1,
C lt e h(x) sign(wx) qu(x) P(wx)
1
0
wx
L
-1
C
M
M
Want (2) EXqu(X) h(X) ? e - Pr wx ? C lt
e (c) h regular ( wx ?
normal) - wx ? M ? qu(X) h(X) lt e (b) - Pr
wx L lt e/qu(L)
Q.e.d.
18
Construction of P Approximation Theory
P degree k (1/e)2 (a) ? x P(x) ? sign(x)
(b) ? x ? M P(x) - sign(x) lt e (c) M gt 1,
C lt e h(x) sign(wx) qu(x) P(wx)
1
0
wx
-1
C
M
M
P best uniform approximation to sign on M
(a) Chebychev alternation theorem
(b,c) Jackson theorem
19
Outline

• Overview and our results
• Proof
• Regular halfspaces
• Non-regular halfspaces

20
Non-regular halfspaces

• Halfspace h(x) sign(w1x1wnxn- t)
• Recall regular ? wi small w.r.t. (Si wi2)1/2
  (at most e fraction)
• Definition critical index minimum number of
  variables
• to fix to make halfspace regular
• Example
• Case analysis based on critical index vs. J
  (1/e)2

critical index 3
21
Case small critical index

• If h has small critical index lt J (1/e)2
• recall fool regular halfspace with (k
  (1/e)2)-wise indep.
• Claim (J k)-wise independent fools h
• Proof Fixing J variables ? halfspace regular and
• 
  still k-wise independence
  
  
• 
  Q.e.d.

22
Case large critical index

• If h has large critical index gt J (1/e)2
• ? ? J large weights w1, , wJ
• Claim ? (J 2)-independent distribution
• with high probability x1, , xJ determine
  outcome
• Proof
• Uniform x1, , xJ ? w1 x1 wJ xJ large
• Other variables xgtJ still 2-wise independent
• ? wgtJ xgtJ concentrated ? rarely
  changes outcome
• 
  
  Q.e.d.

23
Conclusion

• Thm k-wise indep. D ?-fool halfspaces h
  -1,1n?-1,1
• E h(X) - E h(D) ?
  for k (1/?)2 polylog(1/?)
• Tight up to polylog(1/?)
• Corollary First generator G -1,1s ? -1,1n
  that
• ?-fools halfspaces seed s (log n)
  (1/?)2 polylog(1/?)
• Open Improve dependence on e. s O(log (n/e))
  ?
• Higher degree? E.g. h(x)
  sign(?wi,jxixj - t)

24

• ????????????????????????