Divide-and-conquer

1
Divide-and-conquer

• Dr. Deshi Ye
• yedeshi_at_zju.edu.cn

2
Divide and Conquer???

• Divide
• the problem into a number of subproblems that
  are themselves smaller instances of the same type
  of problem.
• Conquer
• Recursively solving these subproblems. If the
  subproblems are small enough, solve them
  straightforward.
• Combine
• the solutions to the subproblems into
• the solution of original problem.

3
Most common usage

• Break up problem of size n into two equal parts
  of size n/2.
• Solve two parts recursively
• Combine two solutions into overall solution in
  linear time.

4
Which are more difficult?
Divided
Conquer
Combine
5
Sort

• Obviously application
• Sort a list of names.
• Organize an MP3 library.
• Display Google PageRank results.
• List RSS news items in reverse chronological
  order.
• Problems become easy once items are in sorted
  order
• Find the median.
• Find the closest pair.
• Binary search in a database.
• Identify statistical outliers.
• Find duplicates in a mailing list.

6

• Non-obvious applications
• Data compression.
• Computer graphics.
• Computational biology.
• Supply chain management.
• Book recommendations on Amazon.
• Load balancing on a parallel computer.
• ....

7
Merge Sort

• John von Neumann 1945.

Divide
Conquer
Combine
Key subroutine Merge
8
Merging two
sorted arrays
8 4 2
9 6 3
9
Merging two
sorted arrays
8 4 2
9 6 3
2
10
Merging two
sorted arrays
8 4 2
9 6 3
8 4
9 6 3
2
11
Merging two
sorted arrays
8 4 2
9 6 3
8 4
9 6 3
2
12
Merging two
sorted arrays
8 4 2
9 6 3
8 4
9 6 3
8 4
9 6
2
13
Merging two
sorted arrays
8 4 2
9 6 3
8 4
9 6 3
8 4
9 6
4
2
14
Merging two
sorted arrays
8
9 6
8 4 2
9 6 3
8 4
9 6 3
8 4
9 6
4
2
15
Merging two
sorted arrays
8
9 6
8 4 2
9 6 3
8 4
9 6 3
8 4
9 6
4
2
6
16
Merging two
sorted arrays
8 4 2
9 6 3
8 4
9 6 3
8 4
9 6
8
9 6
8
9
4
3
2
6
17
Merging two
sorted arrays
8 4 2
9 6 3
8 4
9 6 3
8 4
9 6
8
9 6
8
9
4
3
2
6
8
18
Merging two
sorted arrays
8 4 2
9 6 3
8 4
9 6 3
8 4
9 6
8
9 6
8
9
4
3
2
6
8
9
Time Q(n) to merge a total of n elements
(linear time).
19
Analyzing merge sort
MERGE-SORT A1 . . n
T(n) Q(1) 2T(n/2) Q(n)

1. If n 1, done.
2. Recursively sort A 1 . . ?n/2? and A ?n/2?1
   . . n .
3. Merge the 2 sorted lists

Sloppiness Should be T( ?n/2? ) T( ?n/2? ) ,
but it turns out not to matter asymptotically.
20
Recurrence for merge sort

• We shall usually omit stating the base case when
  T(n) Q(1) for sufficiently small n, but only
  when it has no effect on the asymptotic solution
  to the recurrence.
• Master theorem can find a good upper bound on
  T(n).

21
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
22
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
T(n)
23
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
24
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
25
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
cn
cn/2
cn/2
cn/4
cn/4
cn/4
cn/4

Q(1)
26
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
cn
cn/2
cn/2
h lg n
cn/4
cn/4
cn/4
cn/4

Q(1)
27
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
cn
cn
cn/2
cn/2
h lg n
cn/4
cn/4
cn/4
cn/4

Q(1)
28
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
cn
cn
cn/2
cn
cn/2
h lg n
cn/4
cn/4
cn/4
cn/4

Q(1)
29
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
cn
cn
cn/2
cn
cn/2
h lg n
cn/4
cn/4
cn
cn/4
cn/4


Q(1)
30
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
cn
cn
cn/2
cn
cn/2
h lg n
cn/4
cn/4
cn
cn/4
cn/4


Q(1)
leaves n
Q(n)
31
Recursion tree
Solve T(n) 2T(n/2) cn, where c gt 0 is
constant.
cn
cn
cn/2
cn
cn/2
h lg n
cn/4
cn/4
cn
cn/4
cn/4


Q(1)
leaves n
Q(n)
Total Q(n lg n)
32
Computing

• For any integer x and n, please compute the value
• Calculate the Fibonacci number
• Hint

33
Integer Multiplication
34
Integer Multiplication

• Complex multiplication.
• (a bi) (c di) x yi.
• Grade-school.
• x ac - bd, y bc ad.
• Gauss
• x ac - bd, y (a b) (c d) - ac - bd.
• Remark.
• Improvement if no hardware multiply.

4 multiplications, 2 additions
3 multiplications, 5 additions
35
Integer Arithmetic

• Add. Given two n-bit integers a and b, compute a
  b.
• Grade-school. T(n) bit operations.
• Multiply. Given two n-bit integers a and b,
  compute a b.
• Grade-school. T(n2) bit operations.

36
To multiply two n-bit integers

• Multiply four n/2-bit integers.
• Add two n/2-bit integers, and shift to obtain
  result.
• x (10ma b), y (10mc d)
• Ex x 1234567890, m5, a12345, b67890
• (10ma b) (10mc d) 102mac 10m(bc ad) bd

37
Multiply (x, y, n) If n 1 return x y Else
m n/2 a x/10m, b x mod 10m c
y/10m, d y mod 10m e multiply(a, c,m) f
multiply(b, d, m) g multiply(b, c, m) h
multiply(a, d, m) Reutrn 102m e 10m(gh) f
38
Fast multiply

• T(n) 4T(n/2) O(n), T(1) 1
• Which solves to T(n) O(n2) by the master
  theorem.
• Anatolii, Karatsuba in 1962
• T(n) 3 T(n/2) O(n)
• T(n) O(nlg3)O(n1.585)

Ac bd - (a - b)(c - d) bc ad
39
FastMultiply (x, y, n) If n 1 return x
y Else m n/2 a x/10m, b x mod 10m
c y/10m, d y mod 10m e multiply(a, c,m)
f multiply(b, d, m) g multiply(a - b, c
- d, m) Reutrn 102m e 10m(e f - g) f
40
Fast Integer Division Too

• Integer division. Given two n-bit (or less)
  integers a and b, compute quotient
• q a / b and remainder r a mod b.
• Complexity of integer division is same as
  multiplication. To compute quotient q
• Approximate x 1 / b using Newton's method
• xi1 2xi bxi2 apply fast
  multiply
• After log n iterations, q a x

41
Newton Method

• Goal. Given a function f (x), find a value x
  such that f(x) 0.
• Newton's method.
• Start with initial guess x0.
• Compute a sequence
• of approximations

42
Exam.

• Approximately compute x 1 / t using exact
  arithmetic. Let t 7. t 0.142857142857142857142
  85714285714
• x0 0.1
• x1 0.13
• x2 0.1417
• x3 0.142847770
• x4 0.14285714224218970
• x5 0.14285714285714285449568544449737
• x6 0.1428571428571428571428571428571428080902311
  3867839307631644158170

43
Newtons method

• xn1 xn f(xn)/f'(xn)
• Let x 1/b, f(x) 1/x b, f' (x) - 1/x2

44
Integer Division Newton's Method

• (q, r) NewtonDivision(s, t)

Choose x to be unique fractional power of 2 in
interval (1 / (2t), 1 / t repeat lg n times
x 2x t x2 set q integer(s x ) set r s
q t
45
Analysis

• Lemma 1. Iterates converge monotonically.
• Lemma 2. Iterates converge quadratically to 1 /
  t

46
Root

• The approximation of
• Let f(x) x2 - a

47
Matrix multiplication

• Given two n-by-n matrices X and Y, compute Z
  XY.

j
i


Y
Z
X
48
Running time

• Brute force
• Fundamental question can you improve?

49
Matrix multiplication

• Divide-and-conquer
• Divide partition A and B into n/2-by-n/2 blocks.
• Conquer multiply 8 n/2-by-n/2 recursively.
• Combine add appropriate products using 4 matrix
  additions.

50
Runnig time
51
Key Idea

• Key idea. multiply 2-by-2 block matrices with
  only 7 multiplications.
• where

52
(No Transcript)
53
Fast Matrix multiply

• Fast matrix multiplication. Strassen, 1969
• Divide partition X and Y into n/2-by-n/2 blocks.
• Compute 14 n/2-by-n/2 matrices via 10 matrix
  additions.
• Conquer multiply 7 pairs of n/2-by-n/2 matrices
  recursively.
• Combine 7 products into 4 terms using 8 matrix
  additions.
• Analysis

54
Fast Matrix Multiplication in Theory

• Q Multiply two 2-by-2 matrices with 7 scalar
  multiplications?
• A Yes! Strassen, 1969
• Q Multiply two 2-by-2 matrices with 6 scalar
  multiplications?
• A Impossible. Hopcroft and Kerr, 1971
• Q Two 3-by-3 matrices with 21 scalar
  multiplications?
• A Also impossible.
• New Can be solved in 23 scalar multiplications.
  Laderman, 1976 Courtois 2011

55
Fast Matrix Multiplication in Practice

• Implementation issues.
• Sparsity.
• Caching effects.
• Numerical stability.
• Odd matrix dimensions.
• Crossover to classical algorithm around n 128.
• Common misperception Strassen is only a
  theoretical curiosity.
• Advanced Computation Group at Apple Computer
  reports 8x speedup on G4 Velocity Engine when n
  2,500.
• Range of instances where it's useful is a subject
  of controversy.

56
Fast Matrix Multiplication in Theory

• Best known. O(n2.376)
• Coppersmith-Winograd, 1987
• Conjecture.
• Caveat. Theoretical improvements to Strassen are
  progressively less practical.

57
Fast Fourier Transform

• Applications
• Optics, acoustics, quantum physics,
  telecommunications, radar, control systems,
  signal processing, speech recognition, data
  compression, image processing, seismology, mass
  spectrometry
• Digital media. DVD, JPEG, MP3, H.264
• Medical diagnostics. MRI, CT, PET scans,
  ultrasound
• Numerical solutions to Poisson's equation.
• Shor's quantum factoring algorithm

58

• The FFT is one of the truly great computational
  developments of the 20th century. It has
  changed the face of science and engineering so
  much that it is not an exaggeration to say that
  life as we know it would be very different
  without the FFT.
• -- Charles van Loan

59
Polynomials Coefficient Representation

• Polynomial. coefficient representation
• Add. O(n) arithmetic operations.
• Multiply (convolve). O(n2) using brute force.

60
Polynomials Point-Value
Representation

• Gauss, PhD thesis A degree n polynomial is
  uniquely characterized by its values at any n1
  distinct points. (Two points determine a line)

yA(xi )
xi
61
Polynomials Point-Value
Representation

• Polynomial. point-value representation
• Add. O(n) arithmetic operations.
• Multiply (convolve). O(n) but need 2n-1 points.

62
Evaluate

• How to evaluate A(x0) for the point (x0,y0)
• Horners Rule
• It takes for computing all A(x0)

63
Converting Between Two Representations Brute
Force

• Coefficient point-value. Given a polynomial
  A(xi) , evaluate at n different point xi, i.e. to
  compute
• Hence, we could get (xi, yi)

64
Converting Between Two Representations Brute
Force

• point-value Coefficient. Given n distinct
  points and n values
• find unique polynomial
• Vander-monder matrix is invertible

65
Vander-monde Matrix

• Vander-monde matrix is invertible iff xi
  different
• Matrix , its determinant is

66
Converting Between Two Polynomial
Representations

• Tradeoff. Fast evaluation or fast multiplication.
  We want both!
• Goal. Make all ops fast by efficiently converting
  between two representations.

Representation Multiply Evaluate
Coefficient O(n2) O(n)
Point value O(n) O(n2)
67
Algorithm Polynomial multiplication

• Input.
• Coefficient of two polynomials, A and B of degree
  n
• Output.
• Their product CAB
• Selection.
• Pick some points x0 , ..., x2n-2
• Evaluation
• Compute A(xi), B(xi)
• Multiplication.
• Ci A(xi)B(xi)
• Interpolation.
• Recover coefficient of C

O(n2)
O(n)
O(n2)
68
Polynomial multiply
Coefficient representation
a0 ,a1 ,...,an-1 b0 ,b1 ,...,bn-1
c0 ,c1 ,...,c2n-2
Inverse FFT O(n log n)
FFT O(n log n)
Point-value multiply
A(x0) ... A(x2n-2) B(x0) ... B(x2n-2)
Point-value representation
C(x0)...C(x2n-2)
O(n)
69
Coefficient to Point-Value Intuition

• Divide. Break polynomial up into even and odd
  powers.
• Intuition. Choose two points to be

70

• Can evaluate polynomial of degree n at 2 points
  by evaluating two polynomials of degree n/2 at 1
  point.
• T(n) 2T(n/2) O(n), hence T(n) O(n lgn)
• Question left

71
Discrete Fourier Transform

• Key idea. Choose where is
  principal nth root of unity.

Fourier Matrix
Discrete Fourier transform
72
Roots of Unity

• Def. An nth root of unity is a complex number x
  such that xn 1.
• Fact. The nth root of unity are
• where and
• Proof.

b
r
a
73
Root of unity

• Fact. The (n/2)th roots of unity are
• Note.

74
Fast Fourier Transform

• Goal. Evaluate a degree n-1 polynomial
• At its nth root of unity
• Divide.

75
FFT con.

• Conquer. Evaluate Aeven(x) and Aodd(x) at its
  (n/2)th root of unity
• Combine.

76
FFT Summary

• Theorem. FFT algorithm evaluates a degree n-1
  polynomial at each of the nth roots of unity in
  O(n log n) steps.
• Running time.
• Now, coefficient -gt point-value is done

77
Point - coefficient

• To get the coefficient from point value
  representation

78
Inverse FFT

• Claim. Inverse of Fourier matrix is given by
  following formula.
• Consequence. To compute inverse FFT, apply same
  algorithm but use as nth root
  of unity

79
Inverse FFT Proof of Correctness

• Claim. Fn and Gn are inverses.
• Pf.
• Summation lemma.
• Pf. If k is a multiple of n,
• is a root of
• otherwise , we have

80
Inverse FFT summary

• Theorem. Inverse FFT algorithm evaluates a degree
  n-1 polynomial at each of the nth roots of unity
  in O(n log n) steps.
• Running time

81
Polynomial multiply

• Theorem. Can multiply two degree n-1 polynomials
  in O(n log n) steps.

Coefficient representation
a0 ,a1 ,...,an-1 b0 ,b1 ,...,bn-1
c0 ,c1 ,...,c2n-2
Inverse FFT O(n log n)
FFT O(n log n)
Point-value multiply
A(x0) ... A(x2n-2) B(x0) ... B(x2n-2)
Point-value representation
C(x0)...C(x2n-2)
O(n)
82
Big Num

• Number representation
• Using radix b positional notation, an interger N
• Can be written as
• b gt1 is the base
• For all i, ni are the digits of N written in base
  b
• Class java.math.BigNum

83
Some big numbers

• Pi 3.14159265358979323846264338327950288419716939
  93751058209749445923078164062862089986280348253421
  17067982148086513282306647093844609550582231725359
  40812848111745028410270193852110555964462294895493
  03819644288109756659334461284756482337867831652712
  01909145648566923460348610454326648213393607260249
  14127372458700660631558817488152092096282925409171
  53643678925903600113305305488204665213841469519415
  11609433057270365759591953092186117381932611793105
  11854807446237996274956735188575272489122793818301
  19491298336733624406566430860213949463952247371907
  02179860943702770539217176293176752384674818467669
  40513200056812714526356082778577134275778960917363
  71787214684409012249534301465495853710507922796892
  58923542019956112129021960864034418159813629774771
  30996051870721134999999837297804995105973173281609
  63185950244594553469083026425223082533446850352619
  31188171010003137838752886587533208381420617177669
  14730359825349042875546873115956286388235378759375
  19577818577805321712268066130019278766111959092164
  20198938095257201065485863278865936153381827968230
  30195203530185296899577362259941389124972177528347
  913151557485724245415

84
e

• http//world.std.com/reinhold/BigNumCalc.html