Introduction to Algorithms

1
Introduction to Algorithms

• Jiafen Liu

Sept. 2013
2
Todays Tasks

• Binary Search Trees(BST)
• Analyzing height
• Informal Analysis
• Formal Proof
• Convexity lemma
• Jensens inequality
• Exponential height

3
Binary Search Tree

• Binary search tree is an popular data structure,
  it supports several dynamic operations.
• Search
• Minimum
• Maximum
• Predecessor
• Successor
• Insert
• Delete
• Tree Walk

4
Data structure of binary search tree

• A BST tree can be represented by a linked data
  structure in which each node is an structure.
• Key field
• Satellite data
• Pointers left, right, and parent
• For any node x, the keys in the left subtree of x
  less than or equal to keyx, and the keys in the
  right subtree of x are bigger than keyx.

5
Different Shapes of BST

• 2 binary search trees contains the same 6 keys.
• (a) A binary search tree with height 3.
• (b) A binary search tree with height 5.
• Which one is better?
• Balanced Tree and Unbalanced Tree, Whats the
  worst case of BST?

6
Whats the cost of basic operations?

• It takes T(n) time to walk an n-node binary
  search tree.
• Other basic operations on a binary search tree
  take time proportional to the height of the tree
  T(h) .

7
Tree Walk

• If x is the root of an n-node subtree, then the
  call INORDER-TREE-WALK(x) takes T(n) time.
• We will prove by induction.

8
Proof of the theorem

• Proof
• Let T(n) denote the time taken this function.
• T(1) is a constant.
• For n gt 1, suppose that function is called on a
  node x whose left subtree has k nodes and whose
  right subtree has n - k - 1 nodes.
• T(n) T(k) T(n - k - 1) d
• How to Solve it?
• substitution method

9
Search

• TREE-SEARCH (x, k)
• 1 if x NIL or k keyx
• 2 then return x
• 3 if k lt keyx
• 4 then return TREE-SEARCH(leftx, k)
• 5 else return TREE-SEARCH(rightx, k)

10
Minimum and maximum

• TREE-MINIMUM (x)
• 1 while leftx ? NIL
• 2 do x ? leftx
• 3 return x
• TREE-MAXIMUM(x)
• 1 while rightx ? NIL
• 2 do x ? rightx
• 3 return x

11
Successor and predecessor

• TREE-SUCCESSOR(x)
• 1 if rightx ? NIL
• 2 then return TREE-MINIMUM (rightx)
• 3 y ? px
• 4 while y ? NIL and x righty
• 5 do x ? y
• 6 y ? py
• 7 return y
• if the right subtree of node x is empty and x has
  a successor y, then y is the lowest ancestor of x
  whose left child is also an ancestor of x.

12
Insertion

• TREE-INSERT(T, z)
• 1 y ? NIL
• 2 x ? rootT
• 3 while x ? NIL
• 4 do y ? x
• 5 if keyz lt keyx
• 6 then x ? leftx
• 7 else x ? rightx
• 8 pz ? y
• 9 if y NIL
• 10 then rootT ? z // Tree T
  was empty
• 11 else if keyz lt keyy
• 12 then lefty ? z
• 13 else righty ? z

13
Deletion
14
Deletion

• Which node is actually removed depends on how
  many children z has.
• If z has no children, we just remove it.
• If z has only one child, we splice out z.
• If z has two children, we splice out its
  successor y, which has at most one child, and
  then replace z's key and satellite data with y's
  key and satellite data.

15
Sorting and BST

• if there is an array A, can we sort this array
  using binary search tree operations as a black
  box?
• Build the binary search tree, and then traverse
  it in order.

16
Sorting and BST

• Example A 3 1 8 2 6 7 5, what we will get?
• What's the running time of the algorithm?

17
Cost of build a BST of n nodes

• Can anybody guess an answer?
• T(nlgn)
• in most case
• T(n2)
• in the worst case
• Does this looks familiar and remind you of any
  algorithm we've seen before?
• Quicksort
• Process of Quicksort

18
BST sort and Quicksort

• BST sort performs the same comparisons as
  Quicksort, but in a different order!
• So, the expected time to build the tree is
  asymptotically the same as the running time of
  Quicksort.

19
Informal Proof

• The depth of a node equals to the number of
  comparisons made during TREE-INSERT.
• Assuming all input permutations are equally
  likely, we have
• Average node depth()
• (comparison times to insert node i)
• T(nlgn)
• T(lgn)
• The expected running time of quicksort on n
  elements is?

20
Expected tree height

• Average node depth of a randomly built BST
  T(lgn).
• Does this necessarily means that its expected
  height is also T(lgn)?

21
Outline of Formal Proof

• Prove Jensens inequality, which says that
  f(EX) Ef(X) for any convex function f and
  random variable X.
• Analyze the exponential height of a randomly
  built BST on n nodes, which is the random
  variable Yn 2Xn, where Xn is the random variable
  denoting the height of the BST.
• Prove that 2EXn E2Xn EYn O(n3), and
  hence that EXn O(lgn).

22
Convex functions

• A function f is convex if for all a, ß0 such
  that aß1, we have
• f(ax ßy) af(x)ßf(y)
• for all x,y?R.

23
Convexity lemma

• Lemma.
• Let f be a convex function, and let a1, a2 , ,
  an be nonnegative real numbers such that
  . Then, for any real numbers x1, x2, , xn, we
  have
• How to prove that?

24
Proof of convexity lemma

• Proof.
• By induction on n. For n1, we have a11
• and hence f(a1x1) a1f(x1) trivially.
• Inductive step
• why we
  introduce 1-an?
• By
  convexity
• 
  By induction

25
Convexity lemma infinite case

• Lemma.
• Let f be a convex function, and let a1, a2 , be
  nonnegative real numbers such that
• . Then, for any real numbers x1,
  x2, , we have
• assuming that these summations exist.
• We will not prove that, but intuitively its true.

26
Jensens inequality

• Jensens inequality, which says that f(EX)
  Ef(X) for any convex function f and random
  variable X.
• Proof.
• By definition of expectation
• By Convexity lemma
  (infinite case)

27
Analysis of BST height

• Let Xn be the random variable denoting the height
  of a randomly built binary search tree on n
  nodes, and let Yn 2Xn be its exponential height.
• If the root of the tree has rank k, then
• Xn 1 maxXk1,Xnk
• since each of the left and right subtrees are
  randomly built. Hence, we have
• Yn 2maxYk1,Ynk.

28
Analysis (continued)

• Define the indicator random variable Znk as
• Thus, PrZnk 1 EZnk 1/n, and
• Take expectation of both
  sides.

29
Analysis (continued)

• Linearity of expectation.
• Independence of the rank of the root from the
  ranks of subtree roots.
• The max of two nonnegative numbers is at most
  their sum, and EZnk 1/n.

Each term appears twice, and reindex.
30
Analysis (continued)

• How to solve this?
• Substitution Method
• Guess EYn cn3 for some positive constant c.
• Initial conditions for n1, EY12X12c, we
  can pick c sufficiently large to handle.

31
Analysis (continued)

• Substitution
• How to compute the expression on right?
• Integral method.

32
The grand finale

• By Jensens inequality, since f(x) 2x is
  convex, we have
• What we just showed
• Taking lg of both sides yields

33
Post mortem

• Why bother with analyzing exponential height?
• Why not just develop the recurrence on
• Xn 1 maxXk1,Xnk
• directly?

34
Answer

• The inequality maxa,b ab provides a poor
  upper bound, since the RHS approaches the LHS
  slowly as ab increases.
• The bound
• allows the RHS to approach the LHS far more
  quickly as ab increases. By using the
  convexity of f(x) 2x via Jensens inequality,
  we can manipulate the sum of exponentials,
  resulting in a tight analysis.

35
Have FUN !