G64ADS Advanced Data Structures

1
G64ADSAdvanced Data Structures

• Guoping Qiu
• Room C34, CS Building

2
About this course

• Study advanced data structures, algorithm design,
  analysis and implementation techniques
• To obtain advanced knowledge and practical skills
  in the efficient implementation of algorithms on
  modern computers.

3
About this course

• Pre-requisites
• Good programming experience (this course does not
  teach programming)
• Java, C, or C

4
About this course

• Timetable
• Lecture
• Friday, 1000 - 1200, C60
• Labs
• Tuesday 9.00-11.00, B52

5
About this course

• Assessments
• Final Exam 50
• Continuous assessment 50
• Homework assignments
• Programming projects

6
About this course

• References and learning materials
• Textbooks
• Mark Allen Weiss, Data Stuctures and Algorithm
  Analysis in Java, 2nd Edition, Addison Wesley,
  2007
• Mark Allen Weiss, Data Stuctures and Problem
  Solving using Java, 3rd Edition, Addison Wesley,
  2006
• Robert Sedgewick, Algorithms in C, 3rd Edition,
  Addison Wesley, 1998
• Robert Sedgewick, Algorithms in C, 3rd Edition,
  Addison Wesley, 1998
• Robert Sedgewick, Algorithms in Java, 3rd
  Edition, Addison Wesley, 2003
• Course web page
• http//www.cs.nott.ac.uk/qiu/Teaching/G64ADS
• Slides, coursework, other materials

7
Course Overview

• Advanced data structures
• ??
• Trees, hash tables, heaps, disjoint sets, graphs
• ??
• Algorithm development and analysis
• ??
• Insert, delete, search, sort
• Applications
• Implementation in Java, C, C

8
Advanced Data Structures

• Why not just use a big array?
• Example problem
• Search for a number k in a set of N numbers
• Solution
• Store numbers in an array of size N
• Iterate through array until find k
• Number of checks
• Best case 1 (k29)
• Worst case N (k25)
• Average case N/2

29
20
65
39
21
80
25
9
Advanced Data Structures

• Solution 2
• Store numbers in a binary search tree
• Search tree until find k
• Number of checks
• Best case 1 (k29)
• Worst case log2N (k25)
• Average case (log2N) / 2

29
20
65
25
80
21
39
10
Analysis

• Does it matter?
• N vs. (log2N)

11
Analysis

• Does it matter?
• Assume
• N 1,000,000,000
• 1 billion (Walmart transactions in 100 days)
• 1 Ghz processor 109 cycles per second
• Solution 1 (10 cycles per check)
• Worst case 1 billion checks 10 seconds
• Solution 2 (100 cycles per check)
• Worst case 30 checks 0.000003 seconds

12
Advanced Data Structures

• Does it matter?
• The Message
• Appropriate data structures ease design and
  improve performance
• The Challenge
• Design appropriate data structure and associated
  algorithms for a problem
• Analyze to show improved performance

13
Algorithm Analysis
14
Purpose

• Why bother analyzing code isnt getting it to
  work enough?
• Estimate time and memory in the average case and
  worst case
• Identify bottlenecks, i.e., where to reduce time
• Speed up critical algorithms

15
Algorithm

• Problem
• Specifies the desired input-output relationship
• Algorithm
• Well-defined computational procedure for
  transforming inputs to outputs
• Correct algorithm
• Produces the correct output for every possible
  input in finite time
• Solves the problem

16
Algorithm Analysis

• Predict resource utilization of an algorithm
• Running time
• Memory
• Dependent on architecture
• Serial
• Parallel
• Quantum

17
What to Analyze

• Main focus is on running time
• Memory/time tradeoff
• Simple serial computing model
• Single processor, infinite memory

18
What to Analyze

• Running time T(N)
• N is typically the size of the input
• Sorting?
• Multiplying two integers?
• Multiplying two matrices?
• Traversing a graph?
• T(N) measures number of primitive operations
  performed
• e.g., addition, multiplication, comparison,
  assignment

19
Example
20
Example

• General Rules
• Rule 1 for loop
• The running time of a for loop is at most the
  running time of the statements inside the for
  loop times the number of iterations

21
Example

• General Rules
• Rule 2 nested loop
• for (i 0 iltni)
• for(j0 jltn j)
• k
• This fragment is O(N2)

22
Example

• General Rules
• Rule 3 Consecutive statements
• for (i 0 iltni)
• ai0
• for (i 0 iltni)
• for(j0 jltn j)
• aiajij
• This fragment is O(N) work followed by O(N2) work
  -gt O(N2)

23
Example

• General Rules
• Rule 4 if/else
• If (condition)
• S1
• else
• S2
• No more than the running time of the test plus
  max S1,S2)

24
What to Analyze

• Worst-case running time Tworst(N)
• Average-case running time Tavg(N)
• Tavg(N) lt Tworst(N)
• Typically analyze worst-case behavior
• Average case hard to compute
• Worst-case gives guaranteed upper bound

25
Rate of Growth

• Exact expressions for T(N) meaningless and hard
  to compare
• Rate of growth
• Asymptotic behavior of T(N) as N gets big
• Usually expressed as fastest growing term in
  T(N), dropping constant coefficients
• e.g., T(N) 3N2 N 1 ? T(N2)

26
Rate of Growth

• T(N) O(f(N)) if there are positive constants c
  and n0 such that T(N) cf(N) when N n0
• Asymptotic upper bound
• Big-Oh notation
• T(N) O(g(N)) if there are positive constants c
  and n0 such that T(N) cg(N) when N n0
• Asymptotic lower bound
• Big-Omega notation

27
Rate of Growth

• T(N) T(h(N)) if and only if T(N) O(h(N)) and
  T(N) O(h(N))
• Asymptotic tight bound
• T(N) o(p(N)) if for all constants c there
  exists an n0 such that T(N) lt cp(N) when Ngtn0
• i.e., T(N) o(p(N)) if T(N) O(p(N)) and T(N)
  ?T(p(N))
• Little-oh notation

28
Rate of Growth

• N2 O(N2) O(N3) O(2N)
• N2 O(1) O(N) O(N2)
• N2 T(N2)
• N2 o(N3)
• 2N2 1 T(?)
• N2 N T(?)

29
Rate of Growth

• Rule 1 If T1(N) O(f(N)) and T2(N) O(g(N)),
  then
• T1(N) T2(N) O(f(N) g(N))
• T1(N) T2(N) O(f(N) g(N))
• Rule 2 If T(N) is a polynomial of degree k, then
  T(N) T(Nk)
• Rule 3 logkN O(N) for any constant k

30
Rate of Growth
31
Example

• Maximum Subsequence Sum problem
• Given (possibly negative) integers, A1, A2, ,
  AN, find the maximum value of
• e.g, for input -2, 11, -4, 13, -5, -2, the answer
  is 20 (A2 through A4)

32
Example

• Maximum Subsequence Sum problem
• Given (possibly negative) integers, A1, A2, ,
  AN, find the maximum value of
• e.g, for input -2, 11, -4, 13, -5, -2, the answer
  is 20 (A2 through A4)

33
Example

• Algorithm 1
• Compute each possible subsequence independently
• MaxSubSum1 (A)
• maxSum 0
• for i 1 to N
• for j i to N
• sum 0
• for k i to j
• sum sum Ak
• if (sum gt maxSum)
• then maxSum sum
• return maxSum

34
Example

• Algorithm 1

1. /
2. Cubic maximum contiguous subsequence sum
   algorithm.
3. /
4. public static int maxSubSum1( int a )
6. int maxSum 0
7. for( int i 0 i lt a.length i )
8. for( int j i j lt a.length j )
10. int thisSum 0
11. for( int k i k lt j k )
12. thisSum a k
14. if( thisSum gt maxSum )
15. maxSum thisSum

35
Example

• Algorithm 1 Analysis

36
Example

• Algorithm 2
• Note that
• No reason to re-compute sum each time
• MaxSubSum2 (A)
• maxSum 0
• for i 1 to N
• sum 0
• for j i to N
• sum sum Aj
• if (sum gt maxSum)
• then maxSum sum
• return maxSum

37
Example

• Algorithm 2
• /
• Quadratic maximum contiguous subsequence
  sum algorithm.
• /
• public static int maxSubSum2( int a )
• int maxSum 0
• for( int i 0 i lt a.length i )
• int thisSum 0
• for( int j i j lt a.length j )
• thisSum a j
• if( thisSum gt maxSum )
• maxSum thisSum

38
Example

• Algorithm 2 Analysis

39
Example

• Algorithm 3
• Recursive, divide and conquer
• Divide sequence in half
• A(1 ... center) and A(center1 ... N)
• Recursively compute MaxSubSum of left half
• Recursively compute MaxSubSum of right half
• Compute MaxSubSum of sequence constrained to
  use A(center) and A(center1)
• e.g., lt4, -3, 5, -2, -1, 2, 6, -2gt

40
Example

• Algorithm 3
• MaxSubSum3 (A, i, j)
• maxSum 0
• if (i j)
• then if Ai gt 0
• then maxSum Ai
• else k floor((ij)/2)
• maxSumLeft MaxSubSum3(A,i,k)
• maxSumRight MaxSubSum3(A,k1,j)
• // compute maxSumThruCenter
• maxSum maximum(maxSumLeft,maxSumRight,maxSumThru
  Center)
• return maxSum

41
Example

• Algorithm 3
• /
• Recursive maximum contiguous subsequence
  sum algorithm.
• Finds maximum sum in subarray spanning
  aleft..right.
• Does not attempt to maintain actual best
  sequence.
• /
• private static int maxSumRec( int a, int
  left, int right )
• if( left right ) // Base case
• if( a left gt 0 )
• return a left
• else
• return 0

42
Example

• int center ( left right ) / 2
• int maxLeftSum maxSumRec( a, left,
  center )
• int maxRightSum maxSumRec( a, center
  1, right )
• int maxLeftBorderSum 0, leftBorderSum
  0
• for( int i center i gt left i-- )
• leftBorderSum a i
• if( leftBorderSum gt maxLeftBorderSum
  )
• maxLeftBorderSum leftBorderSum
• int maxRightBorderSum 0, rightBorderSum
  0
• for( int i center 1 i lt right i
  )
• rightBorderSum a i
• if( rightBorderSum gt
  maxRightBorderSum )

43
Example

• Algorithm 3 Analysis

44
Example

• Algorithm 4
• Observation
• Any negative subsequence cannot be a prefix to
  the maximum sequence
• Or, a positive, contiguous subsequence is always
  worth adding
• T(N) ?

MaxSubSum4 (A) maxSum 0 sum 0
for j 1 to N sum sum
Aj if (sum gt maxSum) then
maxSum sum else if (sum lt 0)
then sum 0 return maxSum
45
Example

• Algorithm 4
• / Linear-time maximum contiguous
  subsequence sum algorithm/
• public static int maxSubSum4( int a )
• int maxSum 0, thisSum 0
• for( int j 0 j lt a.length j )
• thisSum a j
• if( thisSum gt maxSum )
• maxSum thisSum
• else if( thisSum lt 0 )
• thisSum 0
• return maxSum

46
MaxSubSum Running Times
47
MaxSubSum Running Times
48
MaxSubSum Running Times
49
Logarithmic Behaviour

• T(N) O(log2 N), usually occurs when
• Problem can be halved in constant time
• Solutions to sub-problems combined in constant
  time
• Examples
• Binary search
• Euclids algorithm
• Exponentiation

50
Binary Search

• Given an integer X and integers A0,A1,,AN-1,
  which are presorted and already in memory, find i
  such that
• AiX, or return i -1 if X is not in the input
• T(N) O(log2 N)
• T(N) T(log2 N) ?

51
Binary Search
52
Euclids Algorithm

• Compute the greatest common divisor, gcd(M,N)
  between the integers M and N
• i.e., largest integer that divides both
• Used in encryption

53
Euclids Algorithm
54
Euclids Algorithm Analysis

• Note After two iterations, remainder is at most
  half its original value
• Theorem 2.1 If M gt N, then M mod N lt M/2
• T(N) 2 log2 N O(log2 N)
• log2225 7.8, T(225) 16 (?)
• Better worst case T(N) 1.44 log2 N
• T(225) 11
• Average case T(N) 12ln2ln(N)/pi21.47
• T(225) 6

55
Exponentiation
56
Exponentiation