Computational complexity

1
Computational complexity

• The cost of computing

2
Complexity in the algorithm analysis context

• means the cost of a program's execution
• (running time, memory, ...)
• rather than
• The cost of creating the program
• ( of statements, development time)
• In this context, less-complex programs may
  require more development time.

3
Complexity--an introduction
2

• Time complexity the time required to solve a
  problem of a particular size
• Space complexityanalysis of the computer memory

4
Several questions can be asked about a particular
algorithm

• Is it correct?, i.e., does it do what it claims
  to do when given valid input data?
• How long does it take to run?
• How much space is needed to execute the
  algorithm? i.e., how much storage is needed to
  hold data, arrays, program variables, etc.

5
Functions associated with a program

• Consider a program with one natural number as
  input.
• Two functions that can be associated with the
  program aref(n), the function computed by the
  programT(n), the running time of the program

6
Possible size measuresfor T(n)

• Total number of bits used to encode the input.
• Number of data values in the input (e.g. size of
  an array)The second is viewed as an
  approximation to the first.

7
Primitive Operations

• These are operations which we don't further
  decompose in our analysis.
• They are considered the fundamental building
  blocks of the algorithm, e.g. - /
  if( )
• Typically, the time they take is assumed to be
  constant.
• This assumption is not always valid.

8
Is multiplication really primitive?

• In doing arithmetic on arbitrarily-large numbers,
  the size of the numerals representing those
  numbers may have a definite effect on the time
  required.
• 2 x 2
• vs.
• 26378491562329846 x 786431258901237

9
Size of Numerals

• For a single number (n) input, size of the
  corresponding numeral is typically on the order
  of log(n)
• e.g. decimal numeral encoding
• size(n) digits of n
• log10n
• x smallest integer gt x (called the ceiling
  of x)

10
Asymptotic Analysis

• Typically in measuring complexity, we look at the
  growth-rate of the time function rather than the
  value itself.
• There is therefore a tendency to pay less
  attention to constant factors.
• This is called asymptotic analysis.
• It is only one part of the story sometimes
  constants are important.

11
Step Counting

• Using exact running time to measure an algorithm
  requires calibration based on the type of
  machine, clock rate, etc.
• Instead, we usually just count steps taken in the
  algorithm.
• Often we will assume primitives take one step
  each.
• This usually gives us an accurate view of the
  growth rate.

12
Straight-Line Code
x x 1 v x / z w x v
3 operations, therefore 3 steps
13
Loop Code
(These count as steps too.)
for( int i 0 i lt n i ) x x 1 v
x / z w x v
n iterations x 5 steps 1 5n1 steps
14
Non-Numeric Loop Control
for( Polylist L A !L.isEmpty() L L.rest()
) ... loop body ...
of iterations A.length()
15
Recursive Code
fac(n) n 0 ? 1 nfac(n-1)
2n 1 steps, if we count multiply and - as
steps Steps are involved in the overhead for
function calls too. The number of such steps
would be proportional to n in this case.
16
Time and space requirements

• Ex/ Suppose, given a set X consisting of n
  elements, some labeled red, some labeled
  black.
• Like to find the number of subsets of X that
  contain at least one red item
• A set with n elements has how many subsets?
• 2n
• Hence, at least 2n steps to execute
• Infeasible except for small values of n

17
Finding largest value in a finite sequence

• Algorithm searches sequence S(1),S(2),,S(N) for
  the largest value and returns it as LARGE
• Initialization Set I1, LARGES(1) // I
  current element under examination
• Find a larger value If S(I)gtLARGE, then
  LARGES(I) //If larger value is found, update
  LARGE
• Terminate?If IN, then STOP//largest value is
  LARGE
• Continue SearchII1, Go to STEP 2

18
Execution24,56,19,99,41

• I1, LARGE 24
• I2,56gt24,LARGE56
• I3,19lt56
• I4,99gt56,LARGE 99
• I5, 41lt99,DONE

19
Time/space

• To measure space requirementsdetermine amount
  of memory required to hold variables, array
  elements, etc.
• To measure time requirements, count the number of
  instructions executed ( of times each loop is
  executed, or of comparisons
• Usually, interested in complexity of input
  selected from a particular class
• Might select input of size n that gives best time
  or space performance
• Look at best case, worst case, also average case

20
Definition 4

• Let f and g be functions on 1,2,3,. Write
  f(n)O(g(n)) and say f(n) is of order at most
  g(n) if there exists a positive constant C such
  that f(n)ltCg(n) for all but finitely many
  positive integers n
• Ex/n2 n 3 lt 3n2 3n2 3n29n2 C9,
  so n2 n 3 O(n2)
• Ex/1 2nn(n1)/2(n2)/2 (n/2) lt
  (n2/2)(n2/2)n2 deduce 1 2nO(n2)
• Ex/3n36n2-4n2?

21
O Notation

• O is letter Oh (for order)
• This is big-Oh little-Oh has a different
  meaning
• Used to express upper bounds on running time
  (number of steps)
• T e O(f) means that
• (Think of O(f) as the set of functions that grow
  no faster than a constant times f.)

22
O Notation

• Constants are irrelevant in O comparisons
• If g is a function, then any function
  n gt dg(n), where g is a constant, is e O(g).
• Examples
• n gt 2.5n2 e O(n gt n2 )
• n gt 100000000n2 e O(n gt n2 )

23
Notation Abuse

• It is customary to drop the n gt and just use the
  body expression of the anonymous function
• Examples
• O(n2) instead of O(n gt n2 )
• O(n) instead of O(n gt n )
• O(1) instead of O(n gt 1)

24
Notation Abuse

• Examples
• 2.5n2 e O(n2 )
• 100000000n e O(n)
• 10000000000 e O(1)

25
Definition 3

• ANS 6n36n36n36n324n3, so O(n3)
• If an algorithm requires f(n) units of time to
  terminate for output of size n, and f(n)
  O(g(n)), then the time required by the algorithm
  is of order at most g(n) or O(g(n))
• Ex/ Suppose an algorithm is known to require
  exactly n2 n 3 units of memory for an input
  of size n. n2 n 3id O(n2), so the space
  required by this algorithm is O(n2).

26
Searching an unordered sequence

• Algorithm searches sequence S(1),S(2),,S(N) for
  the first occurrence of KEY and returns its
  location I. If KEY is not found, the algorithm
  returns the value 0
• Initialization Set I1
• Not found? If IgtN, set I0 and terminate (not
  successful)
• TestIf KEY S(I), then terminate (search
  successful)
• continue search If II1, Go to STEP 2

27
Analysis of Searching an unordered sequence

• Best case analysis If S(1)KEY, Step 3 is
  executed once
• Best case runtime (O(1))
• Worst case analysis KEY not in sequence, execute
  N times
• Worst case runtime (O(N))

28
Analysis of Searching an unordered sequence

• Average If KEY is found in the Ith position,
  Step 3 is executed I times
• If KEY not in sequence, Step 3 is executed N
  times
• Hence, average is ((123N)N)/(N1)
• ((N(N1)/2)N)/(N1)lt ((N(N1)/2)N1)/(N1)N/2
  1ltN1lt2N, so O(N) is average

29
Analysis of Searching an unordered sequence

• Normally, choose the best g(N) for O(g(N))
• Beware 300N, 5N2?when N5, second is better
  when N500, first is better

30
(No Transcript)
31
(No Transcript)
32
Examples

•  void method1(int n)
• for (int i1 i lt n i)
• System.out.println(n)
• Method1prints numbers from 1 to NN steps to
  completeO(N)

33
Examples

• void method2(int n)
• for (int i 0 i lt 2n i)
• for (int j 0 j lt n j)
• System.out.println(ij)
• Method 2 2 loopsinner and outer. For every
  fixed i in the outer loop, the inner loop will
  run from j0 to jn-1 in steps of 1. Therefore,
  for every fixed I there are n steps in the inner
  loop. Since the outer loop runs from i0 to
  i2n-1, it will execute 2n times. Together with
  the inner loop, the inside call to S.O.P.
  executes 2nn 2n2 timesO(n2)

34
Examples

•  void method5(int n)
• for (int i -2 i lt n i)
• for (int j 4 j lt n-3 j)
  System.out.println(ji)
• 2 loops, like method 2outer loop executes (2n)
  times, inner loop executes n-7times. Inner S.O.P
  executes(2n)(n-7)
• --O(n2)
• void method6(int n)
• for (int i 0 i lt 10 i)
• System.out.println(in)
• Exactly 10 timesO(1)

35
Examples

•  int method8(int n)
• int sum 0
• for (int i 0 i lt n i)
• sum i
• return sum
• Adds numbers from 1 to nn stepsO(n)

36
Examples--recursive

• void method7(int n)
• if (n gt 2)
• method7(n-2)
• System.out.println(n)
• Ex/method7(10) calls method7(8) then method7(6)
  then method7(4)
• --othersroughly, n/2 times O(n)

37
Examples--recursive

• void method3(int n)
• if (n gt 0)
• method3(n-1)
• System.out.println(n)
• method3(n-1)
• method3(1) calls method3(0) twice, so if n 1,
  the method makes two recursive calls
• method3(2) calls method3(1) twice, each of
  which calls method3(0) twice (as we already
  figured out) therefore, the method makes 2 2
  4 recursive calls

38
Examples--recursive

• void method3(int n)
• if (n gt 0)
• method3(n-1)
• System.out.println(n)
• method3(n-1)
• method3(3) calls method3(2) twice. That method,
  as we already say, makes 4 recursive calls, so
  that method3(3) will make 24 8 recursive
  calls.
• method3(4) calls method3(3) twice. That method,
  from before, makes 8 recursive calls, so that
  method3(4) will make 28 16 recursive calls.

39
Examples--recursive

• void method3(int n)
• if (n gt 0)
• method3(n-1)
• System.out.println(n)
• method3(n-1) 
• Now, however, the pattern should be clear if the
  input to method3 is n, the method will make 2n
  recursive calls. Therefore, this method is O(2n).
  Again, instead of relying on "pattern
  recognition" one could use the principle of
  induction to mathematically prove that this is
  indeed the proper order of growth.

40
Induction proof for complexity of method3
3

• Base Let n1 method3(1) calls method3(0) --
  prints out 1 calls method3(0)--then jumps out
  2(1)2 steps
• Induction Hypothesis Assume method3(n) has 2n
  steps
• Induction Step method3(n1) makes 2 calls to
  method3(n), and this has 2n 2n22n2(n1)

41
Examples--recursive

•  void method4(int n)
• if (n gt 0)
• System.out.println(n)
• method4(n / 2)
• method(16)makes 4 recursive calls to itself if
  n64, calls(32,16,8,4,2,1)

42
Examples--recursive

•  void method4(int n)
• if (n gt 0)
• System.out.println(n)
• method4(n / 2)
• if the input is a power of 2 if n 2k, the
  method calls itself k times. n 2k gt
  log_2(n) k? O(log(n ))

43
Summary

•  method1 single loop--O(n)
•  method2 double loop--O(n2)
•  method3 double recursion--O(2n) 
• method4 recursion with half the input
• O(log(n))
•  method5 double loop--O(n2)
•  method6 no dependence on n--O(1)
•  method7 simple recursion--O(n)
•  method8 single loop--O(n)

44
Analysis

• gt plot(2x,x3,log_2(x),x,1,x-10..10,y-10..10)
  
• gt

45
Why it MattersRunning Time as a function of
Complexity
Even a computer a trillion times faster wont
help in this region
46
Allowable Problem Size as a Function of
Available Time
47
Doubling Input Size
48
Black-box vs. White-box Complexity

• Black-box We have a copy of the program, with no
  code. We can run it for different sizes of
  input.
• White-box (aka Clear box) We have the code. We
  can analyze it.

49
Black-Box Complexity

• Run the program for different sizes of data set
  try to get a fix on the growth rate.
• What sizes?
• An approach is to repeatedly double the input
  size, until testing becomes infeasible (e.g. it
  takes too long, or the program breaks).

50
Black-Box Complexity

• Run on sizes 32, 64, 128, 512, or10, 100,
  1000, 10000, ...
• For each n, get time T(n).
• How can we estimate the order of run-time (e.g.
  O(n2), O(n3), etc.)?

51
Black-Box Complexity

• Suppose we are trying to establish correctness of
  the hypothesis T(n) e O(f(n))
• From the definition, we know this means there is
  a constant c such that for all n, T(n) lt c f(n)
• If our hypothesis is correct, then we expect for
  all n, T(n)/ f(n) lt c
• We can simply examine this ratio

52
Black-Box Complexity

• If we see T(n)/ f(n) lt c
• then the hypothesis T(n) e O(f(n)) is supported.
• If T(n)/ f(n) is approximately a constant as n
  increases, then the bound appears to be tight.
• If T(n)/ f(n) decreases as n increase, the bound
  is loose.
• If T(n)/ f(n) increases, we dont have a bound.

53
Algorithmfind the largest element in a sequence
of integers

• Set temp maxfirst element in the sequence
  (temporary largest integer at every stage)
• Compare next int to temp if larger than temp,
  set temp equal to the integer
• Repeat 2 if there are more integers
• Stop when no integers are lefttemp is the
  largest in the sequence

54
Analysis of finding the max element of a set with
n elements

• 2 comparisons for each termone to determine
  that the end of the list has not been reached,
  and another to determine whether to update the
  temporary maximum
• 2(n-1) ? 2 comparisons each for the second
  through the nth element
• 1 to exit the loop
• 2(n-1) 1 2n-1
• Hence, the order of the algorithm is generally
  polynomial specifically, it islinearO(n)

55
Consider a linear search for a particular element
in a list

• Worst case analysis
• of comparisons each step of the loop has 2
  comparisonsone to see if we are at the end of
  the list, and one to compare the element with the
  terms of the list
• One comparison is made outside the loop
• 2i1 comparisons inside
• At most2n2 when not in the list (2n in loop, 1
  to exit loop, one outside of loop)
• Once again, polynomial specifically, it
  islinearO(n)