# Class 21: Algorithm Analysis (2)

Back to Algorithm Analysis (1). On to Searching.

Held: Wednesday, 25 February 2004

Summary: Today we continue with our investigation of algorithm analysis. In particular, we consider recurrence relations in some detail.

Related Pages:

Due

Assignments:

Notes:

• I hope to have exams graded tomorrow.
• Optional reading: John Stone's Algorithms for Functional Programming.
• Cool panel tomorrow in South lounge from 11:00 to 12:45. Free lunch!

Overview:

• Big O notation, revisited.
• Doing Big-O Analysis.
• Dominant terms.
• Big-O Analysis of Recursive Procedures.
• Experimental analysis.

## Review

• f(n) is in O(g(n)) iff
• there exists a number n0
• there exists a number d > 0
• for all n > n0, abs(f(n)) <= abs(d*g(n))
• This notation provides a convenient mechanism for talking about running time without worrying about too many potentially useless details.

## Eliminating Constants

• One of the nice things about asymptotic analysis is that it makes constants unimportant because they can be hidden in the d.
• If f(n) is 100n seconds and g(n) is 0.5n seconds, then
• f(n) is in O(g(n)) [let d be 200]
• g(n) is in f(n) [let d be 1]
• If f(n) is 100n seconds and g(n) is n2 seconds, then f(n) is in O(g(n)) [let n0 be 100 and d be 1]
• However, g(n) is not in O(f(n)). Why not?
• Suppose there were an n0 and a d.
• Consider what happens for n = 101d.
• d*f(n) = d*100*101*d = d*d*100*101.
• However, g(n) = d*d*101*101, which is even larger.
• If n0 is greater than 101d, we'll still have this problem [proof left to reader].
• Since constants can be eliminated, we normally don't write them.
• That is, we say that the running time of an algorithm is O(n) or O(n2) or ....

## Doing Big-O Analysis

• Most computer scientists begin their study of Big-O by analyzing simple iterative functions.
• Here's how you start:
• Count each step as one.
• Count each conditional as taking one plus the maximum of the number of steps in each branch.
• Count each loop as taking the number of repetitions times the cost of the body (perhaps plus one).
• Count each procedure call as taking one plus the number of steps in the procedure call.
• You can then simplify the result by dropping lower-order terms and eliminating constants.

## Care in Analysis

• Sometimes it seems that we're overly casual in analyzing algorithm.
• For example, in our analysis of selection sort last class, we ignored that the `findSmallest` algorithm was working with smaller and smaller lists.
• Sometimes this casual approach makes a differnce.
• At other times, such as in this case, it makes no difference.
• Note that the total cost of all the calls to `findSmallest` is n (finding the smallest in the full vector) + n-1 (finding the smallest remaining after one was removed) + n-2 (etc.) + .... + 2 + 1.
• That sum, as many people know is n(n+1)/2, which is O(n2).

## Relating Functions

• Note that some of the counting can be hard, so we may also need to estimate throughout the process.
• We can often express the running time of an algorithm as a composition of other functions.
• For example, we might have a primary control structure which repeats its contents O(h(n)) times and within that control structure, we have a subalgorithm that takes O(g(n)) time.
• Similarly, we might break our algorithm up into two independent algorithms, one of which takes O(h(n)) time and one of which takes O(g(n)) time.
• We might also look at other relationships.
• Here are some questions we might ask ourselves about composition of functions.
• If f(n) is in O(h(n)) and g(n) is O(k(n)), then what can we say about f(n) + g(n)?
• If f(n) is in O(h(n)) and g(n) is O(k(n)), then what can we say about f(n) * g(n)?
• If f(n) is in O(g(n)) and g(n) is O(h(n)) then what can we say about f(n) and h(n)?
• If f(n) is in O(kg(n)) and k is a constant, then what else can we say about f(n) and g(n)?
• If f(n) is in O(g(n) + h(n)) and g(n) is in O(h(n)), then what simpler thing can we say about f(n)?

## Analyzing Recursive Procedures

• Unfortunately, the techniques for analyzing algorithms we discussed above focuse primarily on iterative algorithms. What about recursive algorithms?
• The process seems inherently more difficult.
• However, we can use a similar strategy.
• Count the number of recursive calls.
• Count the time spent during each recursive call (except for the time spent in subsequent recursive calls).
• How do you count the number of recursive calls? Often, the shrink function gives you a sense of how many recursive calls you need to do
• That is, how many calls to shrink does it take to reduce the argument to the base case.
• It is also possible to use the concept of recurrence relations to determine the running time of an algorithm.
• For example, if a recursive algorithm takes c steps and reduces the parameter by one, we might express the running time as a function, time(n) which represents the running time on input of n elements:
• time(n) = c + time(n-1)
• time(1) = c
• This equation is relatively easy to solve "intuitively".
• time(n) = c + time(n-1)
• = c + c + time(n-2)
• = ...
• = c + c + ... + time(1)
• = c + c + ... + c [n times]
• = c*n
• Other interesting recurrence relations:
• time(n) = n + time(n-1)
• time(n) = c n + time(n-1)
• time(n) = c + time(n/2)
• time(n) = n + time(n/2)
• time(n) = c * time(n-1)
• time(n) = c * time(n/2)

## Experimental Analysis

• Particularly as you start to analyze algorithms, it may be helpful to analyze them experimentally as well as abstractly.
• What do I mean when I say analyze them experimentally? I mean that we can time them on a variety of inputs and graph the results.
• If the experimental and the abstract results match, we can be fairly confident in the abstract results.
• If they don't, we may need to try again.
• Note that some analyses can be very difficult.

## History

Back to Algorithm Analysis (1). On to Searching.

Disclaimer: I usually create these pages on the fly, which means that I rarely proofread them and they may contain bad grammar and incorrect details. It also means that I tend to update them regularly (see the history for more details). Feel free to contact me with any suggestions for changes.

This document was generated by Siteweaver on Fri May 7 09:43:15 2004.
The source to the document was last modified on Tue Jan 13 10:26:11 2004.
This document may be found at `http://www.cs.grinnell.edu/~rebelsky/Courses/CS153/2004S/Outlines/outline.21.html`.

You may wish to validate this document's HTML ; ; Check with Bobby

Samuel A. Rebelsky, rebelsky@grinnell.edu