Held: Wednesday, 25 February 2004
Summary:
Today we continue with our investigation of algorithm analysis. In particular, we consider recurrence relations in some detail.
Related Pages:
Due
Assignments:
Notes:
- I hope to have exams graded tomorrow.
- Optional reading: John Stone's Algorithms for Functional Programming.
- Cool panel tomorrow in South lounge from 11:00 to 12:45. Free lunch!
Overview:
- Big O notation, revisited.
- Doing Big-O Analysis.
- Dominant terms.
- Big-O Analysis of Recursive Procedures.
- Experimental analysis.
- f(n) is in O(g(n)) iff
- there exists a number n_{0}
- there exists a number d > 0
- for all n > n_{0},
abs(f(n)) <= abs(d*g(n))
- This notation provides a convenient mechanism for talking about
running time without worrying about too many potentially useless
details.
- One of the nice things about asymptotic analysis is that it makes
constants
unimportant
because they can be hidden
in the d.
- If f(n) is 100n seconds and g(n)
is 0.5n seconds, then
- f(n) is in O(g(n))
[let d be 200]
- g(n) is in f(n)
[let d be 1]
- If f(n) is 100n seconds and g(n) is
n^{2} seconds, then
f(n) is in O(g(n))
[let n_{0} be 100 and d be 1]
- However, g(n) is not in O(f(n)).
Why not?
- Suppose there were an n_{0} and a d.
- Consider what happens for n = 101d.
- d*f(n) = d*100*101*d =
d*d*100*101.
- However, g(n) = d*d*101*101, which is
even larger.
- If n_{0} is greater than 101d, we'll still have
this problem [proof left to reader].
- Since constants can be eliminated, we normally don't write them.
- That is, we say that the running time of an algorithm is
O(n) or O(n^{2}) or ....
- Most computer scientists begin their study of Big-O by analyzing
simple iterative functions.
- Here's how you start:
- Count each step as one.
- Count each conditional as taking one plus the maximum of the
number of steps in each branch.
- Count each loop as taking the number of repetitions times
the cost of the body (perhaps plus one).
- Count each procedure call as taking one plus the number of
steps in the procedure call.
- You can then simplify the result by dropping lower-order terms
and eliminating constants.
- Sometimes it seems that we're overly casual in analyzing algorithm.
- For example, in our analysis of selection sort last class, we ignored that the
findSmallest
algorithm was working with smaller and smaller lists.
- Sometimes this casual approach makes a differnce.
- At other times, such as in this case, it makes no difference.
- Note that the total cost of all the calls to
findSmallest
is n (finding the smallest in the full vector) + n-1 (finding the smallest remaining after one was removed) + n-2 (etc.) + .... + 2 + 1.
- That sum, as many people know is n(n+1)/2, which is O(n^{2}).
- Note that some of the counting can be hard, so we may also need to
estimate throughout the process.
- We can often express the running time of an algorithm as a composition of
other functions.
- For example, we might have a primary control structure which repeats
its contents O(h(n)) times and within that control
structure, we have a subalgorithm that takes O(g(n)) time.
- Similarly, we might break our algorithm up into two independent
algorithms, one of which takes O(h(n)) time and one of
which takes O(g(n)) time.
- We might also look at other relationships.
- Here are some questions we might ask ourselves about composition of functions.
- If f(n) is in O(h(n)) and g(n) is
O(k(n)), then what can we say about f(n) +
g(n)?
- If f(n) is in O(h(n)) and g(n) is
O(k(n)), then what can we say about
f(n) * g(n)?
- We might also ask more general questions about Big-O notation.
- If f(n) is in O(g(n)) and g(n) is
O(h(n)) then what can we say about f(n) and
h(n)?
- If f(n) is in O(kg(n)) and k
is a
constant
, then what else can we say about
f(n) and g(n)?
- If f(n) is in O(g(n) + h(n)) and
g(n) is in O(h(n)), then what simpler thing
can we say about f(n)?
- Unfortunately, the techniques for analyzing algorithms we discussed
above focuse primarily on iterative algorithms. What about
recursive algorithms?
- The process seems inherently more difficult.
- However, we can use a similar strategy.
- Count the number of recursive calls.
- Count the time spent during each recursive call (except for the time
spent in subsequent recursive calls).
- How do you count the number of recursive calls? Often, the
shrink
function gives you a sense of how many recursive calls you need to do
- That is, how many calls to
shrink
does it take to reduce the
argument to the base case.
- It is also possible to use the concept of recurrence relations to
determine the running time of an algorithm.
- For example, if a recursive algorithm takes c steps and reduces the
parameter by one, we might express the running time as a function,
time(n) which represents the running time on input of
n elements:
- time(n) = c + time(n-1)
- time(1) = c
- This equation is relatively easy to solve "intuitively".
- time(n) = c + time(n-1)
- = c + c + time(n-2)
- = ...
- = c + c + ... + time(1)
- = c + c + ... + c [n times]
- = c*n
- Other interesting recurrence relations:
- time(n) = n + time(n-1)
- time(n) = c n + time(n-1)
- time(n) = c + time(n/2)
- time(n) = n + time(n/2)
- time(n) = c * time(n-1)
- time(n) = c * time(n/2)
- Particularly as you start to analyze algorithms, it may be helpful
to analyze them experimentally as well as abstractly.
- What do I mean when I say
analyze them experimentally
? I mean
that we can time them on a variety of inputs and graph the results.
- If the experimental and the abstract results match, we can be fairly
confident in the abstract results.
- If they don't, we may need to try again.
- Note that some analyses can be very difficult.