Reading: Chapter 17

My algorithm is faster!

No, mine is!

Where will this lead? Will Alice resort to silencing Bob? Will Bob just give up? Is there any way they can settle their dispute? Find out today with...

(with special guest star Spot!)

Approach 1: Implement and Test

Alice and Bob could program their algorithms and try them out on some sample inputs.

But my algorithm is too complicated to implement if we're just going to throw it away!

Maybe we'll miss some inputs on which your algorithm is bad!

Arf! [translation: Maybe Alice's algorithm will look better on small test inputs, but when we get to the really big, interesting problems, Bob's is better.]

Doesn't this depend on which machine you use?

This isn't really a bad idea, but they have a point.

Approach 2: Graph and Extrapolate

Graph performance against problem size, and extrapolate.

     |                  .
     |     current      .
     |    computing     .
     |      power -+   .
     |     limits  |   .
     |             |  .
  ^  |             | .
  |  |             v.
time |            _~
     |           -
     |         _~
     |      _-~
     |___--~
     |
     +---------------------------
          problem size -->

But how do I know if the extrapolation is correct?

It would be better if we didn't treat the algorithm as a black box.

Approach 3: Create a formula

Write down the algorithm and create a formula for the running time.

  Prime-Test-All

       i <- 2
         |
         v          no
 +-> i <= sqrt(N)? -----> output IS_PRIME
 |       |
 |       | yes
 |       v           yes
 |  does i divide N? ---> output NOT_PRIME
 |       |
 |       | no
 |       v
 +--- i <- i + 1

The time consumed by Prime-Test-All for input N is at most

T[PTA](N) <= T[i <- 2] + (sqrt(N) - 1) (T[i <= sqrt(N)?] + T[i divide N?]
                                          + T[i <- i + 1]) + T[output]

No implementation!

But what a mess!

Approach 4: Approximate

For really big inputs, we can ignore everything but the fastest-growing term: T[PTA](N) <= c[PTA] sqrt(N) (where c[PTA] is a constant that will change between machines).

Say Bob's algorithm takes T[B](N) <= c[B] sqrt(N) log_2(N) time.

For what N is Bob's algorithm better?

Ignore the Constants

To compare algorithms with very different bounds, we can just ignore the constants. We write T[PTA] = O(sqrt(N)) or T[B] = O(sqrt(N) log_2(N)).

This is called big-O notation.

Aren't you simplifying a bit too much?

Yes, this is a gross simplification. But it often works!

Big-O with mathematical functions

50 x² + 25 x + 40 = ??
5,096 log₂ n + 0.02 n = ??
4,236,121 = ??
4 * 2ⁿ log_2 n + n^2 = ??

Big-O as upper bound

Note that big-O is just an upper bound.

An O(n) algorithm is also an O(2ⁿ) algorithm.

Isn't this outrageously weak then?

Sometimes we don't know how to find the exact bound. Big-O still applies.

Big-O with analyzing algorithms

A few intuitive rules:

Constant rule: Any segment of computer code takes O(1) time if it does not involve any function calls or loops.

Sequence rule: If one thing takes O(f(n)) time, and another takes O(g(n)), then doing one after another takes O(max{f(n), g(n)}) time.

Iteration rule: If each iteration of a loop takes O(f(n)) time, and there are O(g(n)) iterations, then the entire loop takes O(f(n)g(n)) time.

While always correct, the Iteration Rule sometimes gives weaker bounds than possible.

Lots of examples...

Algorithm Add-Matrices(a, b): for each i from 1 to m, do:
    for each j from 1 to n, do:
        Let c[i,j] be a[i,j] + b[i,j].
    end of loop
end of loop
return c.

double exponentiate(double x, int n) {
    double ret = 1.0;
    for(int i = 0; i < n; i = i + 1) {
        ret = ret * x;
    }
    return ret;
}

double exponentiate(double x, int n) {
    double ret = 1.0;
    double y = x;
    int i = n;
    while(i > 0) { // always at this point y^n == x^i * ret
        if(i % 2 == 0) { // i is even
            i = i / 2;
        } else {
            ret = ret * x;
            i = (i - 1) / 2;
        }
        x = x * x;
    }
    return ret; // i must be 0, so y^n == x^0 * ret == ret
}

Algorithm Count-Primes(N): // count primes <= N
Let count be 0.
for each i between 2 and N, do:
    if Prime-Test-All(i) == true, do:
        Add 1 to count.
    end of if
end of loop
return count.

Algorithm Prime-Test-Odds(n):
if n is even and n != 2, then:
    return false.
end of if
for each odd integer i from 2 to sqrt(n, do:
    if i divides n, then:
        return false.
    end of if
end of loop
return true.
stop.

We multiply two numbers a and b as we learned in grade school:

  a =    2210
* b =     125
     --------
        11050
        4420
     + 2210
    ---------
       276250