.po 1i
.ll 6.5i
.ce 10
\fB\s16Homework 3: Formal Learnability\fP\s0
.sp
.sz 14
CS 395T: Machine Learning
.sp .5
Due: Thursday, March 8
.sp
.ce 0
.nr pp 12
.nr sp 12

.np
Consider the hypothesis space consisting of descriptions of \fIk\fP separate
objects each described as a purely conjunctive description on \fIn\fP nominal
features each having \fIv\fP values (like the two-object description language
used in Homework 1).  Note that the order of the object descriptions is
irrelevant but that duplicate object descriptions are allowed.

i) Determine an upper-bound on the sample complexity of any learning algorithm
that uses this hypothesis space consistently.  (i.e. a sufficient number of
examples to guarantee with probability at least 1-\(*d that the version-space
is \(*e-exhausted.)

ii) Using this upper-bound, calculate the number of examples needed when
\(*e = 0.1, \(*d = 0.1, n = 20, k = 3, and v = 5.

iii) Simplify your upper-bound by restating it in order notation (O(n)) instead
an exact numerical bound.

.np
Prove that for any hypothesis space H, if ln |H| = O(VCdim(H)) then any
learning algorithm that uses H consistently has a provably optimal sample
complexity to within a constant factor.  Note: This is almost trivial given the
results presented in the Haussler paper.

.np
Show that for k-term-DNF on \fIn\fP binary features that |H| = O(3\*[kn\*]).
Next, determine a lower bound on VCdim(H) that will allow you to show that
ln |H| = O(VCdim(H)) for constant \fIk\fP.  Given the proof for problem 2, this
demonstrates that any learning algorithm that uses k-term-DNF consistently has
a provably optimal sample complexity to within a constant factor.  The
problem is that using k-term-DNF consistently is NP-hard.


