;;; This file is a trace of running the universal tester to compare different
;;; learning systems by automatically generating learning curves. 
;;; Subsequently added comments.

> (cd "/u/mooney/ml-progs")  ; switch to ml-progs directory
 #P"/tmp_mnt/v/sally/v1/mooney/ml-progs/"

> (load "universal-tester")  ; load the universal tester
;;; Loading binary file "universal-tester.sbin"
;;; Loading binary file "/u/mooney/ml-progs/data-utilities.sbin"
;;; Loading binary file "/u/mooney/ml-progs/t-test.sbin"
#P"/tmp_mnt/v/sally/v1/mooney/ml-progs/universal-tester.sbin"

> (load "id3-all")  ; load the ID3 system for inducing decision trees
;;; Loading binary file "id3-all.sbin"
#P"/tmp_mnt/v/sally/v1/mooney/ml-progs/id3-all.sbin"

> (load "pfoil")    ; load the propositional FOIL system for inducing rules
;;; Loading binary file "pfoil.sbin"
#P"/tmp_mnt/v/sally/v1/mooney/ml-progs/pfoil.sbin"

;;; First make a file for storing the exact training/test splits to be run so
;;; that later we can run additional experiments on exactly the same training
;;; and test examples.  "sample-dna-saved-tests.lisp" is the name of the file
;;; we want to save the test information in. 10 is the number of random trials
;;; we want to run.  80 is the total number of training examples we want to use.
;;; (10 25 40 80) is the set of points (in number of training examples) we want 
;;; have plotted on the learning curve (alternatively you can specify an
;;; increment like 10 for plotting after every 10 examples). NIL means to
;;; default the number of test examples to using all the remaining examples in
;;; the data file (otherwise give a number). "/u/mooney/data/dna-standard.lisp"
;;; is the file to use as the data (106 examples for DNA Promoter recognition).

> (make-saved-tests "sample-dna-saved-tests.lisp" 10 80 '(10 25 40 80) nil "/u/mooney/data/dna-standard.lisp")
;;; Loading source file "/tmp_mnt/v/sally/v1/mooney/data/dna-standard.lisp"
;;; Warning: File "/tmp_mnt/v/sally/v1/mooney/data/dna-standard.lisp" does not begin with IN-PACKAGE.  Loading into package "USER"
NIL

;;; Now we run tests to compare ID3 and PFOIL using these saved tests.
;;; (id3 pfoil) is the list of systems we want to compare.
;;; "sample-dna-saved-tests.lisp" is the saved test data file.
;;; "sample-dna-results" is the file in which to store the test results.

> (run-saved-tests '(id3 pfoil) "sample-dna-saved-tests.lisp" "sample-dna-results")

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 1:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 34.62%
Training Time: 0.01 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 34.62%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.28 sec   Test Time: 0.00 sec
Concept Complexity: 4


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.04 sec   Test Time: 0.00 sec
Concept Complexity: 1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.37 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.12 sec   Test Time: 0.01 sec
Concept Complexity: 3

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.93 sec   Test Time: 0.01 sec
Concept Complexity: 16


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.26 sec   Test Time: 0.01 sec
Concept Complexity: 6

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 1.50 sec   Test Time: 0.00 sec
Concept Complexity: 25


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.62 sec   Test Time: 0.00 sec
Concept Complexity: 11


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 2:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.30 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.05 sec   Test Time: 0.00 sec
Concept Complexity: 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.54 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.15 sec   Test Time: 0.01 sec
Concept Complexity: 4

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.67 sec   Test Time: 0.01 sec
Concept Complexity: 10


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.26 sec   Test Time: 0.00 sec
Concept Complexity: 6

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 1.41 sec   Test Time: 0.00 sec
Concept Complexity: 22


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.67 sec   Test Time: 0.01 sec
Concept Complexity: 12


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 3:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 38.46%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 42.31%
Training Time: 0.28 sec   Test Time: 0.01 sec
Concept Complexity: 7


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.05 sec   Test Time: 0.00 sec
Concept Complexity: 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.38 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.11 sec   Test Time: 0.01 sec
Concept Complexity: 3

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.91 sec   Test Time: 0.01 sec
Concept Complexity: 16


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.34 sec   Test Time: 0.01 sec
Concept Complexity: 9

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 1.73 sec   Test Time: 0.00 sec
Concept Complexity: 28


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.64 sec   Test Time: 0.01 sec
Concept Complexity: 12


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 4:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.19 sec   Test Time: 0.00 sec
Concept Complexity: 4


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.04 sec   Test Time: 0.00 sec
Concept Complexity: 1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.56 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.18 sec   Test Time: 0.00 sec
Concept Complexity: 5

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.64 sec   Test Time: 0.01 sec
Concept Complexity: 10


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.30 sec   Test Time: 0.00 sec
Concept Complexity: 8

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 1.60 sec   Test Time: 0.00 sec
Concept Complexity: 25


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.74 sec   Test Time: 0.01 sec
Concept Complexity: 14


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 5:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 30.77%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: 1


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.30 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.05 sec   Test Time: 0.01 sec
Concept Complexity: 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.52 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.16 sec   Test Time: 0.00 sec
Concept Complexity: 4

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.76 sec   Test Time: 0.01 sec
Concept Complexity: 13


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.32 sec   Test Time: 0.00 sec
Concept Complexity: 7

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 1.67 sec   Test Time: 0.00 sec
Concept Complexity: 28


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.71 sec   Test Time: 0.01 sec
Concept Complexity: 13


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 6:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.28 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.05 sec   Test Time: 0.00 sec
Concept Complexity: 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.55 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 92.31%
Training Time: 0.15 sec   Test Time: 0.00 sec
Concept Complexity: 4

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.83 sec   Test Time: 0.01 sec
Concept Complexity: 16


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.23 sec   Test Time: 0.01 sec
Concept Complexity: 6

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 1.49 sec   Test Time: 0.00 sec
Concept Complexity: 22


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.71 sec   Test Time: 0.01 sec
Concept Complexity: 13


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 7:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: 1


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.17 sec   Test Time: 0.00 sec
Concept Complexity: 4


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.05 sec   Test Time: 0.00 sec
Concept Complexity: 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.64 sec   Test Time: 0.00 sec
Concept Complexity: 13


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.16 sec   Test Time: 0.01 sec
Concept Complexity: 4

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.95 sec   Test Time: 0.01 sec
Concept Complexity: 16


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.31 sec   Test Time: 0.01 sec
Concept Complexity: 8

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 1.84 sec   Test Time: 0.00 sec
Concept Complexity: 31


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.68 sec   Test Time: 0.00 sec
Concept Complexity: 11


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 8:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 42.31%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 42.31%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.18 sec   Test Time: 0.01 sec
Concept Complexity: 4


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.03 sec   Test Time: 0.00 sec
Concept Complexity: 1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.54 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.15 sec   Test Time: 0.01 sec
Concept Complexity: 4

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.86 sec   Test Time: 0.00 sec
Concept Complexity: 16


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.22 sec   Test Time: 0.00 sec
Concept Complexity: 5

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 1.57 sec   Test Time: 0.00 sec
Concept Complexity: 25


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.53 sec   Test Time: 0.01 sec
Concept Complexity: 10


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 9:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: 1


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.18 sec   Test Time: 0.01 sec
Concept Complexity: 4


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.04 sec   Test Time: 0.00 sec
Concept Complexity: 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.53 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.15 sec   Test Time: 0.00 sec
Concept Complexity: 4

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.77 sec   Test Time: 0.01 sec
Concept Complexity: 13


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.31 sec   Test Time: 0.01 sec
Concept Complexity: 8

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 1.61 sec   Test Time: 0.01 sec
Concept Complexity: 25


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.72 sec   Test Time: 0.01 sec
Concept Complexity: 13


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 10:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.19 sec   Test Time: 0.01 sec
Concept Complexity: 4


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.04 sec   Test Time: 0.00 sec
Concept Complexity: 1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.53 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.16 sec   Test Time: 0.01 sec
Concept Complexity: 4

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.87 sec   Test Time: 0.00 sec
Concept Complexity: 16


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.33 sec   Test Time: 0.01 sec
Concept Complexity: 8

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 1.89 sec   Test Time: 0.00 sec
Concept Complexity: 31


Training PFOIL...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.66 sec   Test Time: 0.00 sec
Concept Complexity: 11


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NIL

;;; Now we want to run a statistical t-test to compare the overall results.
;;; Compares test-accuracy by default and uses 2-tailed test to be conservative.

> (t-test-file "sample-dna-results")
Total number of samples in data: 10

Comparing ID3 to PFOIL:

For 0 training examples:
Analysis of ID3 mean = 48.077 (sd 9.615) versus PFOIL mean = 51.154 (sd 9.738).
Difference: -3.077
Paired t-test results: t = -0.647, df = 9 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -13.834 to 7.680

For 10 training examples:
Analysis of ID3 mean = 56.538 (sd 13.218) versus PFOIL mean = 68.077 (sd 10.885).
Difference: -11.538*
Paired t-test results: t = -2.620, df = 9 (significant at the 0.05 level for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -21.500 to -1.577

For 25 training examples:
Analysis of ID3 mean = 71.538 (sd 10.059) versus PFOIL mean = 74.615 (sd 9.911).
Difference: -3.077
Paired t-test results: t = -0.605, df = 9 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -14.590 to 8.436

For 40 training examples:
Analysis of ID3 mean = 71.154 (sd 6.934) versus PFOIL mean = 74.615 (sd 4.615).
Difference: -3.462
Paired t-test results: t = -1.132, df = 9 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -10.379 to 3.456

For 80 training examples:
Analysis of ID3 mean = 76.154 (sd 8.566) versus PFOIL mean = 77.308 (sd 5.000).
Difference: -1.154
Paired t-test results: t = -0.461, df = 9 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -6.814 to 4.507
NIL

;;; Make a plot file suitable for the XGRAPH system for generating graphs.
;;; This creates a file sample-dna-results.test-accuracy-plot in the
;;; current directory that can be displayed by calling:
;;; xgraph sample-dna-results.test-accuracy-plot in UNIX

> (make-plot-file "sample-dna-results")
NIL

;;; We can also make a graph for training time.
;;; This creates a file sample-dna-results.train-time-plot
;;; (train-accuracy test-accuracy train-time test-time concept-complexity)
;;; are the different dependent variables we can plot.

> (make-plot-file "sample-dna-results" 'train-time)
NIL

;;; We can also use a 1-tailed test to compare training-time.
;;; (train-accuracy test-accuracy train-time test-time concept-complexity)
;;; are the list of dependent variables we can compare.

> (t-test-file "sample-dna-results" 1 'train-time)
Total number of samples in data: 10

Comparing ID3 to PFOIL:

For 0 training examples:
Analysis of ID3 mean = 0.001 (sd 0.003) versus PFOIL mean = 0.000 (sd 0.000).
Difference: 0.001
Paired t-test results: t = 1.000, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -0.001 to 0.003

For 10 training examples:
Analysis of ID3 mean = 0.235 (sd 0.054) versus PFOIL mean = 0.044 (sd 0.007).
Difference: 0.191*
Paired t-test results: t = 11.439, df = 9 (significant at the 0.0005 level for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.160 to 0.222

For 25 training examples:
Analysis of ID3 mean = 0.516 (sd 0.077) versus PFOIL mean = 0.149 (sd 0.019).
Difference: 0.367*
Paired t-test results: t = 17.765, df = 9 (significant at the 0.0005 level for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.329 to 0.405

For 40 training examples:
Analysis of ID3 mean = 0.819 (sd 0.101) versus PFOIL mean = 0.288 (sd 0.040).
Difference: 0.531*
Paired t-test results: t = 14.927, df = 9 (significant at the 0.0005 level for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.466 to 0.596

For 80 training examples:
Analysis of ID3 mean = 1.631 (sd 0.146) versus PFOIL mean = 0.668 (sd 0.058).
Difference: 0.963*
Paired t-test results: t = 18.718, df = 9 (significant at the 0.0005 level for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.869 to 1.057
NIL

;;; Later we decide we would like to run the same tests on the Nearest Neighbor
;;; instance-based method.

> (load "knn")  ; Load K Nearest Neighbor method (K=1 by default)
;;; Loading binary file "knn.sbin"
#P"/tmp_mnt/v/sally/v1/mooney/ml-progs/knn.sbin"

;;; Run the same tests on KNN and save in the file "sample-dna-knn-results"

> (run-saved-tests '(knn) "sample-dna-saved-tests.lisp" "sample-dna-knn-results")

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 1:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 38.46%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.00 sec   Test Time: 0.28 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.00 sec   Test Time: 0.71 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.00 sec   Test Time: 1.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.00 sec   Test Time: 2.15 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 2:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 38.46%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.00 sec   Test Time: 0.26 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.00 sec   Test Time: 0.67 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.00 sec   Test Time: 1.07 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.00 sec   Test Time: 2.12 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 3:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.00 sec   Test Time: 0.27 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 92.31%
Training Time: 0.00 sec   Test Time: 0.67 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 96.15%
Training Time: 0.00 sec   Test Time: 1.06 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 96.15%
Training Time: 0.00 sec   Test Time: 2.13 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 4:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.00 sec   Test Time: 0.26 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.00 sec   Test Time: 0.67 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.00 sec   Test Time: 1.08 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.00 sec   Test Time: 2.15 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 5:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.26 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.00 sec   Test Time: 0.67 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.00 sec   Test Time: 1.06 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.00 sec   Test Time: 2.14 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 6:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 34.62%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.00 sec   Test Time: 0.27 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 0.00 sec   Test Time: 0.68 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 0.00 sec   Test Time: 1.07 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.00 sec   Test Time: 2.16 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 7:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.00 sec   Test Time: 0.27 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.00 sec   Test Time: 0.67 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.00 sec   Test Time: 1.12 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.00 sec   Test Time: 2.13 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 8:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.00 sec   Test Time: 0.27 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.00 sec   Test Time: 0.68 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.00 sec   Test Time: 1.09 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.00 sec   Test Time: 2.14 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 9:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.00 sec   Test Time: 0.27 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.00 sec   Test Time: 0.67 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.00 sec   Test Time: 1.08 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.00 sec   Test Time: 2.12 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 10:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.00 sec   Test Time: 0.27 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.00 sec   Test Time: 0.67 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.00 sec   Test Time: 1.08 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training KNN...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.00 sec   Test Time: 2.13 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NIL

;;; In order to compare the new results to the old, we first change the old result
;;; file to add the new results to it.  If you don't want to overwrite the old result 
;;; file use (combine-test-results file1 file2 new-file) to combine the two result
;;; files into a new file.

> (append-test-results "sample-dna-results" "sample-dna-knn-results")
#P"/tmp_mnt/v/sally/v1/mooney/ml-progs/sample-dna-results"
#P"/tmp_mnt/v/sally/v1/mooney/ml-progs/temp"
#P"/tmp_mnt/v/sally/v1/mooney/ml-progs/sample-dna-results"

;;; Now rerun the t-test, which will compare every pair of systems in the
;;; new version of the result file

> (t-test-file "sample-dna-results" 1)
Total number of samples in data: 10

Comparing ID3 to PFOIL:

For 0 training examples:
Analysis of ID3 mean = 48.077 (sd 9.615) versus PFOIL mean = 51.154 (sd 9.738).
Difference: -3.077
Paired t-test results: t = -0.647, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -11.794 to 5.640

For 10 training examples:
Analysis of ID3 mean = 56.538 (sd 13.218) versus PFOIL mean = 68.077 (sd 10.885).
Difference: -11.538*
Paired t-test results: t = -2.620, df = 9 (significant at the 0.025 level for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -19.611 to -3.466

For 25 training examples:
Analysis of ID3 mean = 71.538 (sd 10.059) versus PFOIL mean = 74.615 (sd 9.911).
Difference: -3.077
Paired t-test results: t = -0.605, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -12.406 to 6.252

For 40 training examples:
Analysis of ID3 mean = 71.154 (sd 6.934) versus PFOIL mean = 74.615 (sd 4.615).
Difference: -3.462
Paired t-test results: t = -1.132, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -9.067 to 2.144

For 80 training examples:
Analysis of ID3 mean = 76.154 (sd 8.566) versus PFOIL mean = 77.308 (sd 5.000).
Difference: -1.154
Paired t-test results: t = -0.461, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -5.741 to 3.433


Comparing ID3 to KNN:

For 0 training examples:
Analysis of ID3 mean = 48.077 (sd 9.615) versus KNN mean = 49.615 (sd 9.948).
Difference: -1.538
Paired t-test results: t = -0.361, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -9.347 to 6.270

For 10 training examples:
Analysis of ID3 mean = 56.538 (sd 13.218) versus KNN mean = 69.615 (sd 11.207).
Difference: -13.077
Paired t-test results: t = -1.813, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -26.296 to 0.142

For 25 training examples:
Analysis of ID3 mean = 71.538 (sd 10.059) versus KNN mean = 75.769 (sd 12.047).
Difference: -4.231
Paired t-test results: t = -0.661, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -15.955 to 7.493

For 40 training examples:
Analysis of ID3 mean = 71.154 (sd 6.934) versus KNN mean = 76.923 (sd 11.279).
Difference: -5.769
Paired t-test results: t = -1.103, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -15.358 to 3.820

For 80 training examples:
Analysis of ID3 mean = 76.154 (sd 8.566) versus KNN mean = 79.615 (sd 7.892).
Difference: -3.462
Paired t-test results: t = -0.906, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -10.468 to 3.545


Comparing PFOIL to KNN:

For 0 training examples:
Analysis of PFOIL mean = 51.154 (sd 9.738) versus KNN mean = 49.615 (sd 9.948).
Difference: 1.538
Paired t-test results: t = 0.332, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -6.948 to 10.025

For 10 training examples:
Analysis of PFOIL mean = 68.077 (sd 10.885) versus KNN mean = 69.615 (sd 11.207).
Difference: -1.538
Paired t-test results: t = -0.252, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -12.720 to 9.643

For 25 training examples:
Analysis of PFOIL mean = 74.615 (sd 9.911) versus KNN mean = 75.769 (sd 12.047).
Difference: -1.154
Paired t-test results: t = -0.203, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -11.560 to 9.253

For 40 training examples:
Analysis of PFOIL mean = 74.615 (sd 4.615) versus KNN mean = 76.923 (sd 11.279).
Difference: -2.308
Paired t-test results: t = -0.502, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -10.728 to 6.113

For 80 training examples:
Analysis of PFOIL mean = 77.308 (sd 5.000) versus KNN mean = 79.615 (sd 7.892).
Difference: -2.308
Paired t-test results: t = -0.643, df = 9 (not significant for 1-tailed test)
The 0.05 level confidence interval on the difference between the means: -8.888 to 4.272
NIL

;;; If you don't care to save the actual training/test splits used and want to run a
;;; one-shot test, you can use RUN-STANDARD-TESTS instead.

> (load "perceptron")  ; Load the Perceptron neural-net learning system
;;; Loading binary file "perceptron.sbin"
;;; Loading binary file "/u/mooney/ml-progs/binary-encoder.sbin"
#P"/tmp_mnt/v/sally/v1/mooney/ml-progs/perceptron.sbin"

;;; Run one-shot tests comparing ID3 and perceptron over 15 random trials
;;; with 80 max training examples, plotting points (10 25 40 80), using
;;; all remaining examples for testing, using "/u/mooney/data/dna-standard"
;;; as the data file, and "sample-dna-results2" to put the results in.

> (run-standard-tests '(id3 perceptron) 15 80 '(10 25 40 80) nil
		      "/u/mooney/data/dna-standard" "sample-dna-results2")

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 1:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 42.31%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 38.46%
Training Time: 0.81 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.29 sec   Test Time: 0.01 sec
Concept Complexity: 7


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.11 sec   Test Time: 0.18 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 38.46%
Training Time: 0.57 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.43 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.82 sec   Test Time: 0.01 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 1.07 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 1.85 sec   Test Time: 0.01 sec
Concept Complexity: 31


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 2.05 sec   Test Time: 0.11 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 2:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.01 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.28 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.14 sec   Test Time: 0.14 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.77 sec   Test Time: 0.00 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.51 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 0.94 sec   Test Time: 0.00 sec
Concept Complexity: 16


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 92.31%
Training Time: 0.81 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 1.79 sec   Test Time: 0.00 sec
Concept Complexity: 31


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 96.15%
Training Time: 2.39 sec   Test Time: 0.11 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 3:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.01 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.18 sec   Test Time: 0.01 sec
Concept Complexity: 4


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.12 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.54 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.47 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 1.05 sec   Test Time: 0.00 sec
Concept Complexity: 19


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.85 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 1.71 sec   Test Time: 0.00 sec
Concept Complexity: 28


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 2.48 sec   Test Time: 0.10 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 4:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.00 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.18 sec   Test Time: 0.00 sec
Concept Complexity: 4


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.10 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.45 sec   Test Time: 0.01 sec
Concept Complexity: 7


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.38 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 1.11 sec   Test Time: 0.00 sec
Concept Complexity: 19


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.88 sec   Test Time: 0.16 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 1.82 sec   Test Time: 0.01 sec
Concept Complexity: 28


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 2.00 sec   Test Time: 0.11 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 5:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.18 sec   Test Time: 0.01 sec
Concept Complexity: 4


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.13 sec   Test Time: 0.14 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.56 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.55 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.80 sec   Test Time: 0.01 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.75 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 1.49 sec   Test Time: 0.00 sec
Concept Complexity: 22


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 1.99 sec   Test Time: 0.11 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 6:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.01 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.33 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.12 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.67 sec   Test Time: 0.00 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 0.42 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.93 sec   Test Time: 0.00 sec
Concept Complexity: 16


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 0.91 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 1.40 sec   Test Time: 0.00 sec
Concept Complexity: 19


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 2.10 sec   Test Time: 0.21 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 7:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 42.31%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 38.46%
Training Time: 0.27 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 42.31%
Training Time: 0.10 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.56 sec   Test Time: 0.01 sec
Concept Complexity: 10


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.42 sec   Test Time: 0.15 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.77 sec   Test Time: 0.00 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.75 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 1.47 sec   Test Time: 0.01 sec
Concept Complexity: 22


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 2.09 sec   Test Time: 0.10 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 8:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 38.46%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.00 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.18 sec   Test Time: 0.00 sec
Concept Complexity: 4


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.13 sec   Test Time: 0.13 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.54 sec   Test Time: 0.01 sec
Concept Complexity: 10


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.64 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.79 sec   Test Time: 0.00 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 92.31%
Training Time: 0.86 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 1.81 sec   Test Time: 0.00 sec
Concept Complexity: 28


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 92.31%
Training Time: 2.15 sec   Test Time: 0.11 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 9:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.01 sec   Test Time: 0.12 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.20 sec   Test Time: 0.01 sec
Concept Complexity: 4


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.12 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.56 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.42 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.67 sec   Test Time: 0.01 sec
Concept Complexity: 10


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.71 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 1.60 sec   Test Time: 0.00 sec
Concept Complexity: 25


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 1.71 sec   Test Time: 0.11 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 10:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 42.31%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.01 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.17 sec   Test Time: 0.00 sec
Concept Complexity: 4


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.09 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.45 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.51 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.74 sec   Test Time: 0.00 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 0.85 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 1.84 sec   Test Time: 0.00 sec
Concept Complexity: 31


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 2.36 sec   Test Time: 0.11 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 11:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.01 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.35 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 38.46%
Training Time: 0.13 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.70 sec   Test Time: 0.00 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 0.45 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.93 sec   Test Time: 0.00 sec
Concept Complexity: 16


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.90 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 1.93 sec   Test Time: 0.00 sec
Concept Complexity: 31


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 2.09 sec   Test Time: 0.21 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 12:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 38.46%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 42.31%
Training Time: 0.00 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.18 sec   Test Time: 0.01 sec
Concept Complexity: 4


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.12 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.44 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.50 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.87 sec   Test Time: 0.00 sec
Concept Complexity: 16


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.76 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 1.59 sec   Test Time: 0.01 sec
Concept Complexity: 25


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 1.93 sec   Test Time: 0.11 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 13:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.01 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 23.08%
Training Time: 0.00 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.27 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 61.54%
Training Time: 0.15 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.68 sec   Test Time: 0.00 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.47 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.82 sec   Test Time: 0.01 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.84 sec   Test Time: 0.16 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 1.71 sec   Test Time: 0.00 sec
Concept Complexity: 28


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 88.46%
Training Time: 1.90 sec   Test Time: 0.10 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 14:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 42.31%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.01 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 46.15%
Training Time: 0.18 sec   Test Time: 0.01 sec
Concept Complexity: 4


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.12 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.57 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 65.38%
Training Time: 0.45 sec   Test Time: 0.14 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.64 sec   Test Time: 0.00 sec
Concept Complexity: 10


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 76.92%
Training Time: 0.89 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 1.61 sec   Test Time: 0.00 sec
Concept Complexity: 25


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 92.31%
Training Time: 2.59 sec   Test Time: 0.10 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Results for trial 15:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 0 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 50.00%
Training Time: 0.00 sec   Test Time: 0.00 sec
Concept Complexity: 1


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 57.69%
Training Time: 0.00 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 10 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 53.85%
Training Time: 0.30 sec   Test Time: 0.00 sec
Concept Complexity: 7


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 0.12 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 25 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 84.62%
Training Time: 0.65 sec   Test Time: 0.01 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.56 sec   Test Time: 0.11 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 40 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 73.08%
Training Time: 0.82 sec   Test Time: 0.00 sec
Concept Complexity: 13


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 80.77%
Training Time: 0.99 sec   Test Time: 0.10 sec
Concept Complexity: NIL

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Test results for 80 training examples:

Training ID3...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 69.23%
Training Time: 1.70 sec   Test Time: 0.00 sec
Concept Complexity: 28


Training PERCEPTRON...Testing Training Data...Testing Test Data...

Training Accuracy: 100.00%   Test Accuracy: 92.31%
Training Time: 1.91 sec   Test Time: 0.11 sec
Concept Complexity: NIL


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NIL

;;; Run a t-test to compare the new test accuracy results

> (t-test-file "sample-dna-results2")
Total number of samples in data: 15

Comparing ID3 to PERCEPTRON:

For 0 training examples:
Analysis of ID3 mean = 47.179 (sd 5.699) versus PERCEPTRON mean = 47.949 (sd 8.642).
Difference: -0.769
Paired t-test results: t = -0.305, df = 14 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -6.182 to 4.644

For 10 training examples:
Analysis of ID3 mean = 62.564 (sd 12.754) versus PERCEPTRON mean = 61.795 (sd 13.059).
Difference: 0.769
Paired t-test results: t = 0.233, df = 14 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -6.305 to 7.843

For 25 training examples:
Analysis of ID3 mean = 67.436 (sd 12.938) versus PERCEPTRON mean = 76.923 (sd 10.320).
Difference: -9.487
Paired t-test results: t = -2.038, df = 14 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -19.474 to 0.500

For 40 training examples:
Analysis of ID3 mean = 73.077 (sd 9.525) versus PERCEPTRON mean = 80.769 (sd 8.309).
Difference: -7.692*
Paired t-test results: t = -3.282, df = 14 (significant at the 0.01 level for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -12.720 to -2.664

For 80 training examples:
Analysis of ID3 mean = 73.077 (sd 7.431) versus PERCEPTRON mean = 86.154 (sd 5.756).
Difference: -13.077*
Paired t-test results: t = -4.888, df = 14 (significant at the 0.001 level for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -18.815 to -7.339
NIL

;;; Check the training accuracy 

> (t-test-file "sample-dna-results2" 2 'train-accuracy)
Total number of samples in data: 15

Comparing ID3 to PERCEPTRON:

For 0 training examples:
Analysis of ID3 mean = 100.000 (sd 0.000) versus PERCEPTRON mean = 100.000 (sd 0.000).
Difference: 0.000
Paired t-test results: t = 0.000, df = 14 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.000 to 0.000

For 10 training examples:
Analysis of ID3 mean = 100.000 (sd 0.000) versus PERCEPTRON mean = 100.000 (sd 0.000).
Difference: 0.000
Paired t-test results: t = 0.000, df = 14 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.000 to 0.000

For 25 training examples:
Analysis of ID3 mean = 100.000 (sd 0.000) versus PERCEPTRON mean = 100.000 (sd 0.000).
Difference: 0.000
Paired t-test results: t = 0.000, df = 14 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.000 to 0.000

For 40 training examples:
Analysis of ID3 mean = 100.000 (sd 0.000) versus PERCEPTRON mean = 100.000 (sd 0.000).
Difference: 0.000
Paired t-test results: t = 0.000, df = 14 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.000 to 0.000

For 80 training examples:
Analysis of ID3 mean = 100.000 (sd 0.000) versus PERCEPTRON mean = 100.000 (sd 0.000).
Difference: 0.000
Paired t-test results: t = 0.000, df = 14 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.000 to 0.000
NIL

;;; Also compare training time.

> (t-test-file "sample-dna-results2" 2 'train-time)
Total number of samples in data: 15

Comparing ID3 to PERCEPTRON:

For 0 training examples:
Analysis of ID3 mean = 0.000 (sd 0.000) versus PERCEPTRON mean = 0.059 (sd 0.201).
Difference: -0.059
Paired t-test results: t = -1.093, df = 14 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -0.174 to 0.056

For 10 training examples:
Analysis of ID3 mean = 0.236 (sd 0.062) versus PERCEPTRON mean = 0.120 (sd 0.015).
Difference: 0.116*
Paired t-test results: t = 7.277, df = 14 (significant at the 0.001 level for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.082 to 0.150

For 25 training examples:
Analysis of ID3 mean = 0.581 (sd 0.094) versus PERCEPTRON mean = 0.479 (sd 0.066).
Difference: 0.102*
Paired t-test results: t = 3.400, df = 14 (significant at the 0.01 level for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: 0.038 to 0.166

For 40 training examples:
Analysis of ID3 mean = 0.847 (sd 0.125) versus PERCEPTRON mean = 0.855 (sd 0.092).
Difference: -0.008
Paired t-test results: t = -0.212, df = 14 (not significant for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -0.089 to 0.073

For 80 training examples:
Analysis of ID3 mean = 1.688 (sd 0.152) versus PERCEPTRON mean = 2.116 (sd 0.233).
Difference: -0.428*
Paired t-test results: t = -6.259, df = 14 (significant at the 0.001 level for 2-tailed test)
The 0.05 level confidence interval on the difference between the means: -0.575 to -0.281
NIL
