[28.1] In a machine learning model, how do the training, validation, and test sets differ?
Solution:
Training Set is the largest portion of your data (typically ~70%). The model learns from this data by adjusting its internal parameters to recognize patterns. Think of it as the actual studying phase.
Validation Set (~15%) is used during development to check how the model is performing on data it wasn't trained on. You use it repeatedly to tune parameters to prevent overfitting which is when a model memorizes training data so well it performs poorly on new data
Test Set (~15%) is held back entirely until the very end. It's only used once to give a final, unbiased measure of real-world performance. Using it multiple times would essentially "leak" information and make it act like a second validation set.
[28.2] True / False questions.
a) Testing data includes data from the training stage.
b) The model is run on the testing data multiple times like validation data.
c) A lot of data is always good.
d) The more training data there is, the more accurate the model becomes.
e) Validation data is evaluated multiple times during model development.
f) The test data is used to repeatedly adjust and improve the model.
g) The accuracy score is calculated by comparing predicted results with actual results.
Solution:
a) False
b) False
c) False
d) True
e) True
f) False
g) True
[28.3] Why is it bad to use training or validation data when testing the model?
Solution: Because the model has already learned from it, which leads to inaccurate evaluation. It is like taking an exam but you have already practiced on the same exam; it will not evaluate correctly your learning. And it will not correctly predict how you will do in exams problems you have not seen before.
[28.4] Ariel uses 100% of her data to train her model and then tests and validates using the same data. Is this good?
Solution: Using 100% of data for training and then testing on the same data is deeply flawed. The model has already memorized the answers, so it will score unrealistically high. This is called overfitting. It tells you nothing about how the model will perform on new, unseen data in the real world. It's like grading a student only on homework they've already completed.
[28.5] James uses 70% training, 10% validation, and 20% testing. Is this a good approach?
Solution: The 70/10/20 split is a well-established and reasonable strategy. Each subset has a distinct, non-overlapping role:
70% training gives the model enough data to learn meaningful patterns
10% validation allows tuning and catching overfitting during development
20% testing provides a fair, unbiased final evaluation
The key is that no data is shared between the three sets, so each evaluation is honest.
[28.6] Carlos trains on all data and tests using the same dataset. Should he use this model?
Solution: This is the same fundamental mistake as Ariel's, just framed differently. Training and testing on the same data produces overfitting — the model learns the training data too specifically and loses the ability to generalize. The reported accuracy will be artificially inflated and completely unreliable as a measure of real-world performance. The model should be retrained properly with separate splits before being trusted.
[28.7] In machine learning, the data is split into the training set and the ______ set.
Solution: testing
Even if we do not use a valdiation set, there is always a trainign and testing set.
[28.8] What does it mean for a model to be overfitting?
Solution: The model learns the training data too closely and performs poorly on new data.
[28.9] Why do we use heuristics?
Solution: To find good enough solutions (not the best or most perfect solution) when exact solutions take too long to compute.