Machine Learning Chapter 28: Cross-Validation

What this chapter covers

This chapter contains 9 topics. Technical terms are followed by plain-language meanings in parentheses where they first appear. Code is included only when it naturally helps demonstrate the concept; architecture, workflow, governance, and comparison topics use practical scenarios instead.

28.1 Validation Data

Validation Data (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 28, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Imagine a small machine-learning project. Use Validation Data to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Validation Data
const records = [3, 5, 7, 9, 11];
const transform = value => ({ input: value, output: value * 2 + 1 });
const results = records.map(transform);

console.log(results);

Code explanation

The sample starts with a small list of inputs so every result can be checked manually.
`transform()` represents the main operation for this topic in a deliberately simple form.
`map()` applies the same rule consistently to every item and returns a new result array.
Use this pattern to focus on input, transformation, and output before replacing the toy rule with a more advanced method.

Expected result: A transformed result is printed for each input value.

Practice exercise

Create a small real-world example for Validation Data. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

28.2 Holdout Validation

Holdout Validation (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 28, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Imagine a small machine-learning project. Use Holdout Validation to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Holdout Validation
const records = [3, 5, 7, 9, 11];
const transform = value => ({ input: value, output: value * 2 + 1 });
const results = records.map(transform);

console.log(results);

Code explanation

The sample starts with a small list of inputs so every result can be checked manually.
`transform()` represents the main operation for this topic in a deliberately simple form.
`map()` applies the same rule consistently to every item and returns a new result array.
Use this pattern to focus on input, transformation, and output before replacing the toy rule with a more advanced method.

Expected result: A transformed result is printed for each input value.

Practice exercise

Create a small real-world example for Holdout Validation. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

28.3 K-Fold Cross-Validation

K-Fold Cross-Validation (repeatedly splitting data to estimate performance on unseen examples). Within Chapter 28, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Split a dataset into several parts. Train on most parts and test on the remaining part, then repeat so each part is used for testing once. Average the results for a more dependable estimate.

Coding example

// K-Fold Cross-Validation
const data = [1,2,3,4,5,6,7,8,9,10];
const folds = 5;
for (let fold = 0; fold < folds; fold++) {
  const test = data.filter((_,i) => i % folds === fold);
  const train = data.filter((_,i) => i % folds !== fold);
  console.log({ fold: fold + 1, train, test });
}

Code explanation

The sample dataset is divided into several folds.
For each round, one fold becomes the test set and all remaining values become the training set.
Repeating the process lets every item appear in a held-out set once.
This demonstrates why cross-validation gives a more stable evaluation than relying on one lucky train/test split.

Expected result: Five train/test splits are printed.

Practice exercise

Create a second example for K-Fold Cross-Validation. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

28.4 Stratified K-Fold

Stratified K-Fold (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 28, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Split a dataset into several parts. Train on most parts and test on the remaining part, then repeat so each part is used for testing once. Average the results for a more dependable estimate.

Coding example

// Stratified K-Fold
const data = [1,2,3,4,5,6,7,8,9,10];
const folds = 5;
for (let fold = 0; fold < folds; fold++) {
  const test = data.filter((_,i) => i % folds === fold);
  const train = data.filter((_,i) => i % folds !== fold);
  console.log({ fold: fold + 1, train, test });
}

Code explanation

The sample dataset is divided into several folds.
For each round, one fold becomes the test set and all remaining values become the training set.
Repeating the process lets every item appear in a held-out set once.
This demonstrates why cross-validation gives a more stable evaluation than relying on one lucky train/test split.

Expected result: Five train/test splits are printed.

Practice exercise

Create a second example for Stratified K-Fold. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

28.5 Leave-One-Out

Leave-One-Out (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 28, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Imagine a small machine-learning project. Use Leave-One-Out to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Leave-One-Out
const records = [3, 5, 7, 9, 11];
const transform = value => ({ input: value, output: value * 2 + 1 });
const results = records.map(transform);

console.log(results);

Code explanation

The sample starts with a small list of inputs so every result can be checked manually.
`transform()` represents the main operation for this topic in a deliberately simple form.
`map()` applies the same rule consistently to every item and returns a new result array.
Use this pattern to focus on input, transformation, and output before replacing the toy rule with a more advanced method.

Expected result: A transformed result is printed for each input value.

Practice exercise

Create a small real-world example for Leave-One-Out. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

28.6 Repeated Cross-Validation

Repeated Cross-Validation (repeatedly splitting data to estimate performance on unseen examples). Within Chapter 28, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Split a dataset into several parts. Train on most parts and test on the remaining part, then repeat so each part is used for testing once. Average the results for a more dependable estimate.

Coding example

// Repeated Cross-Validation
const data = [1,2,3,4,5,6,7,8,9,10];
const folds = 5;
for (let fold = 0; fold < folds; fold++) {
  const test = data.filter((_,i) => i % folds === fold);
  const train = data.filter((_,i) => i % folds !== fold);
  console.log({ fold: fold + 1, train, test });
}

Code explanation

The sample dataset is divided into several folds.
For each round, one fold becomes the test set and all remaining values become the training set.
Repeating the process lets every item appear in a held-out set once.
This demonstrates why cross-validation gives a more stable evaluation than relying on one lucky train/test split.

Expected result: Five train/test splits are printed.

Practice exercise

Create a second example for Repeated Cross-Validation. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

28.7 Grouped Cross-Validation

Grouped Cross-Validation (repeatedly splitting data to estimate performance on unseen examples). Within Chapter 28, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Split a dataset into several parts. Train on most parts and test on the remaining part, then repeat so each part is used for testing once. Average the results for a more dependable estimate.

Coding example

// Grouped Cross-Validation
const data = [1,2,3,4,5,6,7,8,9,10];
const folds = 5;
for (let fold = 0; fold < folds; fold++) {
  const test = data.filter((_,i) => i % folds === fold);
  const train = data.filter((_,i) => i % folds !== fold);
  console.log({ fold: fold + 1, train, test });
}

Code explanation

The sample dataset is divided into several folds.
For each round, one fold becomes the test set and all remaining values become the training set.
Repeating the process lets every item appear in a held-out set once.
This demonstrates why cross-validation gives a more stable evaluation than relying on one lucky train/test split.

Expected result: Five train/test splits are printed.

Practice exercise

Create a second example for Grouped Cross-Validation. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

28.8 Time-Series Validation

Time-Series Validation (data recorded in time order). Within Chapter 28, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Imagine a small machine-learning project. Use Time-Series Validation to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Time-Series Validation
const sequence = [2,4,3,5,7];
let state = 0;
const alpha = 0.6;
const states = sequence.map(x => {
  state = alpha * x + (1-alpha) * state;
  return Number(state.toFixed(2));
});
console.log(states);

Code explanation

The input values arrive in order, so earlier information can influence later calculations.
`state` stores a running memory instead of treating every value independently.
The update blends the new input with the previous state.
The printed states demonstrate the idea of sequential models and online updates maintaining information through time.

Expected result: A state value is printed for every step in the sequence.

Practice exercise

Create a small real-world example for Time-Series Validation. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

28.9 Nested Cross-Validation

Nested Cross-Validation (repeatedly splitting data to estimate performance on unseen examples). Within Chapter 28, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Split a dataset into several parts. Train on most parts and test on the remaining part, then repeat so each part is used for testing once. Average the results for a more dependable estimate.

Coding example

// Nested Cross-Validation
const data = [1,2,3,4,5,6,7,8,9,10];
const folds = 5;
for (let fold = 0; fold < folds; fold++) {
  const test = data.filter((_,i) => i % folds === fold);
  const train = data.filter((_,i) => i % folds !== fold);
  console.log({ fold: fold + 1, train, test });
}

Code explanation

The sample dataset is divided into several folds.
For each round, one fold becomes the test set and all remaining values become the training set.
Repeating the process lets every item appear in a held-out set once.
This demonstrates why cross-validation gives a more stable evaluation than relying on one lucky train/test split.

Expected result: Five train/test splits are printed.

Practice exercise

Create a second example for Nested Cross-Validation. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

Chapter 28 Review Questions and Answers

Q1. What is Validation Data?

Answer: Validation Data is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q2. What is Holdout Validation?

Answer: Holdout Validation is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q3. What is K-Fold Cross-Validation?

Answer: K-Fold Cross-Validation is repeatedly splitting data to estimate performance on unseen examples. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q4. What is Stratified K-Fold?

Answer: Stratified K-Fold is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q5. What is Leave-One-Out?

Answer: Leave-One-Out is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q6. What is Repeated Cross-Validation?

Answer: Repeated Cross-Validation is repeatedly splitting data to estimate performance on unseen examples. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q7. What is Grouped Cross-Validation?

Answer: Grouped Cross-Validation is repeatedly splitting data to estimate performance on unseen examples. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q8. What is Time-Series Validation?

Answer: Time-Series Validation is data recorded in time order. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q9. What is Nested Cross-Validation?

Answer: Nested Cross-Validation is repeatedly splitting data to estimate performance on unseen examples. In this chapter, focus on the input, the method or decision, and the result that should be checked.