Chapter 29: Hyperparameter Optimization
Learn Machine Learning from very beginner to expert with detailed topic guidance, practical examples, practice exercises, and review questions.
What this chapter covers
This chapter contains 10 topics. Technical terms are followed by plain-language meanings in parentheses where they first appear. Code is included only when it naturally helps demonstrate the concept; architecture, workflow, governance, and comparison topics use practical scenarios instead.
29.1 Parameters vs Hyperparameters
Parameters vs Hyperparameters (a model or training setting chosen outside the learned parameters). Within Chapter 29, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.
When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.
Example
Imagine a small machine-learning project. Use Parameters vs Hyperparameters to decide what information is needed, what step happens next, and what result should be checked.
Coding example
// Parameters vs Hyperparameters
const choices = [0.01, 0.05, 0.1, 0.2];
const evaluate = value => 1 - Math.abs(value - 0.08);
const results = choices.map(value => ({ value, score: evaluate(value) }));
results.sort((a,b) => b.score - a.score);
console.log('best choice:', results[0]);Code explanation
- `choices` represents candidate settings that could be tried automatically.
- `evaluate()` stands in for a validation process that assigns each candidate a score.
- All candidates are evaluated and sorted from best to worst.
- The highest-scoring setting is selected, demonstrating the core search loop behind many tuning systems.
Expected result: The best candidate setting and its score are printed.
Practice exercise
Create a small real-world example for Parameters vs Hyperparameters. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.
29.2 Manual Tuning
Manual Tuning (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 29, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.
When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.
Example
Imagine a small machine-learning project. Use Manual Tuning to decide what information is needed, what step happens next, and what result should be checked.
Coding example
// Manual Tuning
const records = [3, 5, 7, 9, 11];
const transform = value => ({ input: value, output: value * 2 + 1 });
const results = records.map(transform);
console.log(results);Code explanation
- The sample starts with a small list of inputs so every result can be checked manually.
- `transform()` represents the main operation for this topic in a deliberately simple form.
- `map()` applies the same rule consistently to every item and returns a new result array.
- Use this pattern to focus on input, transformation, and output before replacing the toy rule with a more advanced method.
Expected result: A transformed result is printed for each input value.
Practice exercise
Create a small real-world example for Manual Tuning. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.
29.3 Grid Search
Grid Search (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 29, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.
When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.
Example
A decision tree may be tested with several depth settings. Compare the validation results and keep the setting that performs best without overfitting.
Coding example
// Grid Search
const loss = x => (x - 4) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;
let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
value -= rate * derivative(value);
}
console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });Code explanation
- `loss()` gives a simple objective: values closer to the target produce a smaller error.
- `derivative()` estimates the slope by checking the loss just to the left and right of the current value.
- The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
- Printing both the final value and loss lets you confirm that the search moved toward a better solution.
Expected result: The value moves toward the target and the loss becomes smaller.
Practice exercise
Create a second example for Grid Search. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.
29.4 Random Search
Random Search (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 29, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.
When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.
Example
Imagine a small machine-learning project. Use Random Search to decide what information is needed, what step happens next, and what result should be checked.
Coding example
// Random Search
const loss = x => (x - 11) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;
let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
value -= rate * derivative(value);
}
console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });Code explanation
- `loss()` gives a simple objective: values closer to the target produce a smaller error.
- `derivative()` estimates the slope by checking the loss just to the left and right of the current value.
- The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
- Printing both the final value and loss lets you confirm that the search moved toward a better solution.
Expected result: The value moves toward the target and the loss becomes smaller.
Practice exercise
Create a small real-world example for Random Search. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.
29.5 Bayesian Optimization
Bayesian Optimization (reasoning that represents uncertainty with probability and updates beliefs when new evidence arrives). Within Chapter 29, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.
When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.
Example
Imagine a small machine-learning project. Use Bayesian Optimization to decide what information is needed, what step happens next, and what result should be checked.
Coding example
// Bayesian Optimization
const loss = x => (x - 9) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;
let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
value -= rate * derivative(value);
}
console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });Code explanation
- `loss()` gives a simple objective: values closer to the target produce a smaller error.
- `derivative()` estimates the slope by checking the loss just to the left and right of the current value.
- The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
- Printing both the final value and loss lets you confirm that the search moved toward a better solution.
Expected result: The value moves toward the target and the loss becomes smaller.
Practice exercise
Create a small real-world example for Bayesian Optimization. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.
29.6 Successive Halving
Successive Halving (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 29, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.
When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.
Example
Imagine a small machine-learning project. Use Successive Halving to decide what information is needed, what step happens next, and what result should be checked.
Coding example
// Successive Halving
const records = [3, 5, 7, 9, 11];
const transform = value => ({ input: value, output: value * 2 + 1 });
const results = records.map(transform);
console.log(results);Code explanation
- The sample starts with a small list of inputs so every result can be checked manually.
- `transform()` represents the main operation for this topic in a deliberately simple form.
- `map()` applies the same rule consistently to every item and returns a new result array.
- Use this pattern to focus on input, transformation, and output before replacing the toy rule with a more advanced method.
Expected result: A transformed result is printed for each input value.
Practice exercise
Create a small real-world example for Successive Halving. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.
29.7 Early Stopping
Early Stopping (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 29, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.
When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.
Example
Imagine a small machine-learning project. Use Early Stopping to decide what information is needed, what step happens next, and what result should be checked.
Coding example
// Early Stopping
const records = [3, 5, 7, 9, 11];
const transform = value => ({ input: value, output: value * 2 + 1 });
const results = records.map(transform);
console.log(results);Code explanation
- The sample starts with a small list of inputs so every result can be checked manually.
- `transform()` represents the main operation for this topic in a deliberately simple form.
- `map()` applies the same rule consistently to every item and returns a new result array.
- Use this pattern to focus on input, transformation, and output before replacing the toy rule with a more advanced method.
Expected result: A transformed result is printed for each input value.
Practice exercise
Create a small real-world example for Early Stopping. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.
29.8 Search Spaces
Search Spaces (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 29, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.
When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.
Example
Imagine a small machine-learning project. Use Search Spaces to decide what information is needed, what step happens next, and what result should be checked.
Coding example
// Search Spaces
const loss = x => (x - 3) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;
let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
value -= rate * derivative(value);
}
console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });Code explanation
- `loss()` gives a simple objective: values closer to the target produce a smaller error.
- `derivative()` estimates the slope by checking the loss just to the left and right of the current value.
- The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
- Printing both the final value and loss lets you confirm that the search moved toward a better solution.
Expected result: The value moves toward the target and the loss becomes smaller.
Practice exercise
Create a small real-world example for Search Spaces. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.
29.9 Optimization Metrics
Optimization Metrics (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 29, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.
When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.
Example
Imagine a small machine-learning project. Use Optimization Metrics to decide what information is needed, what step happens next, and what result should be checked.
Coding example
// Optimization Metrics
const loss = x => (x - 10) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;
let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
value -= rate * derivative(value);
}
console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });Code explanation
- `loss()` gives a simple objective: values closer to the target produce a smaller error.
- `derivative()` estimates the slope by checking the loss just to the left and right of the current value.
- The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
- Printing both the final value and loss lets you confirm that the search moved toward a better solution.
Expected result: The value moves toward the target and the loss becomes smaller.
Practice exercise
Create a small real-world example for Optimization Metrics. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.
29.10 Automated Tuning
Automated Tuning (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 29, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.
When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.
Example
Imagine a small machine-learning project. Use Automated Tuning to decide what information is needed, what step happens next, and what result should be checked.
Coding example
// Automated Tuning
const records = [3, 5, 7, 9, 11];
const transform = value => ({ input: value, output: value * 2 + 1 });
const results = records.map(transform);
console.log(results);Code explanation
- The sample starts with a small list of inputs so every result can be checked manually.
- `transform()` represents the main operation for this topic in a deliberately simple form.
- `map()` applies the same rule consistently to every item and returns a new result array.
- Use this pattern to focus on input, transformation, and output before replacing the toy rule with a more advanced method.
Expected result: A transformed result is printed for each input value.
Practice exercise
Create a small real-world example for Automated Tuning. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.
Chapter 29 Review Questions and Answers
Q1. What is Parameters vs Hyperparameters?
Answer: Parameters vs Hyperparameters is a model or training setting chosen outside the learned parameters. In this chapter, focus on the input, the method or decision, and the result that should be checked.
Q2. What is Manual Tuning?
Answer: Manual Tuning is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.
Q3. What is Grid Search?
Answer: Grid Search is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.
Q4. What is Random Search?
Answer: Random Search is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.
Q5. What is Bayesian Optimization?
Answer: Bayesian Optimization is reasoning that represents uncertainty with probability and updates beliefs when new evidence arrives. In this chapter, focus on the input, the method or decision, and the result that should be checked.
Q6. What is Successive Halving?
Answer: Successive Halving is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.
Q7. What is Early Stopping?
Answer: Early Stopping is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.
Q8. What is Search Spaces?
Answer: Search Spaces is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.
Q9. What is Optimization Metrics?
Answer: Optimization Metrics is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.
Q10. What is Automated Tuning?
Answer: Automated Tuning is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.