Machine Learning Chapter 40: Training Neural Networks

What this chapter covers

This chapter contains 12 topics. Technical terms are followed by plain-language meanings in parentheses where they first appear. Code is included only when it naturally helps demonstrate the concept; architecture, workflow, governance, and comparison topics use practical scenarios instead.

40.1 Gradient Descent

Gradient Descent (a vector showing the direction and rate of fastest increase of a function). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

A model starts with a poor weight value. It measures how the error changes, moves the weight a small step in the direction that reduces error, and repeats until improvement slows.

Coding example

// Gradient Descent
const loss = x => (x - 6) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;

let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
  value -= rate * derivative(value);
}

console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });

Code explanation

`loss()` gives a simple objective: values closer to the target produce a smaller error.
`derivative()` estimates the slope by checking the loss just to the left and right of the current value.
The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
Printing both the final value and loss lets you confirm that the search moved toward a better solution.

Expected result: The value moves toward the target and the loss becomes smaller.

Practice exercise

Create a second example for Gradient Descent. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

40.2 Backpropagation

Backpropagation (the process of calculating how each neural-network parameter contributed to the error). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

Imagine a small machine-learning project. Use Backpropagation to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Backpropagation
const relu = x => Math.max(0, x);
const weights = [0.6, -0.2, 0.5];
const input = [2, 1, 3];
const bias = 0.1;
const weightedSum = input.reduce((s,x,i)=>s+x*weights[i], bias);
const output = relu(weightedSum);

console.log({ weightedSum: weightedSum.toFixed(2), output: output.toFixed(2) });

Code explanation

The input vector contains three features and the weight vector assigns one learned importance to each feature.
The weighted sum combines inputs, weights, and a bias into one number.
The ReLU activation keeps positive values and replaces negative values with zero.
This forward calculation is the basic building block that larger neural networks repeat many times.

Expected result: A weighted sum and activated neuron output are printed.

Practice exercise

Create a small real-world example for Backpropagation. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

40.3 Chain Rule

Chain Rule (a practical concept used within neural networks and deep learning). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

Imagine a small machine-learning project. Use Chain Rule to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Chain Rule
const loss = x => (x - 11) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;

let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
  value -= rate * derivative(value);
}

console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });

Code explanation

`loss()` gives a simple objective: values closer to the target produce a smaller error.
`derivative()` estimates the slope by checking the loss just to the left and right of the current value.
The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
Printing both the final value and loss lets you confirm that the search moved toward a better solution.

Expected result: The value moves toward the target and the loss becomes smaller.

Practice exercise

Create a small real-world example for Chain Rule. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

40.4 Learning Rates

Learning Rates (a practical concept used within neural networks and deep learning). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

Imagine a small real-world project where Learning Rates is the main idea. Identify the input information, the decision or transformation that occurs, and the result you would inspect to decide whether the method is working correctly.

Coding example

// Learning Rates
const loss = x => (x - 9) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;

let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
  value -= rate * derivative(value);
}

console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });

Code explanation

`loss()` gives a simple objective: values closer to the target produce a smaller error.
`derivative()` estimates the slope by checking the loss just to the left and right of the current value.
The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
Printing both the final value and loss lets you confirm that the search moved toward a better solution.

Expected result: The value moves toward the target and the loss becomes smaller.

Practice exercise

Create a second example for Learning Rates. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

40.5 Batch Gradient Descent

Batch Gradient Descent (a vector showing the direction and rate of fastest increase of a function). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

A model starts with a poor weight value. It measures how the error changes, moves the weight a small step in the direction that reduces error, and repeats until improvement slows.

Coding example

// Batch Gradient Descent
const loss = x => (x - 7) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;

let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
  value -= rate * derivative(value);
}

console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });

Code explanation

`loss()` gives a simple objective: values closer to the target produce a smaller error.
`derivative()` estimates the slope by checking the loss just to the left and right of the current value.
The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
Printing both the final value and loss lets you confirm that the search moved toward a better solution.

Expected result: The value moves toward the target and the loss becomes smaller.

Practice exercise

Create a second example for Batch Gradient Descent. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

40.6 Stochastic Gradient Descent

Stochastic Gradient Descent (a vector showing the direction and rate of fastest increase of a function). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

A model starts with a poor weight value. It measures how the error changes, moves the weight a small step in the direction that reduces error, and repeats until improvement slows.

Coding example

// Stochastic Gradient Descent
const loss = x => (x - 5) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;

let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
  value -= rate * derivative(value);
}

console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });

Code explanation

`loss()` gives a simple objective: values closer to the target produce a smaller error.
`derivative()` estimates the slope by checking the loss just to the left and right of the current value.
The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
Printing both the final value and loss lets you confirm that the search moved toward a better solution.

Expected result: The value moves toward the target and the loss becomes smaller.

Practice exercise

Create a second example for Stochastic Gradient Descent. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

40.7 Mini-Batches

Mini-Batches (a practical concept used within neural networks and deep learning). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

Imagine a small machine-learning project. Use Mini-Batches to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Mini-Batches
const records = [3, 5, 7, 9, 11];
const transform = value => ({ input: value, output: value * 2 + 1 });
const results = records.map(transform);

console.log(results);

Code explanation

The sample starts with a small list of inputs so every result can be checked manually.
`transform()` represents the main operation for this topic in a deliberately simple form.
`map()` applies the same rule consistently to every item and returns a new result array.
Use this pattern to focus on input, transformation, and output before replacing the toy rule with a more advanced method.

Expected result: A transformed result is printed for each input value.

Practice exercise

Create a small real-world example for Mini-Batches. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

40.8 Momentum

Momentum (a practical concept used within neural networks and deep learning). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

Imagine a small machine-learning project. Use Momentum to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Momentum
const loss = x => (x - 10) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;

let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
  value -= rate * derivative(value);
}

console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });

Code explanation

`loss()` gives a simple objective: values closer to the target produce a smaller error.
`derivative()` estimates the slope by checking the loss just to the left and right of the current value.
The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
Printing both the final value and loss lets you confirm that the search moved toward a better solution.

Expected result: The value moves toward the target and the loss becomes smaller.

Practice exercise

Create a small real-world example for Momentum. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

40.9 Adaptive Optimization

Adaptive Optimization (a practical concept used within neural networks and deep learning). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

Imagine a small machine-learning project. Use Adaptive Optimization to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Adaptive Optimization
const loss = x => (x - 8) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;

let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
  value -= rate * derivative(value);
}

console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });

Code explanation

`loss()` gives a simple objective: values closer to the target produce a smaller error.
`derivative()` estimates the slope by checking the loss just to the left and right of the current value.
The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
Printing both the final value and loss lets you confirm that the search moved toward a better solution.

Expected result: The value moves toward the target and the loss becomes smaller.

Practice exercise

Create a small real-world example for Adaptive Optimization. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

40.10 Weight Initialization

Weight Initialization (a practical concept used within neural networks and deep learning). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

Imagine a small real-world project where Weight Initialization is the main idea. Identify the input information, the decision or transformation that occurs, and the result you would inspect to decide whether the method is working correctly.

Coding example

// Weight Initialization
const records = [3, 5, 7, 9, 11];
const transform = value => ({ input: value, output: value * 2 + 1 });
const results = records.map(transform);

console.log(results);

Code explanation

The sample starts with a small list of inputs so every result can be checked manually.
`transform()` represents the main operation for this topic in a deliberately simple form.
`map()` applies the same rule consistently to every item and returns a new result array.
Use this pattern to focus on input, transformation, and output before replacing the toy rule with a more advanced method.

Expected result: A transformed result is printed for each input value.

Practice exercise

Create a second example for Weight Initialization. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

40.11 Vanishing Gradients

Vanishing Gradients (a vector showing the direction and rate of fastest increase of a function). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

Imagine a small machine-learning project. Use Vanishing Gradients to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Vanishing Gradients
const loss = x => (x - 4) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;

let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
  value -= rate * derivative(value);
}

console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });

Code explanation

`loss()` gives a simple objective: values closer to the target produce a smaller error.
`derivative()` estimates the slope by checking the loss just to the left and right of the current value.
The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
Printing both the final value and loss lets you confirm that the search moved toward a better solution.

Expected result: The value moves toward the target and the loss becomes smaller.

Practice exercise

Create a small real-world example for Vanishing Gradients. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

40.12 Exploding Gradients

Exploding Gradients (a vector showing the direction and rate of fastest increase of a function). Within Chapter 40, this topic connects directly to neural networks and deep learning. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

Neural-network behavior depends on data shape, parameter initialization, activation functions, optimization, regularization, and computational resources. Trace tensor shapes and loss values carefully, and verify that training performance also transfers to validation or test data.

Example

Imagine a small machine-learning project. Use Exploding Gradients to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Exploding Gradients
const loss = x => (x - 11) ** 2;
const derivative = x => (loss(x + 0.0001) - loss(x - 0.0001)) / 0.0002;

let value = 0;
const rate = 0.1;
for (let step = 0; step < 6; step++) {
  value -= rate * derivative(value);
}

console.log({ value: value.toFixed(3), loss: loss(value).toFixed(3) });

Code explanation

`loss()` gives a simple objective: values closer to the target produce a smaller error.
`derivative()` estimates the slope by checking the loss just to the left and right of the current value.
The loop repeatedly moves the value opposite the slope, which demonstrates the core idea behind gradient-based optimization.
Printing both the final value and loss lets you confirm that the search moved toward a better solution.

Expected result: The value moves toward the target and the loss becomes smaller.

Practice exercise

Create a small real-world example for Exploding Gradients. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

Chapter 40 Review Questions and Answers

Q1. What is Gradient Descent?

Answer: Gradient Descent is a vector showing the direction and rate of fastest increase of a function. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q2. What is Backpropagation?

Answer: Backpropagation is the process of calculating how each neural-network parameter contributed to the error. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q3. What is Chain Rule?

Answer: Chain Rule is a practical concept used within neural networks and deep learning. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q4. What is Learning Rates?

Answer: Learning Rates is a practical concept used within neural networks and deep learning. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q5. What is Batch Gradient Descent?

Answer: Batch Gradient Descent is a vector showing the direction and rate of fastest increase of a function. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q6. What is Stochastic Gradient Descent?

Answer: Stochastic Gradient Descent is a vector showing the direction and rate of fastest increase of a function. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q7. What is Mini-Batches?

Answer: Mini-Batches is a practical concept used within neural networks and deep learning. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q8. What is Momentum?

Answer: Momentum is a practical concept used within neural networks and deep learning. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q9. What is Adaptive Optimization?

Answer: Adaptive Optimization is a practical concept used within neural networks and deep learning. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q10. What is Weight Initialization?

Answer: Weight Initialization is a practical concept used within neural networks and deep learning. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q11. What is Vanishing Gradients?

Answer: Vanishing Gradients is a vector showing the direction and rate of fastest increase of a function. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q12. What is Exploding Gradients?

Answer: Exploding Gradients is a vector showing the direction and rate of fastest increase of a function. In this chapter, focus on the input, the method or decision, and the result that should be checked.