Gradient descent is the most widely used optimization algorithm in machine learning — and surprisingly, the idea behind it is very simple. This essay is the preface for what comes next: without understanding gradient descent, you won't understand the majority of machine learning.
If you were five years old, here is how I'd explain it. Imagine you're trying to adjust the volume on your TV. Too loud and the neighbours complain; too low and you can't enjoy the show. Somewhere in between there's a volume that's just right. The change in volume as you twist the knob — how much louder or quieter it gets per click — is the derivative: in this analogy the volume difference is \(\Delta\), and the rate at which loudness changes per click is the slope. Twisting the knob and measuring the result is differentiation.
Now, the signal you get back from the environment — a complaint from the neighbours, or not being able to hear the show — is the error. The closer you are to the perfect volume, the smaller the error. Repeatedly measuring that error and nudging the knob in the direction that shrinks it, a little at a time, until no one complains — that process is gradient descent.
More formally, gradient descent minimizes a cost function \(J(\theta)\) by updating parameters in the direction of the negative gradient:
Here \(\theta\) is the knob (the model's parameters), \(\eta\) is the learning rate (the size of each nudge), and \(\nabla J\) is the gradient — the slope of the error pointing uphill, so we subtract it to go down. The four sections below build this idea up piece by piece: regression gives the slope, differentiation makes it instantaneous, partial derivatives extend it to many variables, and gradient descent puts it all to work.
When you have two distinct points, like (1, 10) and (10, 1), regression finds the average rate of change between them.
This tells us the macro-level trend: on average, for every 1 step to the right, we drop 1 step down.
A derivative is just a regression line where the two points are infinitely close together. Slide Point B closer to Point A to see the average slope (secant/regression) become the instantaneous slope (tangent/derivative).
Current Math:
Point A: (2.00, 4.00)
Point B: (6.00, 36.00)
This is currently an average rate of change (Regression).
Graph of \( f(x) = x^2 \)
In a multiple regression model, we predict an outcome based on multiple variables. The coefficients in the regression model are literally partial derivatives. They tell you the rate of change for one variable, assuming you hold the others completely frozen.
House Price (\(y\)) = 150(\(x_1\)) − 2000(\(x_2\)) + 50000
\(x_1\) = Size (sq ft) | \(x_2\) = Age (years)
Size: 1,500 sq ft
Age: 10 years
$255,000
Adding 1 sq ft raises price by exactly $150 (if age doesn't change).
Adding 1 year of age drops price by exactly $2,000 (if size doesn't change).
Slope = 150 (Positive)
Slope = −2000 (Negative)
In Machine Learning, we want to find the lowest possible error. We use the derivative (slope) of the error curve to know which direction is "downhill". By taking small steps in the opposite direction of the slope, we eventually reach the bottom!
Current Parameter (x): 8.00
Current Slope (dy/dx): 16.00
The slope is positive, so we step left (negative) to minimize error.
The multiple-regression model in Section 3 — a weighted sum of inputs plus a bias — is, quite literally, a single artificial neuron with its activation function removed. A neuron computes the same thing: \(z = w_1x_1 + w_2x_2 + \dots + w_nx_n + b\). Strip the activation away and what's left is plain linear regression. In other words, the one "predicted price" neuron above is the atom that every neural network is built from.
An activation function is just a small filter applied to that sum, deciding whether the neuron switches on or off (or by how much). It's a simple, bounded, non-linear function — ReLU passes positive values and zeroes out negative ones; sigmoid squashes everything into (0, 1). That single non-linearity is what separates a neural network from a linear regression. Stack layers of these filtered neurons together and you get the expressive power behind deep learning.
Training such a network means asking, for every weight: "if I nudge this weight, how much does the final error change?" Answering that through many stacked layers is what backpropagation does — and it is nothing more than the chain rule of calculus, applied from the output back to the input. The same derivative idea from Section 2, just chained through each layer so each weight learns its share of the blame for the error. The essays that follow build on exactly this foundation.