So, once we’ve framed the input space, what is it for? How do I adjust that line? What’s the intuition behind it? For that I refer to another geometric space, called the parameter space.
Parameters are guitar strings
Let’s take the simplest case, where I’ve chosen a linear structure: a function that’s a simple line. There’s a single feature, as in the previous example, so I have X1 and Y. If I want my model to be a line, I use an equation like this:
f(x) = x1 · w1 + c
I take the value of my feature and multiply it by some number I have to find during learning. The data is given, it’s fixed, I can’t change it. I have to use it in learning to discover these little numbers: the parameters. They’re like the strings of a guitar that I tighten or loosen until I hear the right sound.
Learning is about adjusting the Ws. C is also a parameter; you can see it as one extra W on top of the N features. And on the other side there’s my target: using this formula, I have to get as close as possible to what was observed.
The parameter space looks precisely at these Ws, the parameters I adjust to minimize the error, and at the error produced by each of them. So on one axis I have W1, and on the other the error given by the various values of the parameter.
Trying values by hand
Let’s see visually what’s happening. Imagine we have the usual examples. I don’t know which W minimizes the error, so I pick a W at random.
Say I start from a slightly negative W. If I draw the model, I get a line sloping the wrong way (for now let’s keep C fixed). What’s the error? I measure it, and it’s pretty high: these are all big errors. So for a negative W, the error is up here.
I try another W, high and positive. I draw a model that’s too steep. Here too I make a lot of errors, so the error is up there on the positive side.
I vary it again - there’s actually a smart criterion for varying it, but for now we’re feeling our way. I set it to 1, and get a model that makes a bit fewer errors but isn’t the best yet. The error is lower, somewhere around here.
Finally I find the right angle, the one that makes the fewest errors possible, and I’m at the bottom.
The error surface
There’s a function involved in both cases. In the input space, the model is a function that, given the input, gives you the output. In the parameter space, the function we’re talking about is the error function: given the parameters you’re using to control the model, what error does that configuration produce? So I get a curve shaped like a valley. This is called the error surface.
Here too, with more features, there are more dimensions. Each feature gets a parameter, then they’re all summed together. Instead of one parameter I have two, three, and the error surface has many dimensions too. For every value of all the parameters there’s an error, and every configuration of parameters corresponds to a different model in the input space.
Picture it as an animation: with one W, increasing W tilts the model upward, decreasing it tilts the model downward. With more dimensions, instead of one control I have two, so the model can tilt both this way and that way. These are all angles.
Gradient descent
So what does gradient descent do - the optimization algorithm at the base of linear regression, logistic regression and neural networks?
It starts with all the parameters random, so the model sits somewhere, all crooked, maybe making huge errors. Then, through gradient descent, I check the slope of the error curve at the point I’ve picked. I don’t know the whole shape of the surface - otherwise I could just tell you where the minimum is. But for the W I’ve chosen, I can see how much error I make, and by varying it slightly, using the derivative, I can tell you the slope of the error surface there.
What does that mean? If I’m at a W where the error surface has a negative slope, I increase W, so I move to the right. The next time I measure the error for the new W, it has gone down. The slope is still negative, so I keep increasing W until I reach the bottom. That’s gradient descent. The gradient is nothing more than the derivative in many dimensions.
When I get to the bottom, the derivative of the error surface at that point is zero: I’ve found the minimum, and I stop.
If my W starts out too high compared to the minimum, I check the slope there, it’s positive, so I decrease W. Next time, having decreased W, I’ll have reduced the error; I keep decreasing W until I reach the bottom. In both cases, by following the slope of the error surface with a local check, I manage to walk downhill and find one of the lowest points of the error surface. And that W value corresponds, in the input space, to the model that best interpolates the dots, where the error is lowest.
For mathematical details: you start from an error function, take the derivative of that error function, and you get exactly the rule that tells you mathematically how to vary the Ws based on the slope of the error surface. For now we’ll leave the optimization part at that.
Local minima
With linear models, there’s guaranteed to be a single minimum. With non-linear models - curvy ones - there can be many. Neural networks exist precisely for that need: not to be limited to lines. And there, if you find a minimum, it’s not guaranteed you’ve hit the lowest one. Those are called local minima, but that’s another story.
Input space, parameter space
What matters, the intuition to take away, is input space and parameter space. Learning means finding the right combination of parameters that minimizes the error, because the model depends both on the data and on the parameters. The data is fixed, I can’t change it. The parameters are my guitar strings, the way I adjust the predictions I make.
Every configuration of parameters corresponds to an error. I find the configuration that brings the error as low as possible. In the input space, that means finding the model that best interpolates the points. In the parameter space, it means finding the lowest point of the error surface.