Piero Savastano
Machine Learning - Regressor vs Classifier

Machine Learning - Regressor vs Classifier

May 30, 2025
6 min read
Table of Contents
index

Now that we’ve framed the records of our dataset as points in the input space, let’s make a distinction about the output of machine learning: the difference between a regression model and a classification model, between a regressor and a classifier.

Numbers vs categories

We’re talking about Y, the output of the machine learning model. A regressor is a model that outputs a number, while a classifier is a model that outputs a category.

A quick example with tweets: you’ve surely heard of sentiment analysis. If we build a model that takes a tweet as input and tells you whether it’s positive or negative, that’s a classifier. So is a model that tells you whether someone will repay their mortgage or not, or one that gives you the category of a news item: crime, politics, sports. A, B, C. Blue, green, yellow. White, black. Male, female. These are categories, nominal variables, and classifiers output a nominal variable.

A regressor outputs a variable that is an actual number, a numerical estimate. In sentiment analysis I could predict sentiment on a numerical range between -1 and 1, where -1 is negative, 1 is positive, with everything in between. I might want to predict how much a product will cost, or how long a person will take to pay off their mortgage.

So, based on the output we have available, we decide whether to use a classifier or a regressor.

The regressor, geometrically

Let’s look at the geometric difference, again in our input space and again with one feature and one target - for simplicity. In practice there will be tens, hundreds, thousands of features, so always bring your mind back to the real situation of a space with many dimensions.

Our feature X1 is how much corn Watzlawick the chicken ate at lunch. On Y, for the regressor, we have a numeric variable: how happy he is, measured by how wide his smile is. My dataset has rows of X1 and Y: he ate 100 kernels and his happiness is 10; he ate 50 kernels and his happiness is 7; and so on. Notice that Y is a numeric variable.

I plot the points. For simplicity, let’s say it’s a linear, intuitive relationship. Given the feature, my model has to give me an estimate of Y. Imagine I find this model, F of X, which tries to approximate Y. Its output is a numerical estimate: when the chicken has eaten, say, 72 kernels, I compute the value of this function - this line - at X1 = 72, and it tells me the chicken’s happiness is 80. I made that number up, but it’s a numerical estimate.

The classifier, geometrically

The classifier works differently. Let’s look at the dataset: we still have how much corn he ate, and the target is now a variable like sad/happy, true/false, 0/1, positive/negative. The chicken ate 20 kernels: sad. He ate 80: happy. He ate 55: happy. And so on.

This is the simplest case, a variable with two levels, sad and happy, which I can represent as 0 and 1, a binary variable. Plotting the dots, I get a situation where below a certain amount of corn the chicken is sad, 0, and above a certain amount he becomes happy, 1.

Here there’s a double step compared to the regressor. A line isn’t enough: I’ll never find a line that interpolates these dots. Instead I use a particular shape called a sigmoid, an S, which we’ll look at more closely in the video on logistic regression.

The S is very useful for this purpose, because going toward infinity on one side its minimum is always 0, and going toward infinity on the other its maximum is always 1. It’s always squeezed between 0 and 1, with a sensitive zone of continuous values in the middle.

Once I’ve fitted the S to the dots, I check: the chicken ate 100 kernels, the sigmoid says 1, happy. The chicken ate very few kernels, the sigmoid says 0, sad.

The more interesting case: the chicken ate 55 kernels. I check the sigmoid’s value and get 0.6. How do I interpret 0.6? You may have guessed already: the chicken is 60% happy. I interpret this band of continuous numbers between 0 and 1 as a probability estimate.

The decision boundary

There’s a further step, as I said, because I always want to give my prediction as a category. Even if I have a useful probability estimate - 60% happy - I want a final decision. So I set a threshold, which intuitively can be 0.5, though there are ways to find better ones.

When the sigmoid’s output is above 0.5, I say the chicken is happy. When it’s below, I say he’s sad. This line is called the decision boundary: the whole zone above it belongs to the “happy” category, and the whole zone below it gives the estimate “sad”. Very, very useful.

So, as you’ve seen, there’s a double step. The regressor is simpler: you draw the model and take its output as the estimate. The classifier needs the extra step of squashing the estimate into a probability, then deciding which number to use as the yes/no threshold.

Now, the question is: what happens when the target has more than two levels? I give you a news item, tell me whether it’s politics, sports or crime. I show you an image, tell me whether it’s yellow, black, blue or red. There’s a trick for that, which we’ll see next.