Piero Savastano
Machine Learning - Multiclass Classifier

Machine Learning - Multiclass Classifier

May 30, 2025
4 min read
Table of Contents
index

Let’s imagine our Y is politics, sports and crime. The inputs are texts, and as Y we have sports, politics, crime, crime, politics, sports, and so on.

Don’t encode categories as integers

It would be a serious mistake to represent these categories as integers. A very serious mistake. Don’t worry if you thought of it, because it’s a very common first idea - I had it, my students have it, everybody has it.

What’s the problem with saying, for example, 0 is sports, 1 is crime, 2 is politics, and so on? This way of representing the target is dangerous, because we’re implicitly saying that politics is double crime, or that sports is crime minus 1. Doing operations on these numbers makes no sense at all. And when doing operations on your numbers makes no sense, it means you’ve got the encoding wrong.

One-hot encoding the target

The best encoding is the same one we used in the YouTube course to vectorize categorical variables and texts. If you remember, to vectorize categories we used a procedure called one-hot encoding, and we have to do the same thing here, but on the Y, on the target.

The trick is to build a separate output axis - a separate Y - for each level of the categorical variable. One Y for sports, one Y for politics and one Y for crime. I use three different axes because my output variable has three different values, and on each of these axes I apply exactly the same reasoning I applied earlier to Watzlawick the chicken’s corn, where the output was binary: happy or sad.

So the sports Y tells me whether it’s a sports story or not. The politics Y tells me whether it’s a politics story or not. The crime Y tells me whether it’s a crime story or not. A sports story becomes 1 on the sports axis and 0 on the others; a crime story becomes 1 on the crime axis and 0 elsewhere.

I’ve broken down the problem of having multiple categories for a single target variable into three separate problems that I look at individually, and on each of them I’ll draw an S, a sigmoid. In short, I’m training three different classifiers: one recognizes sports vs not sports, another politics vs not politics, the third crime vs not crime.

Softmax

I run them at the same time and they give me numbers as output. To these numbers I apply a particular function called softmax, which makes the outputs of the three S curves sum to 1. This gives me a probability distribution over my estimates.

Let’s make it clearer. I’ll have my estimated sports Y, my estimated politics Y and my estimated crime Y. Thanks to the three sigmoids and the softmax, the output becomes something like 0.1, 0.2, 0.7. The three sum to 1, so 100% of the possibilities. Looking at them in detail, I’ll say membership in sports is 10%, politics 20%, crime 70%, and I’ll take crime as the final answer.

Another output might be 0.5, 0.0, 0.5: a tie between sports and crime. Another might be 0.8, 0.15, 0.05: an 80% probability estimate that the story is about sports.

Back to the binary case

So I’ve simply decomposed this multiclass problem - a multinomial target variable, a category with many levels - and brought it back to the base case of the binary classifier we saw before, the one that outputs 0 and 1.

Then, with the softmax operation, I make the outputs of the different sigmoids - each estimating membership in the specific category it was fitted for - sum to one, and I interpret these little numbers that add up to one as a probability distribution of membership in each category.