Piero Savastano
Machine Learning - Understanding K-Nearest Neighbors

Machine Learning - Understanding K-Nearest Neighbors

May 30, 2025
5 min read
Table of Contents
index

Let’s see how the simplest machine learning algorithm works. It’s usually the first one people approach, and it’s based on the simplest intuition. There’s no gradient descent here, no splitting the input space into sectors: you simply look at who’s in the neighborhood.

The algorithm is called KNN, k-nearest neighbors. Despite the ugly name, it’s really easy to understand. K is a number we choose: 3, 5, 7, 2. Nearest neighbors means: go look in the input space - which we now know what it is - for the closest examples, and take the prediction from those.

Rugby or basketball?

Let’s take an example with athletes, with two features and one target. X1 is how tall the athlete is, X2 is how much they weigh, and Y is whether they play rugby or basketball. So my record has numbers for height and weight, and the output is 1 if they play basketball, 0 if they play rugby.

As you know, this translates into a geometric space. This time, instead of drawing Y as a vertical axis, I’ll use symbols: a check mark for 1, an X for 0. That way we can see three dimensions on the board, with X1 on one axis and X2 on the other. Again, every row of the dataset is a point in the input space.

Basketball players will probably all be in one area, because they’re tall and weigh a bit less. Rugby players are stockier and heavier, so they’ll be in another area. Of course there will be exceptions.

Predicting with K = 3

Let’s pick K = 3. What does that mean? For the next athlete who comes in, I’ll know how tall they are and how much they weigh, so I know where they sit in the space. That’s the example I need a prediction for.

K-nearest neighbors means: find the K points closest to this one in the input space, and average their targets. If I choose 3 as K, I look at the three closest dots. Say they turn out to be two rugby players and one basketball player. Then my prediction is that, with a probability of 66.6 recurring percent, this is a rugby player. Very easy.

If the Y axis weren’t 0/1 - a category - but a continuous number like house prices, I would simply average the prices of the three nearest houses in the input space.

How do I find the K nearest neighbors? Intuitively, with Pythagoras’ theorem, for example. It works in many dimensions too, with simple geometric distance measures. I go through all the examples, find the K closest ones, look at their Y - their target, their output - and average those outputs. Ultra simple. KNN.

Write it to learn, use a library in production

The pro is that it’s simple to understand and simple to use. Writing it from scratch takes very little time.

I’ll say this now and it holds for all the algorithms: writing them from scratch is a great exercise to understand them. But when you go to production, it’s better to learn a library like scikit-learn, TensorFlow or PyTorch. These libraries already contain the algorithms, implemented, tested, rock solid and very fast.

Writing them yourself helps you understand them, but don’t use your own code in production, because typically you won’t do a better job, in terms of correctness and efficiency, than tens if not hundreds of open source contributors have done over the last 5, 10, 15 years. So: write the algorithms to learn, fine. Use algorithms you wrote yourself for production work? Careful, that’s very dangerous.

Pros and cons

The pro of k-nearest neighbors is that it’s very simple to understand and very simple to write, if you want to write it. It’s implemented in all the major machine learning libraries, including of course scikit-learn in Python.

The con is that it’s slow, because every time I get a new record I have to compare it with all the records I’ve stored. If those are in the thousands, fine; tens of thousands, also fine. Hundreds of thousands or millions? Clearly this approach doesn’t scale, so I need machine learning algorithms that can give me a prediction without scanning all the data.

So: pro, it’s simple. Con, it’s slow.