Classify by finding neighbors
Build an interpretable classifier and understand its sensitivity to scale.
- Compute Euclidean distance
- Predict with nearby labeled examples
- Choose k using validation data
Learning can mean remembering
A nearest-neighbor classifier stores labeled examples and predicts from the closest ones. With k = 1 it copies the nearest label. With a larger k, a majority vote smooths isolated noisy examples but may blur small local patterns.
Distance is a modeling choice. If one coordinate is measured in meters and another in millimeters, the second can dominate. Scale numeric features using training statistics and avoid treating arbitrary category IDs as meaningful distances.
Make tie behavior explicit
Equal vote counts and equal distances can occur. A deterministic tie rule makes repeated runs easier to compare. The example uses label ordering after the vote count; other applications might use distance-weighted voting or send ties for review.
Select k on validation data, then evaluate once on a final test set. Retrieval cost grows with the number of stored examples, so a simple implementation is useful for learning but may require indexing at larger scale.
A small experiment you can run.
These two synthetic clusters are deliberately easy to separate. The vote rule and distance calculation are visible instead of hidden behind a library.
from collections import Counter
from math import dist
train = [((0., 0.), "a"), ((0., 1.), "a"), ((1., 0.), "a"),
((4., 4.), "b"), ((4., 5.), "b"), ((5., 4.), "b")]
def predict(point, k=3):
if not 1 <= k <= len(train):
raise ValueError("k is outside the training range")
nearest = sorted(train, key=lambda row: dist(point, row[0]))[:k]
votes = Counter(label for _, label in nearest)
return sorted(votes, key=lambda label: (-votes[label], label))[0]
print(predict((0.2, 0.3)), predict((4.2, 4.1)))
Save the file, open your terminal in that folder, and run python nearest-neighbor-classification.py. Use python3 or py if required by your installation. Setup guide
The initial predictions are a and b.
Test sensitivity to units.
- Multiply the second coordinate of all points and queries by 100.
- Find a query whose nearest neighbors change.
- Explain how standardization could restore a balanced comparison.
Compare with a suggested solution
Changing units alters distances unless the feature scaling is adjusted. Fit a mean and standard deviation on each training coordinate, then reuse them for every query. Scale can encode an intentional importance weight, but should not change accidentally.
One idea to take with you.
Make it part of your progress.
Finish the practice and answer the knowledge check to mark this lesson complete.
Go deeper with primary documentation
Optional references for further study. This lesson and its examples were written for Artificials.
scikit-learn: model evaluationscikit-learn: common pitfalls