Train your first neural classifier
Optimize a sigmoid neuron using binary cross-entropy.
- Train a one-neuron classifier
- Track cross-entropy loss
- Evaluate inputs beyond the training rows
Match the objective to the output
A sigmoid neuron produces a score between zero and one. Binary cross-entropy penalizes a low score for a positive target and a high score for a negative target. With sigmoid and this loss, the derivative with respect to the pre-activation simplifies to prediction minus target.
A single neuron describes a linear decision boundary in its input space. It can separate a simple one-dimensional threshold, but cannot solve every pattern. More optimization steps cannot compensate for an unsuitable model family.
Keep training observable
Use a small learning rate, average gradients over the batch, and inspect the loss. Clip probabilities only for logarithm evaluation to avoid taking log of zero; do not silently alter the gradient formula.
The toy task here labels negative values as zero and positive values as one. New examples from that same simple rule should behave sensibly, but the result does not establish that the model can solve unrelated classification tasks.
A small experiment you can run.
The symmetry keeps the bias near zero. The ambiguous midpoint gets a score near 0.5 instead of a justified confident label.
import math
xs, ys = [-2., -1., 1., 2.], [0., 0., 1., 1.]
w, b, rate = 0., 0., 0.2
for _ in range(300):
ps = [1/(1+math.exp(-(w*x+b))) for x in xs]
dw = sum((p-y)*x for p, y, x in zip(ps, ys, xs))/len(xs)
db = sum(p-y for p, y in zip(ps, ys))/len(xs)
w, b = w-rate*dw, b-rate*db
for x in [-1.5, 0., 1.5]:
print(x, round(1/(1+math.exp(-(w*x+b))), 3))
Save the file, open your terminal in that folder, and run python train-a-neural-classifier.py. Use python3 or py if required by your installation. Setup guide
The original negative test input gets a low score, the positive input a high score, and zero a score near 0.5.
Inspect an unlearnable label pattern.
- Replace inputs with [-2, -1, 1, 2] and labels with [1, 0, 0, 1].
- Train the same single neuron.
- Explain why one threshold cannot separate both outer points from both inner points.
Compare with a suggested solution
The required positive region has two separated ends. A one-dimensional linear threshold creates only one boundary. Add suitable nonlinear features or a hidden layer, then evaluate instead of just increasing training steps.
One idea to take with you.
Make it part of your progress.
Finish the practice and answer the knowledge check to mark this lesson complete.
Go deeper with primary documentation
Optional references for further study. This lesson and its examples were written for Artificials.
PyTorch: automatic differentiation