A neuron is a small computation
Build a weighted sum and see how a nonlinearity changes it.
- Compute a weighted sum plus bias
- Compare ReLU and sigmoid
- Understand why nonlinearity matters
Weights describe influence
A basic neuron multiplies each input by a weight, adds the products, and adds a bias. A positive weight increases the pre-activation as its input increases; a negative weight decreases it. The bias shifts the response even when inputs are zero.
This is a mathematical building block, not a complete simulation of a biological neuron. A network connects many such blocks and learns weights by optimizing an objective.
Nonlinearity adds expressive power
A stack of purely linear transformations is still a linear transformation. Nonlinear activations allow a network to represent more complex patterns. ReLU returns zero for negative inputs and the input itself for positive ones. Sigmoid maps real numbers into the interval between zero and one.
Sigmoid can saturate near zero or one, producing small gradients. ReLU can have zero gradient on negative inputs. Activation choice affects optimization, but no activation alone makes a network accurate or its output calibrated.
One neuron. Three things you control.
A single input passes through a weight and bias, then a sigmoid activation. Adjust the inputs to see the actual calculation.
A small experiment you can run.
The inputs and weights are fixed for inspection. The interactive neuron lab lets you vary them before the training lesson introduces parameter updates.
import math
inputs, weights, bias = [0.5, 0.8], [1.2, -0.7], 0.1
z = sum(x*w for x, w in zip(inputs, weights)) + bias
relu = max(0., z)
sigmoid = 1/(1+math.exp(-z))
print("Weighted sum:", round(z, 3))
print("ReLU:", round(relu, 3), "Sigmoid:", round(sigmoid, 3))
Save the file, open your terminal in that folder, and run python neurons-and-activations.py. Use python3 or py if required by your installation. Setup guide
The original weighted sum is 0.14 and sigmoid is approximately 0.535.
Change one influence at a time.
- Set the second weight to +0.7.
- Increase the second input from 0.8 to 1.0.
- Predict the direction of change before running the code.
Compare with a suggested solution
With a positive second weight, increasing the second input increases z. With the original negative weight it decreases z. Both activations are monotone, although ReLU stays at zero while z remains negative.
One idea to take with you.
Make it part of your progress.
Finish the practice and answer the knowledge check to mark this lesson complete.
Go deeper with primary documentation
Optional references for further study. This lesson and its examples were written for Artificials.
PyTorch: automatic differentiation