An activation function is the small transform a neuron applies to its weighted sum before passing the value on. The neuron first computes inputs times weights plus bias. That number then goes through the activation, and the result becomes the neuron's output.
Two things happen there. The function bends the straight-line arithmetic, so stacked layers can represent curved decision boundaries. It also controls the range of the signal - squashed between zero and one, or clipped at zero, depending on which function you pick.
The choice has a real training cost. Because backpropagation multiplies the activation's slope at every layer, a function that flattens out will shrink gradients as they travel backwards. The rectified linear unit (ReLU) stays popular partly because its slope is exactly one wherever the input is positive.
Rewriting in plainer words…
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
