A perceptron is the simplest neural unit. It multiplies each input by a weight, sums the results, adds a bias, then passes that total through a step threshold. Output is one or zero. Training nudges weights whenever a prediction is wrong, moving the boundary toward the mistaken example.

What a single layer can do is draw one straight boundary through the input space. That handles linearly separable problems like AND and OR. It cannot solve XOR, because no single line separates those four points.

This limit is why depth exists. Stacking units with a hidden layer between them lets the network bend the boundary into arbitrary shapes. The cost is that the simple perceptron update no longer works, so you need gradient-based training instead.

Every network is organized into three kinds of layer, each with a fixed job. The input layer is not really computing anything. It simply holds your features, one unit per number you feed in - pixel values, sensor readings, or an encoded row.

Hidden layers sit in the middle and do the actual work. Each one takes the previous layer's outputs, applies weights, and produces a new representation. Early hidden layers pick up simple patterns. Later ones combine those into higher-level features.

The output layer is shaped by the task. One unit for a regression target or binary score, ten units for ten classes. Get that shape wrong and the loss function will not match your labels, which is a common early bug.

An activation function is the small transform a neuron applies to its weighted sum before passing the value on. The neuron first computes inputs times weights plus bias. That number then goes through the activation, and the result becomes the neuron's output.

Two things happen there. The function bends the straight-line arithmetic, so stacked layers can represent curved decision boundaries. It also controls the range of the signal - squashed between zero and one, or clipped at zero, depending on which function you pick.

The choice has a real training cost. Because backpropagation multiplies the activation's slope at every layer, a function that flattens out will shrink gradients as they travel backwards. The rectified linear unit (ReLU) stays popular partly because its slope is exactly one wherever the input is positive.

Backpropagation is how a network works out, for every weight, whether nudging it up or down would reduce the loss. It runs after a forward pass. The forward pass computes predictions and stores each layer's intermediate values along the way.

Then the loss gradient flows backwards, layer by layer. At each step the chain rule from calculus combines the gradient arriving from above with the local derivative of that layer. Multiply them and you get the gradient for that layer's weights, plus the signal to hand to the layer below.

The practical cost is memory. All those stored activations sit in graphics processing unit (GPU) memory until the backward pass consumes them. That is why a batch that fits during inference can still blow up during training.

Gradient descent finds good weights by repeatedly stepping downhill on the loss surface. Picture the loss as a landscape where every position is one setting of the weights. The gradient points in the direction of steepest increase, so you move the opposite way.

One iteration is three moves: run data through the network, measure the loss, then subtract a fraction of each gradient from its weight. Repeat over many passes and the weights drift toward a region where the loss stops improving.

Nothing about this guarantees the lowest possible point. You only ever see the slope directly under your feet, never the whole map. In practice that is acceptable, because a good-enough valley found in hours beats a perfect one you never reach.

The learning rate sets how far each weight moves along its gradient on every update. The gradient gives the direction. The learning rate gives the step size. It is usually the first thing to tune, because a wrong value ruins a run no matter how good the architecture is.

Set it too high and the steps overshoot the valley. Loss jumps around, or shoots to not-a-number as weights blow up. Set it too low and training crawls. You burn hours of compute and may stall on a flat stretch before reaching anything useful.

Most teams start around 0.001 with Adam, watch the first few hundred steps, then adjust by factors of ten. Decaying the rate later in training helps the weights settle instead of bouncing near the bottom.

Hyperparameters are the knobs you set before training starts, as opposed to weights, which the training learns. A handful matter far more than the rest.

• Learning rate: the size of each weight update, and by far the most sensitive.
• Batch size: examples per update, which also drives memory use and throughput.
• Epochs: how many passes over the data, usually settled by early stopping.
• Depth and width: layers and units per layer, which set the model's capacity.
• Regularization strength: dropout probability and weight decay, trading fit against generalization.
• Optimizer: plain descent, momentum or Adam, each with its own sensible defaults.

Tune them roughly in that order. Sweeping everything at once wastes compute, because learning rate alone can swamp the effect of every other choice.