Supervised Learning and Neural Networks: Classifiers

A learning program receives examples of the behavior it should learn. In supervised learning, each example pairs an input with its correct output, called a label. The learned rule must also work on inputs that were absent during training.

Identify the task and its labels

Write a training set as $({𝐱}_1,y_1),\dots ,({𝐱}_N,y_N)$. The input describes one case; the label supplies the answer for that case. An unknown target function describes the underlying relationship. With noise-free labels, $y=f(𝐱)$. The program chooses a hypothesis $h$ to approximate that relationship.

For a spam filter, one case is an email. Its input can describe whether it contains "free," how many links it has, and whether its sender is a contact. Its label is spam or ham, meaning wanted email. A training set might contain 5,000 labeled emails; the finished filter is evaluated on 1,000 different labeled emails.

The output defines the task. "Will more than 5 mm of rain fall?" asks for a class; "How many millimeters will fall?" asks for a number. An integer output can still be a class: digit 7 names a category, not seven times digit 1.

Supervised learning uses labeled examples. Unsupervised learning looks for structure without labels, such as customer clusters. Reinforcement learning uses rewards following an agent's actions. Creating reliable labels can require substantial human work.

Choose a family of possible rules

The hypothesis space is the family the program can search, such as linear classifiers using specified features. Learned parameters, including weights or conditional probabilities, select one member of that family. A target outside the family cannot be represented exactly, even with unlimited data.

A hyperparameter controls the learning setup: examples include a learning rate or the number of hidden units. Training fits parameters. Comparisons on validation data help choose hyperparameters. More flexible families capture more patterns but also have more ways to fit noise.

Judge predictions on new cases

Training error measures fit to examples used for learning. A rule that memorizes every training email can have zero training error and still fail on new messages. Use a validation set for model choices and keep a separate test set for the final assessment.

For a classifier, error rate is the number of wrong predictions divided by the number tested. Three errors among 40 emails give

The connection to future performance depends on the sampling setting. Independent and identically distributed, or i.i.d., examples are independent draws from the same distribution. If future spam uses very different wording, the old test set no longer represents that future stream. Report the held-out result for the population it actually represents.

Models work with a numerical description of each case. The choice and order of these features determine what a model can notice. A loss then tells learning how costly its predictions are.

Build a feature vector

A feature can be binary, a count, or a measured number. "Contains free" becomes 0 or 1, a link count might be 4, and floor area might be 85 m². Keep the same feature order for training and prediction. A 28 by 28 grayscale image supplies $28\times 28=784$ pixel features.

For a category without a numerical order, use one-hot encoding. For red, green, and blue, the vectors are $(1,0,0)$, $(0,1,0)$, and $(0,0,1)$. Codes 1, 2, and 3 would impose an artificial ordering and spacing on these colors.

Combine features into a score

A dot product multiplies corresponding weights and features, then adds:

For $𝐰=(2,-1,0.5)$ and $𝐱=(1,3,4)$, the score is

A positive weight raises the score when its feature increases; a negative weight lowers it. Weight magnitude describes sensitivity per feature unit, so comparing magnitudes across differently scaled features needs care.

The bias shifts the score independently of varying inputs. Include a constant first feature $x_0=1$ and let $w_0$ be its weight. Thus $𝐱=(1,x_1,\dots ,x_n)$ includes the bias feature; it is not an extra measured property. The model can then shift a decision boundary without changing the other weights.

Measure the cost of a prediction

A loss function compares a prediction with its label. Different losses express different costs:

For a true temperature of 10 °C and prediction of 7 °C, squared loss is 9 and absolute loss is 3. An error of 6 has squared loss 36, four times the loss for an error of 3. The average 0/1 loss is the classification error rate.

Empirical loss is average loss on the training sample. Predictions $(2,4,7)$ for labels $(3,4,5)$ give squared errors 1, 0, and 4, hence mean squared error $5\mathrm{/}3\approx 1.67$. Add the losses before dividing by the number of examples.

Generalization loss is expected loss on new examples from the same distribution. Minimizing empirical loss can fit accidental details of the sample. Irrelevant features give a learner more opportunities to do this; near-duplicate features can also count the same evidence twice in naive Bayes. Ockham's razor favors a simpler hypothesis when alternatives fit about equally well, because extra flexibility needs support from data.

Naive Bayes combines a class's starting probability with the evidence supplied by an input. Its simplifying assumption makes a large probability model learnable from relatively small samples.

Factor the evidence by class

The model assumes features are conditionally independent given the class. For a class variable and several features,

A Bayesian network shows the class as the parent of each feature. Independence is conditional: features can be strongly related overall because both depend on the class. For 100 binary features, a full joint feature table needs $2^{100}-1$ free probabilities per class. Naive Bayes needs only 100 probabilities per class, one for each feature's yes value.

Bayes' rule gives a class score

Choose the class with the largest score. The omitted evidence denominator is the same for all classes. A score is not yet a posterior probability: to obtain a posterior, divide it by the sum of all class scores.

Work through a binary-feature prediction

Suppose the spam prior is 0.4 and the ham prior is 0.6. The probabilities for "free" are 0.5 in spam and 0.05 in ham. For "meeting," they are 0.1 and 0.4. The new email contains "free" but lacks "meeting."

1. For spam, absence of "meeting" contributes $1-0.1=0.9$. The score is $0.4(0.5)(0.9)=0.18$.
2. For ham, that absence contributes $1-0.4=0.6$. The score is $0.6(0.05)(0.6)=0.018$.
3. The scores sum to 0.198, so the spam posterior is $0.18\mathrm{/}0.198\approx 0.909$.

An absent binary feature supplies evidence. If both words occur, the scores become 0.02 and 0.012, and the spam posterior falls to 0.625. "Meeting" shifts the balance toward ham even though spam still wins.

Keep the direction of conditioning straight. $P(\text{free}\mid \text{spam})=0.5$ describes spam messages. The reverse probability, $P(\text{spam}\mid \text{free})$, also needs the class priors and the ham likelihood; here it is $0.2\mathrm{/}(0.2+0.03)\approx 0.870$.

Use log scores without changing the winner

Long products can underflow to zero in computer arithmetic. Two hundred factors of 0.01 give ${10}^{-400}$, smaller than the least positive double-precision number. Compare

instead. Logarithms preserve order and turn multiplication into addition. Here $\ln 0.18\approx -1.715$ beats $\ln 0.018\approx -4.017$; among negative scores, the less negative score is larger.

Naive Bayes is generative: it models the prior and how features arise within each class. Logistic regression is discriminative: it models the class probability from the input directly. Binary-feature naive Bayes also has a linear log-odds boundary. Correlated features, such as "Hong" and "Kong," can make its posterior too extreme by counting overlapping evidence. A class prediction can be right even when the assigned probability is too extreme.

Naive Bayes learns by counting. The important choices are which population belongs in a denominator and how much protection to give an outcome that has not appeared yet.

Estimate probabilities within the right group

The maximum likelihood estimate, or MLE, makes the observed data most probable. For a class prior, divide the class count by the total number of examples. For a conditional feature probability, restrict both counts to the class:

Consider 100 emails: 40 spam and 60 ham. "Free" appears in 20 spam messages and 3 ham messages; "meeting" appears in 4 spam and 24 ham messages. Thus the spam prior is 0.4. The "free" estimates are $20\mathrm{/}40=0.5$ for spam and $3\mathrm{/}60=0.05$ for ham. The "meeting" estimates are $4\mathrm{/}40=0.1$ and $24\mathrm{/}60=0.4$. Dividing a feature-and-class count by 100 would estimate a joint probability, not the requested conditional probability.

Repair zero counts

Suppose "prize" appears in 8 spam messages and no ham message. Its ham MLE is zero. Any new message containing it gets ham score zero, whatever its other features say. A finite sample's missing outcome has become a claim of impossibility.

Add-k smoothing gives every possible feature value a pseudo-count of $k$:

Here $\mathrm{\mid }F\mathrm{\mid }$ is the number of values of this feature. For a yes/no feature it is 2, not the number of words being modeled. With add-one smoothing,

Both outcomes receive a pseudo-count. In ham, the absent value therefore has probability $61\mathrm{/}62$. The two probabilities sum to 1. Adding only to the numerator would break this normalization.

Choose the strength of smoothing

The strength $k$ is a hyperparameter. At zero it gives MLE. As it becomes large, each probability approaches the uniform value $1\mathrm{/}\mathrm{\mid }F\mathrm{\mid }$ and class-specific evidence weakens. Smoothing trades some fit to the observed counts for more stable predictions on new cases.

Compare settings such as 0.1, 0.5, 1, 2, and 5 on validation data. Keep the test set separate for the final assessment. A large setting can underfit; a very small setting may leave rare features with unstable estimates.

The same pseudo-count matters less when real counts grow. Among 6,000 ham emails with 300 containing "free," the MLE is 0.05 and the add-one value is $301\mathrm{/}6002\approx 0.0502$. With many observations, the data dominate the pseudo-counts. Class priors are commonly left unsmoothed when every class is well represented. A single pass over labeled examples can collect the counts, which makes this training method fast.

A bag of words turns text into word counts. It gives a simple classifier evidence about vocabulary while discarding the order in which words occur.

Count tokens, not just documents

"The cat saw the dog" and "the dog saw the cat" have the same bag: "the" twice and each other word once. A token is an occurrence; the vocabulary is the set of distinct words represented by the model.

In multinomial naive Bayes, every word position is an independent draw from a class-specific word distribution. A word appearing three times contributes three factors:

The matching log score is $\log P(y)+{\sum }_w{n}_w\log P(w\mid y)$. The prior appears once per document. The word count multiplies its log probability.

Estimate word probabilities from token counts within a class. With smoothing,

where $N_y$ counts all training tokens in that class and $\mathrm{\mid }V\mathrm{\mid }$ counts vocabulary words. Suppose spam contains 200 tokens, including "cash" 9 times, and the vocabulary contains 1,000 words. MLE gives $9\mathrm{/}200=0.045$. Add-one gives $10\mathrm{/}1200\approx 0.0083$, while $k=0.1$ gives $9.1\mathrm{/}300\approx 0.0303$. A large vocabulary makes even one pseudo-count per word substantial.

Classify a short document

Sports and politics each have prior 0.5. Word probabilities are:

For "goal goal vote match," the sports score is

The politics score is $0.5(0.005{)}^2(0.04)(0.006)=3\times {10}^{-9}$. Sports wins by a factor of 50, and its normalized posterior is $50\mathrm{/}51\approx 0.98$. Omitting the second "goal" would discard part of the evidence. In practice use the log scores, about −15.71 and −19.62, to avoid tiny products.

Keep the text models distinct

A Bernoulli text model uses one yes/no feature per word and includes every absent word's factor. The multinomial model uses occurrence counts and has no separate absent-word factors. Its denominators count tokens, not documents. Mixing those choices produces neither model correctly.

A vocabulary word unseen in one class needs smoothing. A word outside the vocabulary has no modeled probability and is usually skipped. Alternatively, training can reserve an unknown-word token with its own class-specific probability.

The bag ignores context: "good" supplies the same evidence inside "not good" and "very good." Adding bigrams, or word pairs, can represent a phrase such as "not good." Near-duplicate or neighboring words can still produce overconfident probabilities. The model remains useful as a fast baseline for topics, spam, sentiment, or language identification.

A linear classifier makes a decision from a weighted sum. Geometry shows which patterns that sum can separate and why some tasks require new features.

Read the score and boundary

Include the constant bias feature first: $𝐱=(1,x_1,\dots ,x_n)$. Compute $z=𝐰\cdot 𝐱$ and predict $+1$ for $z\ge 0$ or $-1$ for $z<0$. A zero score belongs to the positive class under this convention.

With at least one nonzero feature weight, the decision boundary is $𝐰\cdot 𝐱=0$. For two varying features it is a line; for three it is a plane; in general it is a hyperplane. The non-bias weight vector is perpendicular to this boundary and points toward the positive side.

For $𝐰=(-3,1,1)$, the score is $-3+x_1+x_2$ and the boundary is $x_1+x_2=3$. Point $(2,2)$ scores 1, $(1,1)$ scores −1, and $(3,0)$ scores zero. The predictions are positive, negative, and positive.

The geometric distance is the score magnitude divided by the length of the non-bias weight vector:

For $(2,2)$ it is $1\mathrm{/}\sqrt{2}\approx 0.71$. Multiplying every weight by a positive constant changes the score but leaves both the distance and the predicted class unchanged. Leaving out the bias restricts boundaries to pass through the origin.

Decide whether a line is enough

Data are linearly separable when one weight vector classifies every point correctly. For AND, use $𝐰=(-1.5,1,1)$. Only $(1,1)$ has positive score; the other three binary inputs have negative scores.

XOR labels $(1,0)$ and $(0,1)$ positive, but $(0,0)$ and $(1,1)$ negative. No line can separate those diagonally opposite pairs. To see why, the negative origin requires $w_0<0$. The two positive points require $w_1+w_2\ge -2w_0$, while the negative point $(1,1)$ requires $w_1+w_2<-w_0$. Together they require $w_0>0$, a contradiction.

The feature representation can change separability. Add the feature $x_1{x}_2$ and use weights $(-0.5,1,1,-2)$. The four XOR inputs then score −0.5, 0.5, 0.5, and −0.5 in the order $(0,0),(1,0),(0,1),(1,1)$. The rule is linear in the expanded feature vector, even though it is nonlinear in the original inputs. A hidden layer can learn useful transformed features.

Extend scoring to several classes

Keep one vector per class and choose the largest dot product. For $𝐱=(1,2)$, vectors ${𝐰}_A=(1,1)$, ${𝐰}_B=(2,0)$, and ${𝐰}_C=(0,1)$ give scores 3, 2, and 2. Class A wins. These are comparative scores, not probabilities; they need not lie between zero and one or sum to one.

The perceptron learns a linear decision rule by correcting mistakes as they occur. It changes weights directly, without fitting probabilities or differentiating a smooth loss.

Update one example at a time

Use labels $y\in \{-1,+1\}$, include the bias feature, and start with zero weights. A score of zero predicts $+1$. For each example, predict using the current weights. Leave them unchanged if correct; otherwise apply

A positive example adds its feature vector. A negative example subtracts it. The score on that example changes by $y\mathrm{\parallel }𝐱{\mathrm{\parallel }}^2$, so it moves toward the correct sign. The bias feature makes the squared length positive. One update can still leave the example wrong and can disturb earlier predictions.

An epoch is a full pass through the training set. Stop after an entire mistake-free epoch or a chosen pass limit.

Trace two passes

Take the examples in this order, with the bias coordinate already included:

1. A scores zero and is wrongly positive. Subtract A to get $(-1,-1,-3)$.
2. B now scores $-1-3-3=-7$. Add B to get $(0,2,-2)$.
3. C scores −2 and D scores 4. Both are correct, so no update occurs.
4. On the next pass, A, B, C, and D score −4, 4, −2, and 4. No mistakes remain, so stop.

The learned boundary is $x_1=x_2$. Predictions must use the weights after all preceding updates, not the weights at the start of the epoch.

Understand convergence and its limits

The perceptron convergence theorem guarantees finitely many mistakes for linearly separable training data. If augmented inputs have lengths at most $R$ and a unit-length separator achieves positive signed margin $\gamma$, the mistake bound is $(R\mathrm{/}\gamma {)}^2$. The guarantee finds a separator; it does not select the widest margin or the best test performance.

Nonseparable data, including XOR in its original features or conflicting labels, never allow a mistake-free pass. Set a training limit and use validation performance to choose weights. An averaged perceptron predicts using the average of weights over training steps. A fixed positive learning rate simply scales every weight when starting from zero, so it leaves the binary prediction sequence unchanged; a changing rate can alter that behavior.

Correct a multiclass prediction

On a multiclass mistake, add the input to the true class vector and subtract it from the predicted class vector. Leave the rest unchanged. For input $(1,2)$, true class B and predicted A, vectors $(1,1)$ and $(2,0)$ become $(0,-1)$ and $(3,2)$. Their new scores are −2 and 7. The unchanged C score is 2, so B now wins. Like the binary perceptron, this model outputs a class, not a probability.

Logistic regression turns a linear score into a probability for a binary class. Its smooth output gives a graded training signal even when a hard classification is already correct.

Convert a score into probability

For labels 0 and 1, compute $z=𝐰\cdot 𝐱$ and

The sigmoid increases from values near zero to values near one. It gives $\sigma (0)=0.5$, $\sigma (1)\approx 0.731$, $\sigma (2)\approx 0.881$, and $\sigma (-2)\approx 0.119$. Predicting class 1 when $p\ge 0.5$ is equivalent to using $z\ge 0$, so this decision boundary remains linear.

The score equals the log-odds:

With weights $(-4,0.8)$ and input $(1,\text{hours studied})$, five hours give score zero and probability 0.5. At 7.5 hours the score is 2 and the probability is about 0.881. Each extra hour adds 0.8 to log-odds and multiplies odds by $e^{0.8}\approx 2.23$. It does not add 0.8 to probability. A predicted probability of 0.7 concerns this example's class; it is not a report of 70% model accuracy.

Train with log loss

For one example, binary log loss is

Only the term for the observed label remains. If $y=1$, use $-\ln p$; if $y=0$, use $-\ln (1-p)$. At $p=0.9$, these losses are about 0.105 and 2.303. A confident wrong prediction is much more costly than an uncertain one; $p=0.5$ gives loss about 0.693 for either label.

Maximum likelihood chooses weights that assign high joint probability to the observed labels. Taking a negative logarithm turns their probability product into the sum of log losses. Gradient descent fits the weights because there is no general closed-form solution.

The logistic log loss is convex in the weights, so every local minimum is global. The 0/1 loss is flat almost everywhere and supplies no useful gradient; squared loss after the sigmoid loses this convexity. On separable training data, unregularized log loss can decrease as weights grow without bound, so a finite maximum-likelihood solution is absent. A weight penalty controls that growth.

Use softmax for more classes

Give each class a score and normalize exponentials:

Scores $(2,1,0)$ produce probabilities about $(0.665,0.245,0.090)$. Adding one constant to every score changes neither the probabilities nor the winning class because the common exponential factor cancels. With two classes, softmax is a sigmoid of the score difference.

Logistic regression models $P(y\mid 𝐱)$ directly and fits weights jointly. It does not assume feature independence. Naive Bayes estimates a prior and feature distributions by counting; it can be especially effective with small data, but correlated evidence can make its probabilities too extreme.

Training often means finding weights that make a loss small. Gradient descent uses the local slope of that loss to choose the next weights, repeating until a stopping rule is met.

Find the downhill direction

A partial derivative measures how the loss changes when one parameter varies and the others stay fixed. The gradient collects all these derivatives. For

At $(2,1)$ the gradient is $(4,6)$. It points in the direction of steepest local increase under Euclidean distance; its negative points downhill. A smooth minimum has zero gradient, but a zero gradient can also occur at a saddle point or maximum.

With positive learning rate $\alpha$, update

Compute the entire gradient before changing any weight. From $(2,1)$ with $\alpha =0.1$, the next point is $(1.6,0.4)$, then $(1.28,0.16)$. Loss falls from 7 to 3.04 to about 1.72. The second coordinate changes faster because its curvature is larger.

See how step size controls convergence

For $L(w)=(w-3{)}^2$, the derivative is $2(w-3)$. From zero with rate 0.1:

1. The first gradient is −6, so the next weight is 0.6.
2. At 0.6 the gradient is −4.8, so the next weight is 1.08.
3. The following values are 1.464, 1.7712, and about 2.017.

Each step multiplies the distance to 3 by $1-2\alpha =0.8$. For this loss, convergence from a nonoptimal start needs $0<\alpha <1$. Rate 1 bounces between 0 and 6; rate 1.1 produces 6.6, −1.32, and 8.184, moving farther away. A smaller rate can be stable but slow. The same rate is not suitable for every loss or feature scale.

Differentiate a prediction loss

For a linear predictor $h=w_0+w_{1}x$ and one example with squared loss $(y-h{)}^2$,

Use $x=2$, $y=5$, and weights $(0,1)$. The prediction is 2, so the gradient is $(-6,-12)$. A rate of 0.05 gives new weights $(0.3,1.6)$. The next prediction is 3.5 and loss falls from 9 to 2.25. The bias derivative has input factor 1, while the slope derivative includes the measured input.

Read the loss surface

A convex loss has no inferior local minima: every local minimum is global. Linear squared loss and logistic log loss are convex. Neural network losses are generally nonconvex, with saddle points and flat regions as well as minima. Initialization can affect the final result.

Feature scales also affect optimization. Area up to 200 m² and room counts up to 6 can stretch the loss surface into a narrow valley. Standardization subtracts each feature's training mean and divides by its training standard deviation. Better-scaled coordinates reduce zigzagging. Stop using a chosen step budget, small gradients, or lack of improvement.

Computing a gradient over every example can be costly. Stochastic and mini-batch updates trade a perfectly averaged direction for more frequent, cheaper steps.

Choose how much data informs an update

An epoch visits the whole training set once. With 10,000 examples and batches of 100, an epoch has 100 update steps. Shuffle examples between epochs so the same groups do not recur in the same order.

A uniformly chosen single-example gradient has expectation equal to the full training gradient. Individual steps can nevertheless increase total loss. Plotting loss over epochs gives a clearer trend than judging one update. Noise can help movement around saddle points and shallow minima, but a fixed large learning rate also keeps weights jittering near a solution. A decreasing learning-rate schedule helps settle them.

Derive the logistic update

Let $p=\sigma (z)$, $z=𝐰\cdot 𝐱$, and use binary log loss. Its derivative with respect to probability is

Multiplying by the sigmoid derivative $p(1-p)$ leaves $\mathrm{\partial }L\mathrm{/}\mathrm{\partial }z=p-y$. Since $\mathrm{\partial }z\mathrm{/}\mathrm{\partial }{w}_i=x_i$, the complete gradient is

Subtracting it gives the equivalent update $𝐰\leftarrow 𝐰+\alpha (y-p)𝐱$. For a batch, average these vectors using the same current weights before making one update. Summing instead of averaging multiplies the effective learning rate by batch size.

Work one stochastic step

Start at $𝐰=(0,0,0)$ with input $𝐱=(1,2,1)$, target 1, and rate 0.5. The score is zero, so $p=0.5$. Then

The new score is $0.25+0.5(2)+0.25=1.5$ and the new probability is about 0.818. This example's log loss falls from about 0.693 to 0.201. That improvement concerns this example; a stochastic step need not improve every training case.

The perceptron also uses a label-related correction times the input, but it uses ±1 targets and changes weights only on mistakes. Logistic regression uses 0/1 targets and a smooth probability error. At finite weights, it can keep updating after the predicted class is correct.

Recognize common optimizer variants

Momentum carries a running average of gradient directions, helping movement along consistent directions and reducing oscillation. Adam combines this with adaptive per-parameter step sizes. Both still need a suitable learning rate. Compare settings with validation results and record validation loss along with training loss to decide when further training stops helping.

A neural network connects many simple units. Hidden layers transform inputs into learned features; nonlinear activations let those features represent patterns that a single linear classifier misses.

Compute units and count parameters

A unit calculates $z={\sum }_i{w}_i{a}_i+b$, then outputs $a=g(z)$. Here the incoming activations are its inputs, and the bias is shown separately. This is equivalent to including a constant bias feature.

A feedforward network has no directed cycles: information moves from input through hidden layers to output. In a fully connected layer, each unit receives every output of the previous layer. A layer with $m$ inputs and $n$ units has $mn$ weights and $n$ biases. Three inputs, four hidden units, and two output units therefore give

trainable parameters. Count one bias per receiving unit, not one per connection. Networks with many hidden layers are called deep.

Follow a network that computes XOR

Use two hidden ReLU units and a linear output:

For $(0,0)$, the hidden values are $(0,0)$, so the output is 0. For either $(0,1)$ or $(1,0)$, they are $(1,0)$, giving 1. For $(1,1)$, they are $(2,1)$, giving $2-2=0$. The hidden representation allows the output to distinguish the XOR classes.

Without nonlinear activations, stacking layers would still make one affine transformation. Ignoring biases for clarity, $W_2(W_1𝐱)=(W_2{W}_1)𝐱$. Depth alone would add no nonlinear expressive power. The bends introduced by activations change what the network can represent.

Match outputs to targets

For a numerical regression target, a linear output and squared loss are a standard choice. For binary labels, use one sigmoid output and binary log loss. For one of several classes, use softmax outputs and cross-entropy $-\ln p_{\text{correct}}$. Probabilities $(0.665,0.245,0.090)$ with the second class correct give loss about 1.41. The loss uses the probability assigned to the true class, even when another class wins.

The universal approximation theorem says a sufficiently wide hidden layer with a suitable nonlinear activation can approximate any continuous function on a closed, bounded input region to arbitrary accuracy. It is an existence result, not a guarantee that training finds the weights or that a small network suffices. Depth can represent some functions much more efficiently.

Hidden representations can progress from image edges and textures to object parts. Architectures also build in useful structure: convolutional networks share small weight sets across image positions, while recurrent networks share weights across sequence steps.

Backpropagation computes how every parameter affects loss. A forward pass stores the values needed by the chain rule; a backward pass reuses them to avoid repeating the same derivative calculations.

Move through a computation graph

A computation graph separates sums, products, and activations into operations. Along a path through an intermediate value,

When a parameter affects loss along several paths, add their contributions. For unit $j$, write ${\delta }_j=\mathrm{\partial }L\mathrm{/}\mathrm{\partial }{z}_j$. Its backward signal is

The sum includes units that receive its output. A weight from activation $a_i$ into unit $j$ has gradient $a_i{\delta }_j$. The bias gradient is simply ${\delta }_j$ because its input is 1.

Work a forward and backward pass

Let $x=2$, $h=\mathrm{ReLU}(w_{1}x)$, $\widehat{y}=w_{2}h$, and $L=(\widehat{y}-y{)}^2$. Start with $w_1=0.5$, $w_2=3$, and target $y=1$.

1. Forward: the hidden sum is 1, hidden output is 1, prediction is 3, and loss is 4.
2. At the output, $\mathrm{\partial }L\mathrm{/}\mathrm{\partial }\widehat{y}=2(3-1)=4$. Thus $\mathrm{\partial }L\mathrm{/}\mathrm{\partial }{w}_2=4h=4$.
3. Going back through the output weight gives $\mathrm{\partial }L\mathrm{/}\mathrm{\partial }h=4(3)=12$. The hidden sum is positive, so the ReLU derivative is 1.
4. The first weight's gradient is therefore $12(1)(2)=24$.

With rate 0.01, update both weights from these stored gradients:

A fresh forward pass gives hidden output 0.52, prediction 1.5392, and loss about 0.291. Using 2.96 during the preceding backward pass would mix two parameter states and produce the wrong gradient.

Check derivatives and recognize weak signals

A numerical check approximates one derivative by

With $\epsilon =0.001$, the first weight's derivative above is 24. This needs two forward passes per weight, so it is useful for checking, not large-scale training. Backpropagation computes all gradients at a cost of the same order as a forward pass. Libraries implement this as reverse-mode automatic differentiation.

Sigmoid has derivative $\sigma (z)(1-\sigma (z))\le 0.25$; tanh has derivative $1-{\tanh }^{2}z$. ReLU's derivative is 1 above zero and 0 below; zero is commonly assigned derivative 0. A ReLU unit whose preactivation is negative for every input is dead and receives no useful gradient through its activation. Softmax cross-entropy has score gradient $𝐩-𝐲$ for one-hot targets.

Repeated small derivative factors cause vanishing gradients. A chain of 10 sigmoid units with connecting weights 1 has derivative at most ${0.25}^{10}\approx 9.5\times {10}^{-7}$. Wider graphs also depend on weights and summed paths. ReLU, suitable initialization, and residual connections reduce this problem. Large backward signals can instead explode; gradient clipping caps their size.

Network training needs more than a correct derivative. Starting weights, input scales, update settings, and held-out results all affect whether useful features are learned.

Break symmetry and scale signals

Initialize weights randomly so hidden units begin differently. If a layer's units have identical weights and the same outgoing connections, they compute the same values, receive the same updates, and remain copies. All-zero network weights create this symmetry. Biases can start at zero because random connection weights already distinguish units.

The random scale matters. Glorot, or Xavier, initialization is suited to tanh and sigmoid units; He initialization accounts for ReLU activations. These schemes aim to keep signal sizes from shrinking or growing too much from layer to layer.

Normalize inputs as well. Dividing pixel values by 255 puts them between zero and one. Standardization subtracts a feature's training mean and divides by its training standard deviation. Apply the same fitted transformation to later inputs. Moderate initial sums help sigmoid and tanh avoid saturated regions with very small derivatives; comparable feature scales also improve optimization.

Run the training loop

1. Initialize the weights and choose the optimizer and settings.
2. At each epoch's start, shuffle the training examples.
3. For each mini-batch, run a forward pass and compute average loss.
4. Backpropagate, then update all parameters using that batch's gradients.
5. At the epoch's end, record training and validation results.

For 784 pixel inputs, 128 hidden ReLU units, and 10 softmax outputs, the parameter count is

With 60,000 images and batch size 64, there are 937 full batches and one final batch of 32 images: 938 updates per epoch. The final short batch still uses its own average gradient.

Read training curves

These patterns guide diagnosis; several causes can share a symptom. For a slowly learning sigmoid network, inspect input scale and activation saturation along with the learning rate. A high training accuracy only shows fit to training examples. A large network can even memorize random labels.

The main hyperparameters include learning rate, batch size, epoch count, width, depth, activation, and optimizer. Compare them on validation data. Keep the test set for evaluating the selected configuration. Increasing width changes parameter count and capacity; increasing epochs gives the same parameters more update steps.

Network loss is nonconvex. Different random starts can reach different weights and accuracies. In large networks, saddle points and flat regions often slow progress more than poor local minima. Momentum and Adam help optimize such surfaces, but held-out performance still decides whether the fitted pattern is useful.

A model can fit its training sample well without learning a dependable pattern. Compare training and held-out results to understand whether the model is too restricted, too flexible, or facing different data.

Compare two kinds of error

Generalization is performance on new examples from the same distribution. Underfitting occurs when the model is too restricted to capture the pattern. Overfitting occurs when it fits sample-specific noise so closely that new predictions suffer.

Choose C from these validation results. A's higher training score is not an advantage on the held-out cases. B's small gap is not enough to make it good: both scores are low relative to what the task allows. Difficulty, noise, and achievable performance matter when interpreting a small gap.

Relate capacity to noise

Capacity describes how flexible the hypothesis family is. Ten noisy points may lie near a straight trend. A line uses two coefficients and misses the points slightly. A degree-9 polynomial uses ten coefficients and can interpolate all ten distinct input points, reaching zero training error. Between and beyond them, that polynomial can swing sharply and predict badly.

As capacity increases, training error generally falls. Held-out error often falls at first as genuine structure becomes representable, then rises as noise is fitted. Compare candidate capacities using held-out results. This familiar pattern is a guide to model choice, not a guarantee that every possible network-size curve has exactly that shape.

High bias means systematic error from a restrictive family. High variance means the learned rule changes substantially across training samples of the same size. Underfitting commonly involves high bias; overfitting commonly involves high variance.

Separate more data from different data

For fixed capacity, more representative examples usually reduce overfitting. A degree-9 polynomial cannot generally interpolate 1,000 noisy points exactly, so its fit must compromise. Large networks can still memorize large samples, including random labels. More data reduce risk without eliminating the need for held-out checks. Wrong or noisy labels add accidental patterns the learner might fit.

Distribution shift is a different problem. A classifier trained and tested on daytime photos can fail on night photos even when its daytime generalization is good. Those new inputs come from a different distribution. More regularization alone does not supply missing night conditions.

Preserve the meaning of the final test

Train candidates on the training set and compare their settings or capacities on validation data. Once a candidate is chosen, measure it on the untouched test set. Using test scores to change architecture or choose an epoch turns that set into another source of model selection, making its reported performance optimistic. Overfitting can affect polynomials, naive Bayes, perceptrons, and networks; it is a relationship between learning and data, not a property unique to one algorithm.

Regularization changes learning to improve predictions on new data. It can constrain weights, limit training duration, interrupt reliance on particular units, or supply more valid examples.

Penalize large weights

A regularized objective combines data loss and a complexity penalty:

For L2, use $C={\sum }_i{w}_i^2$, usually excluding the bias. Weights $(3,-4)$ and $\lambda =0.1$ give penalty $0.1(9+16)=2.5$. The added derivative is $2\lambda w_i$. Under ordinary gradient descent,

With rate 0.1 and strength 0.1, the shrinkage factor is 0.98; a weight of 5 contributes 4.9 before the data-loss step. This is the weight-decay interpretation for this optimizer. With adaptive optimizers such as Adam, an L2 penalty and separately applied weight decay are different operations.

L1 uses $C={\sum }_i\mathrm{\mid }{w}_i\mathrm{\mid }$. It favors sparse solutions with many weights exactly zero, selecting features. L2 usually shrinks weights without making them exactly zero. Neither penalty is a rule to set every weight to zero.

The strength is a hyperparameter. Zero removes the penalty; excessive strength makes the model underfit. Compare values across orders of magnitude on validation data. A positive L2 penalty also controls the divergent weights of separable logistic regression.

Keep the best stopping point

Early stopping monitors validation loss, saves the best weights, and stops after a patience interval without improvement. For losses 0.62, 0.48, 0.41, 0.39, 0.40, 0.43, and 0.47 at epochs 1–7, the best epoch is 4. Patience 3 allows epochs 5–7, then stops and restores epoch 4's weights. The final trained epoch is not necessarily the saved model.

Train with dropout

At each training step, dropout independently sets hidden-unit outputs to zero with probability $p$. Units must learn without relying on a particular partner always being present. At prediction time dropout is disabled and all units participate.

In inverted dropout, divide surviving activations by $1-p$ during training. If $p=0.5$, surviving outputs double, so their expected value matches the undropped activation. This scaling is not applied again at prediction time.

Add valid variation

Collecting more representative data and using a simpler model can reduce overfitting. Data augmentation adds transformed training examples whose labels stay correct, such as suitable shifts, crops, or brightness changes. A transformation must suit the task: rotating a handwritten 6 by 180° can change it into a 9, so keeping label 6 would teach the wrong relationship.

Laplace smoothing is another regularizer: pseudo-counts make naive Bayes less dependent on sparse counts. Use validation data to choose smoothing, penalties, dropout rates, or stopping times. Watch both training and validation error so protection against overfitting does not become excessive restriction.