Neural Networks as Probability Models
The previous article turned latent inference into optimization over approximate distributions. We now need a learnable function that maps an input to distribution parameters. A neural network provides that function.
We only need feedforward networks, the chain rule, and minibatch updates. The central question is how a deterministic network participates in a stochastic model.
Reading guide. Read each purpose, main result, and takeaway first, then expand derivations or code as needed. Interactive figures remain in the main text so the argument can be followed without opening every calculation.
1. Neural network forward computation
Purpose. Start with the network as a function mapping inputs to outputs. Using fixed, existing parameters for a forward computation is commonly called model inference; learning parameters from data and a loss is called training.
1.1 Layers and nonlinearity
Purpose. View a layer as a weighted combination followed by a nonlinear transformation; a multilayer network repeats this process.
For an input vector $u$, a layer applies an affine transformation followed by an elementwise activation:
\[a=Wu+b,\qquad h=\tanh(a).\]The rows of $W$ specify output coordinates and its columns input coordinates. A multilayer network composes such functions:
\[h=\tanh(W_1u+b_1),\qquad o=W_2h+b_2.\]Nonlinearity matters: a composition of affine transformations without nonlinear activations is still affine. We use $\tanh$ for a compact example, not because VAEs require it.
Takeaway. Forward computation determines how inputs affect outputs. Nonlinearity enriches the function, without introducing sampling by itself.
1.2 Inputs, parameters, and training
Purpose. Distinguish supplied inputs, learned weights, and latent samples drawn later from a distribution.
Weights and biases are optimized parameters. Inputs are supplied data, and the loss evaluates the current prediction. A forward pass alone does not learn anything; training also differentiates the loss and changes parameters.
Model inference uses existing parameters to compute outputs. Training also includes a forward pass, followed by gradient computation and parameter updates. Here, model inference means evaluating the network; posterior inference means obtaining the conditional distribution of latent variables. In a VAE, one encoder forward pass can supply approximate-posterior parameters.
A VAE has generative-network parameters $\theta$ and inference-network parameters $\phi$. Both are distinct from the per-observation random latent variables $z_i$.
Takeaway. Training updates weights and biases. Encoder parameters, decoder parameters, and per-observation latent samples have different roles.
Section 1 summary. A network supplies an adjustable function; training changes it. Next its outputs will specify distribution parameters.
2. Network outputs and probability distributions
Purpose. Use the network to specify a distribution through means, variances, or binary probabilities; draw random outcomes afterward.
2.1 Gaussian outputs: mean and log variance
Purpose. For continuous outcomes, predict both a typical location and uncertainty using a Gaussian mean and variance.
Let a network output two vectors:
\[(\mu_\theta(u),r_\theta(u))=f_\theta(u),\qquad r_j=\log\sigma_j^2.\]These define a diagonal Gaussian conditional distribution:
\[p_\theta(y\mid u)=\mathcal N\!\left(y\mid\mu_\theta(u), \operatorname{diag}(\exp r_\theta(u))\right).\]The network remains deterministic. Randomness enters when we subsequently sample from the distribution. Log-variance outputs guarantee positive variances while allowing unconstrained real network outputs. Remember that
\[\sigma^2=\exp r,\qquad\sigma=\exp(r/2).\]Using $\exp r$ as the sampling standard deviation would contradict the declared variance.
This is a forward pass through a fixed small network, not training. Input and mean shows mean ±2σ with dashed curves; Output distribution shows the conditional Gaussian density. These are not confidence intervals for model parameters.
Example parameters: the fixed interactive network
The fixed network in the figure uses
\[\begin{aligned} \mu(x)&=0.8\tanh(1.2x+0.4)-0.5\tanh(-0.7x)+b,\\ r(x)&=-1+0.5\tanh x. \end{aligned}\]Moving the input changes the distribution’s position and width; changing $b$ only shifts the mean. The figure performs no training, and its chosen weights are not a fit to a real dataset.
Takeaway. Fixed inputs and weights give a fixed distribution, while samples can vary. Sampling uses SD, whereas the network here outputs log variance.
2.2 Bernoulli outputs
Purpose. Binary outcomes need a valid probability. Transform the real network output into a probability for the two outcomes.
For a binary outcome, a real output $a_\theta(u)$ can define
\[\pi_\theta(u)=\operatorname{sigmoid}(a_\theta(u)),\qquad p_\theta(y\mid u)=\pi_\theta(u)^y(1-\pi_\theta(u))^{1-y}.\]A network output, a distribution parameter, and a sampled outcome are different objects. Bernoulli outcomes must be 0 or 1. A continuous measurement lying in $[0,1]$ is not automatically Bernoulli. Our minimal VAE experiment uses continuous Gaussian observations.
Takeaway. A real output, its transformed probability, and a sampled zero or one are distinct. The observation model must match the data type.
Section 2 summary. The network sets parameters; the probability model specifies outcomes and sampling. Next derive a training loss from that model.
3. Observation models and training losses
Purpose. Reward network outputs that assign high density to observed data. Negative log likelihood converts that requirement into a minimized loss.
The Gaussian negative log-likelihood for one observation is
\[\ell(y,u;\theta)=\frac12\sum_{j=1}^D \left[(y_j-\mu_j)^2\exp(-r_j)+r_j+\log(2\pi)\right].\]Calculation: mean and log-variance gradients
Both $\mu_j$ and $r_j$ depend on the input through the network. Their derivatives are
\[\frac{\partial\ell}{\partial\mu_j}=(\mu_j-y_j)\exp(-r_j), \qquad \frac{\partial\ell}{\partial r_j} =\frac12\left[1-(y_j-\mu_j)^2\exp(-r_j)\right].\]Increasing variance downweights the squared residual, but the $r_j$ term penalizes excessive variance. With a learned variance, the squared residual alone is not the Gaussian negative log-likelihood.
If the observation SD is fixed at $\tau$ in every coordinate,
\[\ell=\frac{\|y-\mu_\theta(u)\|^2}{2\tau^2} +D\log\tau+\frac D2\log(2\pi).\]The last two terms are constant for network gradients when $\tau$ is fixed, but they change reported values. The later implementation retains them and reports nats per observation.
Takeaway. Fixed Gaussian variance gives weighted squared error. Learned variance requires its additional terms, preventing variance inflation from freely downweighting errors.
4. Backpropagation and parameter updates
Purpose. Given a loss, determine how each weight affects it. Backpropagation computes those derivatives and the optimizer applies an update.
4.1 Chain rule and gradient propagation
Purpose. Start at the output error and follow dependencies backward. Matrix derivatives are optional; the interactive figure shows the same process.
Calculation: backpropagation through two layers
For $o=W_2h+b_2$, $h=\tanh(a)$, and $a=W_1u+b_1$, suppose we already know $\delta_o=\partial\ell/\partial o$. Then
\[\begin{aligned} \nabla_{W_2}\ell&=\delta_o h^\top,&\nabla_{b_2}\ell&=\delta_o,\\ \delta_a&=(W_2^\top\delta_o)\odot(1-h\odot h),\\ \nabla_{W_1}\ell&=\delta_a u^\top,&\nabla_{b_1}\ell&=\delta_a. \end{aligned}\]The symbol $\odot$ denotes elementwise multiplication. Backpropagation applies these local derivatives in reverse dependency order. These expressions use one observation and column vectors; batch implementations sum or average observation gradients.
Automatic differentiation records computational dependencies and applies derivative rules. It cannot decide whether our loss matches the probability model we intended.
One neuron uses μ=tanh(wx), target y=0.6, and J=(μ−y)²/2. Observation SD is 1; a gradient-independent constant is omitted. The arrow previews a step; the button updates the weight. Large steps need not improve the loss.
Follow the computational chain through multiplication, activation, and residual. The dashed segment previews an update; the button actually changes the weight. Take several steps with fixed data, then increase the learning rate and check whether a longer step necessarily helps.
Takeaway. The chain rule combines local effects into gradients for all weights. Automatic differentiation executes this calculation but requires correct dependencies.
4.2 Minibatch gradient updates
Purpose. Adjust weights using the gradient and repeat with another batch. The learning rate sets the step size.
Temporarily collect all parameters into $w$. For a batch $B$,
\[J_B(w)=\frac1{|B|}\sum_{i\in B}\ell_i(w),\qquad w\leftarrow w-\eta\nabla_wJ_B(w).\]The learning rate is $\eta$. Minibatch stochastic gradient descent uses part of the dataset to estimate an update direction. It saves computation but does not guarantee a decrease in the whole-dataset loss at each step. Adam adaptively scales gradient updates; it cannot repair an incorrect model or a mistaken reduction over dimensions.
An implementation clears old gradients, computes the current loss, backpropagates, and then updates parameters. Otherwise gradients may accumulate across steps.
Takeaway. Minibatches save work but individual losses can fluctuate. Clear gradients, differentiate, and update in order to avoid unintended accumulation.
Section 4 summary. Backpropagation computes gradients; the optimizer changes weights. Neither automatically validates the model or loss.
5. Encoder and decoder
Purpose. Return these tools to the latent model: one network infers a latent distribution from observations, and another specifies observations from latent values.
A deterministic autoencoder compresses an input through a bottleneck and reconstructs it:
\[z=e_\phi(x),\qquad\widehat x=d_\theta(z).\]This illustrates the computational directions of encoding and decoding. Reconstruction alone specifies neither a prior from which to sample $z$ nor a guarantee that samples from an arbitrary prior will decode sensibly.
For the VAE, we will instead specify
\[q_\phi(z\mid x)=\mathcal N\!\left(z\mid\mu_\phi(x), \operatorname{diag}(\exp r_\phi(x))\right),\] \[p_\theta(x\mid z)=\mathcal N\!\left(x\mid m_\theta(z),\tau^2I\right).\]The encoder outputs approximate-posterior parameters; the decoder outputs observation-distribution parameters. The generative graph still points $Z\to X$. The inference distribution $q_\phi(z\mid x)$ is separately constructed; it is not an extra reverse edge casually added to the same generative graph.
The remaining question is how to differentiate through the sampled $z$ between the two networks. Continue with Sampling, Reparameterization, and Amortized Inference.
Takeaway. The encoder supplies approximate-posterior parameters and the decoder observation parameters. Differentiating through sampling between them is the next problem.
6. Check your understanding and notation
- With fixed inputs and weights, how can a deterministic network define a random model?
- If the network outputs $r=\log\sigma^2$, which expression gives the sampling SD?
- Why must a learned Gaussian variance retain more than a squared-error term?
- How do a latent sample, encoder parameters, and decoder parameters differ?
| Symbol | Meaning |
|---|---|
| $W,b$ / $h$ | Weights and biases / hidden activations |
| $r$ | Log variance, not log standard deviation |
| $\theta$ / $\phi$ | Generative / inference-network parameters |
| MLP / SGD | Multilayer perceptron / stochastic gradient descent |
7. References
- Christopher M. Bishop, 2006. Pattern Recognition and Machine Learning, §§5.1–5.3 and §12.4.2. Probabilistic outputs, backpropagation, and autoassociative bottlenecks.
- David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, 1986. Learning representations by back-propagating errors. The original representation-learning research.
- Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes, Appendix C. MLP parameterizations of probabilistic encoders and decoders.
Sources and corrections. This article reorganizes the course’s neural-network material and the references around the tools required for a VAE. Formulas follow the conventions stated here; the interactive figure is independently implemented. Rights to the original materials remain with their authors or publishers. Corrections to formulas, terminology, and interpretation are welcome.
AI feedback
AnonymousLoading AI feedback…