06.1 Latent space and variational autoencoders (HL only)

01 | REPRESENT AN IMAGE IN ANOTHER FORM

An image can contain many thousands of channel values. A model can learn another representation that captures useful relationships without keeping the data in its original pixel arrangement.

A latent representation is an internal representation whose features are learned rather than necessarily supplied as explicit labels. Latent space is the space of possible values of that representation.

The case study introduces variational autoencoders as possible hybrid-model components. HL students need a broad understanding of VAEs and flow-based models, so this lesson focuses on meaning and process rather than mathematical derivations.

02 | WHY CALL IT A SPACE?

A vector is an ordered list of numerical values. If a representation has two components, we can picture it as a point on a two-axis diagram. Real latent representations often have many more dimensions.

Different points can decode into different outputs. Nearby points may produce related appearances when the model has learned a useful arrangement.

Each coordinate is not automatically a named property such as hair colour. Learned information can be distributed across many dimensions.

03 | START WITH AN AUTOENCODER

An ordinary autoencoder has an encoder that maps input data into a representation and a decoder that maps the representation back into output data.

Training encourages the output to reconstruct the input under a chosen objective. A smaller representation can encourage the system to retain useful structure while omitting some detail.

This is learned representation and reconstruction, not automatically lossless file compression. A reconstructed image can differ from the original.

04 | RECONSTRUCTION IS A SPECIFIC TASK

For reconstruction, the system starts with an existing image. The decoder attempts to produce an output corresponding to that image after encoding.

A reconstruction loss measures the mismatch under a defined method. A low pixel-based loss does not necessarily mean the output preserves every feature important to a client.

Simply training an ordinary autoencoder does not guarantee that arbitrary points in its representation space will decode into plausible new images.

05 | WHAT MAKES A VAE VARIATIONAL?

A variational autoencoder , or VAE, introduces a probabilistic representation. Instead of only producing one fixed code, its encoder estimates parameters of a distribution over latent values for an input.

In a common formulation, the encoder outputs a mean and a variance-related quantity for a Gaussian distribution. A latent value is then sampled from that distribution.

The decoder uses the sampled latent value to model the input data. This probabilistic arrangement supports learning a latent space that can also be sampled for generation.

06 | MEAN AND SPREAD FROM FIRST PRINCIPLES

A mean describes a distribution’s centre. A standard deviation describes its spread; variance is the square of standard deviation.

For intuition, one input might be represented by values concentrated around a particular location rather than by a single exact point.

The encoder’s mean and spread are learned numerical outputs. They are not a human-written confidence percentage that the image contains a particular object.

07 | FOLLOW THE RECONSTRUCTION PATH

The training objective connects this path to the original input. Sampling can introduce variation, so repeated reconstructions need not be pixel-identical.

The decoder is a learned component, not a lookup table that simply retrieves a photograph attached to each coordinate.

08 | A SMALL SAMPLING EXAMPLE

In a common Gaussian setup, a latent component can be written z = μ + σ × ε . μ is the learned mean, σ is standard deviation and ε is sampled standard normal noise.

For invented teaching values μ = 2, σ = 0.5 and ε = −1, we obtain z = 2 + 0.5 × (−1) = 1.5 .

This illustrates sampling one component. A real representation has multiple components and learned networks. The equation alone is not a complete VAE or a method for choosing an artistic property.

09 | WHY WRITE SAMPLING THAT WAY?

The expression separates the random draw ε from the encoder’s learned outputs μ and σ. This is called the reparameterisation trick .

It allows useful gradients to flow through the transformation to the encoder’s distribution parameters during training. You do not need to derive those gradients for this broad case study understanding.

Randomness does not mean learning stops being numerical. The training procedure is designed so that the sampled representation can still participate in optimisation.

10 | TWO PARTS OF THE LEARNING OBJECTIVE

A common VAE objective balances reconstruction with a term encouraging the encoded latent distributions to remain close to a chosen prior, often a standard normal distribution.

The reconstruction part encourages the output to explain the input. The regularising part helps organise the representation for sampling rather than allowing arbitrary disconnected codes.

This second term is commonly expressed using KL divergence. It measures a distributional difference in a specified direction, not an ordinary pixel distance.

11 | UNDERSTAND THE TRADE-OFF

Emphasising reconstruction can preserve more input detail, while strong regularisation can restrict what the representation retains. The balance affects the learned model.

Some conventional VAEs produce softer reconstructions or samples than desired for high-detail creative work. This depends on architecture and objective, so it is not a rule that every VAE output must be blurry.

Evaluate both reconstruction fidelity and the usefulness of sampled outputs. One objective value alone does not establish artistic suitability.

12 | GENERATE WITHOUT A SOURCE IMAGE

For new generation, a VAE can sample a latent value from its prior and pass it through the trained decoder. An input image is not required for this basic generation path.

This differs from reconstruction, where the encoder first processes an existing example and supplies a distribution associated with that input.

A prior sample is not automatically a copy of one training image, but learned models can retain training information. Artificial generation does not itself guarantee originality or permission.

13 | EXPLORE BETWEEN REPRESENTATIONS

Interpolation means forming values between selected representations. Decoding intermediate values can reveal gradual changes between appearances.

For a two-component teaching example, the midpoint between [0, 2] and [4, 6] is [2, 4]. Real latent geometry can be more complicated than a straight line in a simple diagram.

A midpoint does not guarantee a perfect blend of two semantic concepts. Useful transitions depend on the model, the points and the path chosen.

14 | LATENT DOES NOT MEAN A TINY PHOTOGRAPH

A latent representation can be a vector or a spatial array with learned channels. Its values need not be directly interpretable as visible pixels.

Decoding maps that representation back into image data. The decoder can introduce approximation, so encoding and decoding are not necessarily exact inverses.

Keep VAEs distinct from flow-based models, which the next lesson introduces through reversible transformations. A VAE does not promise lossless reversal.

15 | A VAE CAN BE ONE PART OF A HYBRID

A hybrid can use an encoder and decoder to work with a learned representation while another model performs part of the generation process.

For example, in a compatible latent diffusion arrangement, diffusion operates on encoded representations and a decoder converts the result to an image. The diffusion model’s predictions are not themselves the VAE’s reconstruction objective.

The components must be compatible and appropriately trained. Naming several model types does not explain how they connect or guarantee better results.

16 | POSSIBLE VALUE FOR VISIONARY STUDIOS

A useful latent representation can support experimentation, variation and generation within a learned visual space. A compatible hybrid may reduce the spatial workload of another component.

However, compression can lose fine details and decoding can affect output fidelity. A brand logo or recurring character feature still needs checking.

Do not assume that moving through latent coordinates gives exact controls for every client requirement. Interpretability and controllability require evidence.

17 | EVALUATE THE COMPLETE PATH

Test reconstructions to identify information lost in encoding and decoding. Separately inspect prior samples and any hybrid-generated outputs for quality and useful variety.

Record preparation, encoding, generation and decoding costs where relevant. Compare complete task outcomes rather than only representation size.

18 | EXPLAIN BOTH PATHS

In your book, draw reconstruction and new generation as separate paths. Label the encoder, distribution parameters, sampled latent value, prior and decoder where each belongs.

Then calculate the sampling example and the interpolation midpoint, explaining why neither calculation specifies a guaranteed visual meaning.

HL only. The booklet requires a broad understanding of VAEs. The sampling arithmetic and detailed vocabulary here are supporting teaching explanations, not extra official syllabus objectives.

CHECK YOUR UNDERSTANDING | FOLLOW THE REPRESENTATION