05.3 Training instability, mode collapse and mitigation (HL only)

01 | A CONVINCING IMAGE CAN HIDE A TRAINING PROBLEM

A GAN produces one sharp mountain scene. That looks promising, until the designer requests more candidates and receives almost the same scene repeatedly.

Another run produces varied images at first, then its results deteriorate. These are different warning signs: limited diversity and unreliable training progress.

The 2027 booklet identifies training instability and mode collapse as GAN challenges. This HL lesson explains how to recognise problems and evaluate possible responses without treating any technique as a guaranteed cure.

02 | WHAT IS TRAINING INSTABILITY?

Training instability means the learning process fails to make reliable useful progress. Behaviour may oscillate, deteriorate or encounter numerical failure.

In adversarial training, each network changes the other’s learning task. Updates that improve one objective can make the other’s task harder, so the interaction can be difficult to manage.

Not every fluctuating loss is a failure. Different batches and a changing opponent can produce variation even during useful learning. Look for sustained patterns and inspect outputs.

03 | WHY BALANCE IS DIFFICULT

The generator and discriminator may learn at different rates or have different capacities. A weak discriminator can provide a poor guide to useful realism.

A discriminator that separates sources extremely confidently can also create difficulties for the generator, depending on the loss and optimisation setup. With the original saturating objective, weak gradient signals can be a particular concern.

Balance does not mean equal losses or an automatic one-to-one update ratio. It means an interaction that supports useful learning under the chosen formulation.

04 | OSCILLATION AND DETERIORATION

Suppose the generator learns one kind of image that fools the current discriminator. The discriminator then adapts, and the generator moves towards another strategy. The process can cycle rather than settle into broad, reliable quality.

Loss values and image quality may change together or tell different stories. A rising loss can reflect a stronger opponent, not simply a broken implementation.

Use documented checkpoints and repeated sample sets to see whether progress is useful. A single plot point cannot diagnose an adversarial system.

05 | NUMERICAL INSTABILITY IS A SPECIFIC WARNING

Very large values, infinities or NaN values indicate numerical trouble. Possible causes include unsuitable updates, problematic data or numerical precision limits.

Check the inputs, loss implementation and training records before adjusting settings. Restarting repeatedly can hide a repeatable fault rather than solve it.

Numerical failure differs from a model producing repetitive images with finite losses. Both are problems, but they need different evidence and investigations.

06 | WHAT IS A MODE?

A mode is a high-density region of a distribution. For intuition, a dataset may contain several distinct groups of appearances.

Those groups are not necessarily identical to class labels. An animal dataset can contain many meaningful variations within one species, including poses, colours and backgrounds.

When assessing diversity, define the variations that matter to the task. Counting category names alone may miss important gaps.

07 | WHAT IS MODE COLLAPSE?

Mode collapse occurs when the generator produces a limited range of outputs rather than representing useful diversity in the data.

Different noise vectors may map to very similar images. Individual samples can look realistic, so judging only a favourite output can conceal the failure.

The term does not require every output to be pixel-identical. A model may repeatedly produce a narrow group of plausible scenes while neglecting other important variations.

08 | WHY COLLAPSE CAN OCCUR

A generator may find a narrow set of outputs that receives favourable responses from the current discriminator. Producing more of those outputs can improve its immediate objective without covering the wider distribution.

The discriminator may adapt, and the generator may move to another narrow solution. This can connect poor coverage with unstable training dynamics.

This is a conceptual explanation, not a unique diagnosis of every repetitive model. Bugs, restricted inputs or a narrow dataset can also produce limited variation.

09 | DISTINGUISH COLLAPSE FROM CONSISTENCY

A campaign may deliberately preserve one character’s identity. That is not mode collapse if the requested poses, expressions and settings can still vary usefully.

Likewise, outputs from the same fixed vector may be identical in a deterministic generator. To investigate diversity, vary the intended sampling inputs and inspect a set.

For Visionary Studios, the goal is stable essential features alongside meaningful permitted variation. Neither random changes to identity nor repeated copies of one scene satisfy that aim.

10 | INSPECT A SAMPLE GRID

Generate many candidates using varied inputs under recorded conditions. Arrange them together so repeated composition, pose and textures become easier to spot.

These observations suggest investigations; they do not establish a cause by themselves.

11 | REVIEW THE DATA AND IMPLEMENTATION FIRST

Check for duplicates, missing variation, incorrect preprocessing and accidental reuse of the same noise input. Verify that the intended parameter sets are updated in each training phase.

A narrow training collection can limit output diversity even without the particular adversarial failure called mode collapse. A data-loader bug may repeatedly present the same examples.

Correcting the basic pipeline can be more appropriate than adding a sophisticated mitigation to a faulty experiment.

12 | ADJUST LEARNING RATES AND UPDATE SCHEDULES

Different learning rates or update counts can change the interaction between the networks. A smaller update may reduce overshooting, while a poorly chosen schedule can slow useful progress.

Change settings deliberately and monitor quality, diversity and losses together. There is no universal ratio or learning rate that fixes all GANs.

Keep comparison conditions recorded and repeat promising runs. An improvement after several simultaneous changes does not establish which change helped.

13 | OBJECTIVES CAN IMPROVE LEARNING SIGNALS

A non-saturating generator objective can provide stronger learning signals than the original saturating objective in some conditions, especially when the discriminator confidently rejects generated samples.

Wasserstein-style objectives are another family of approaches. They use a critic and a different interpretation of its output, with suitable constraints, rather than the simple real-class probability explanation.

These are supporting research directions. They do not guarantee stability or diversity, and a changed objective may require changes to the training procedure and evaluation.

14 | REGULARISATION CAN LIMIT UNHELPFUL BEHAVIOUR

Regularisation adds constraints or penalties intended to improve learning behaviour. Examples in appropriate GAN formulations include spectral normalisation or gradient penalties.

Spectral normalisation controls the scale of weight transformations. Gradient penalties constrain aspects of how a critic or discriminator output changes with inputs, depending on the formulation.

Do not treat those methods as interchangeable switches. They have different assumptions, overheads and hyperparameters. Evaluate them in a correctly implemented setup rather than promise a universal cure.

15 | ENCOURAGE ATTENTION TO DIVERSITY

Some approaches allow the discriminator to consider relationships within a batch, helping it detect that generated examples are suspiciously similar. Other objectives compare learned features or encourage broader coverage.

Such methods can make repetition less advantageous, but require implementation and testing. Adding random noise alone does not force a collapsed generator to use that information meaningfully.

Inspect the quality–diversity balance. Outputs that are all different because they contain defects are not evidence of useful coverage.

16 | CHECKPOINTS, METRICS AND REPEATED RUNS

Save checkpoints with training conditions so earlier useful models can be evaluated and interruptions can be investigated. Track failures and recovery effort, not only the final successful run.

Distributional metrics can support comparisons, but depend on representations, sample sizes and reference data. A single number cannot replace inspection of task-relevant quality and coverage.

Repeat trials where feasible. GAN behaviour can vary with initialisation and sampled data, so one lucky run is weak evidence of reliable production.

17 | APPLY IT TO VISIONARY STUDIOS

The booklet identifies the challenge but does not prescribe one mitigation as the company’s solution. Distinguish proposals from reported facts.

18 | PROPOSE AN EVIDENCE-LED RESPONSE

In your book, analyse a run that has finite losses and produces nearly identical scenes from different sampled vectors. State two checks before changing the model and propose one mitigation with a limitation.

Then explain why consistent character identity across varied scenes would not, by itself, demonstrate collapse.

HL only. Read alongside the GAN challenges in the official 2027 booklet. The diagnostic examples and named mitigation techniques are independent teaching extensions.

CHECK YOUR UNDERSTANDING | WHAT DOES THE OBSERVATION SUGGEST?