Understand the technology
From a test question to a defensible comparison.
For May and November 2027. This lesson is for SL and HL students. Plan comparable trials, work through a fictional result set and practise drawing appropriately limited conclusions.
01 | TWO IMPRESSIVE IMAGES DO NOT MAKE A FAIR TEST
Visionary Studios is shown a striking image from Generator A and a disappointing image from Generator B. Should it choose A? Not yet: the images might use different briefs, different effort budgets or carefully selected results.
A fair comparison gives each option a suitable opportunity to meet the same relevant needs, using a documented procedure. Its purpose is to support a defensible decision, rather than produce a preferred winner.
The generators, tests and numerical results on this page are fictional classroom examples, not findings from the case-study booklet.
02 | START WITH THE QUESTION
Decide what you want to compare. “Which model generates a candidate faster?” differs from “Which workflow delivers suitable campaign assets with less total effort?”
The first might isolate generation latency under defined conditions. The second should include prompting, retries, review and editing. Mixing the questions can make a narrow result sound like a broad business conclusion.
For the studio, use the criteria developed in 08.1 and state which decision the experiment is intended to support.
03 | DECIDE WHAT THE COMPARISON INCLUDES
A test of complete workflows can allow each tool’s appropriate prompting or editing features. A tightly controlled capability test may hold more inputs fixed to investigate a particular difference.
Neither approach is automatically wrong, but the conclusion must match the design. If A includes expert editing and B does not, you are not isolating the generators alone.
Define the boundary in advance: model only, configured tool, or complete production process. Record any differences that remain.
04 | USE THE SAME CREATIVE REQUIREMENTS
Both options should address equivalent briefs: the same subject, intended style, identifying features and required output use. Testing A on a simple landscape and B on a complex character scene is an uneven comparison.
Exactly identical wording can be useful for a common-prompt test, but tools may respond to different formats. For a practical workflow comparison, permit equivalent tool-appropriate wording under a stated policy and record the changes.
This distinguishes equivalent requirements from a claim that every prompt must always be character-for-character identical.
05 | VARIABLES: WHAT CHANGES AND WHAT STAYS COMPARABLE?
The independent variable is the factor deliberately varied, such as generator choice. A dependent variable is an observed outcome, such as accepted-output rate or production time.
Control variables are conditions held constant where relevant and practicable, such as the brief, output dimensions and review rubric.
Some differences cannot be removed, especially between hosted services. Document them and limit the claim rather than pretending every condition is identical.
06 | CHOOSE REPRESENTATIVE BRIEFS
Use a planned set reflecting the studio’s intended work: advertising scenes, concept ideas and character variations where relevant. Include a sensible range of difficulty.
Choose the briefs before examining the final results. Selecting only tasks where your favourite tool performs well creates selection bias.
A small classroom test need not cover everything. State which work it represents and which important tasks remain untested.
07 | REPEAT GENERATION AND COUNT THE FAILURES
Image generation involves randomness, so one output may not represent typical performance. Generate multiple candidates for each brief using a defined number of attempts or resource budget.
Keep all outcomes in the record, including unsuitable candidates and errors. A showcase of one excellent result hides the effort needed to obtain it.
Repeated outputs help reveal variation within the tested setup. They do not turn a narrow set of briefs into evidence for every possible task.
08 | SET A FAIR EFFORT BUDGET
If A gets 30 attempts and B gets 3, the best image from A has had a different opportunity to succeed. State the allowed attempts, prompting time and editing effort.
You could compare a fixed number of candidates, or ask each workflow to deliver within the same production-time budget. These answer different questions, so choose a design matching the company’s need.
Record unsuccessful attempts and any extra intervention. Equal budgets should be defined meaningfully, not assumed from the final image count.
09 | SETTINGS AND SEEDS NEED CARE
Record model versions, relevant settings, dimensions and any seed controls available. A seed helps initialise random generation within a particular implementation.
The same numerical seed does not guarantee equivalent noise, composition or difficulty across different generators. Do not treat matching seed numbers as proof that a cross-model comparison is controlled.
Use repeated runs and documented settings. If a provider changes the model, record that version change and consider whether earlier results remain comparable.
10 | HARDWARE AND HOSTED SERVICES
Local generation speed depends partly on hardware and configuration. Hosted response time may also include network delay, queueing and provider load.
If the decision concerns the studio’s actual experience, those delays are relevant and should be measured. If the claim concerns model computation alone, an end-to-end stopwatch cannot isolate it.
Record test conditions, time of testing and relevant device details. Repeat where appropriate and acknowledge conditions outside your control.
11 | AGREE THE RUBRIC BEFORE SCORING
Use descriptors connected to the brief: identity preserved, requested style matched and unacceptable artefacts absent. Apply the same acceptance conditions to both workflows.
Ask reviewers to score independently before discussing differences. Where practicable, hide generator names and mix presentation order to reduce expectations influencing judgement.
Blinding cannot remove every bias: distinctive output styles may reveal the source. Explain what the procedure reduces and what it cannot guarantee.
12 | RECORD A COMPLETE RESULT
| Field | Purpose |
|---|---|
| Brief ID, tool and version | Identify the task and system tested |
| Prompt, settings and attempt number | Document the generation procedure |
| Generation and editing time | Expose effort beyond the selected output |
| Rubric result and rejection reason | Connect judgement to evidence |
| Reviewer and unresolved concerns | Support review and accountability |
Use appropriate access controls for client materials. A useful test record need not be a public collection of confidential prompts.
13 | WORKED COMPARISON: ACCEPTED OUTPUTS
| Measure | Workflow A | Workflow B |
|---|---|---|
| Candidates generated | 40 | 40 |
| Candidates meeting the agreed checks | 24 | 30 |
| Accepted proportion | 60% | 75% |
| Total production time | 240 minutes | 330 minutes |
| Time per accepted asset | 10 minutes | 11 minutes |
A: 24 ÷ 40 × 100 = 60%. B: 30 ÷ 40 × 100 = 75%. B is 15 percentage points higher on this measure.
A uses 240 ÷ 24 = 10 minutes per accepted asset; B uses 330 ÷ 30 = 11 minutes. B has a higher accepted proportion, while A takes less time per accepted asset in this pilot. Which matters more depends on the agreed requirements.
14 | AVERAGES CAN HIDE DIFFERENCES
A total accepted rate might hide a serious weakness in character consistency. Break down results by relevant brief type rather than assuming the overall winner is best at every task.
Time measurements can also vary. For example, 2, 2, 2, 2 and 12 minutes have a mean of 4 minutes and a median of 2 minutes. The median describes a typical middle observation; the mean reflects the long wait in the total.
Report enough information to explain variation. Neither statistic alone tells the whole operational story.
15 | AVOID CONFOUNDING AND CHERRY-PICKING
A confounding factor is an additional influence that prevents you from attributing an observed difference cleanly to the factor being investigated. Different editing expertise could affect an apparent generator comparison.
Cherry-picking means selecting favourable evidence while omitting relevant results. Reporting only A’s best image and B’s worst image is a clear example.
Plan the test, keep the full record and describe remaining differences. If your design compares complete workflows, identify expertise as part of that workflow rather than claiming it was isolated away.
16 | STATE THE LIMITS WITHOUT DISCARDING THE EVIDENCE
Ten briefs can provide useful pilot evidence but may not represent all future commissions. Forty candidates from those briefs are not forty independently chosen business scenarios.
A result may also depend on the tested version, settings, reviewer rubric and production team. Use wording such as “in this pilot” and identify what needs further testing.
Limited evidence can justify a larger trial or a task-specific decision. It does not require either universal confidence or the claim that nothing has been learned.
17 | WRITE A FAIR COMPARISON CONCLUSION
Report the relevant difference, connect it to a company need and mention the important limitation. Avoid declaring one tool “best” without saying best for what.
A recommendation should also respect essential approval conditions. Performance scores do not settle every responsible-use concern.
18 | DESIGN YOUR OWN COMPARISON
Plan a comparison for Visionary Studios using representative briefs, a defined effort budget, agreed criteria and a complete result record. Explain what you will control and what cannot be held constant.
Reveal a possible plan
Choose relevant briefs in advance and give both workflows equivalent requirements. Use a documented prompting policy and equal candidate budgets, recording all attempts, settings and versions. Apply the same rubric with independent reviewers and hide source names where practicable.
Measure accepted proportions and full production time, then inspect variation by brief. Report limits such as provider conditions and sample coverage. The numerical pilot here is fictional; use these independent teaching notes with the official 2027 booklet.
CHECK YOUR UNDERSTANDING | CLASSIFY THE TEST DESIGN ELEMENT