01 | TEST RELIABILITY, NOT JUST POSSIBILITY

A successful run proves that the route worked once under those conditions. It does not show how often it will work when placement, battery, temperature or grip vary.

Repeated trials measure observed performance under a defined test protocol. Keep the code version, setup and success criteria clear so the result is meaningful.

Use the logs from 10.2 and the action outcomes from 10.3, then add independent physical observations of endpoint, heading and scoring.

02 | DEFINE SUCCESS BEFORE THE FIRST TRIAL

Write a pass rule before collecting results. Include every outcome that matters to the route, with measurable tolerances.

The script requires all these conditions. COMPLETE is only an execution label: a completed route that misses its endpoint still fails the overall test.

Do not loosen the criteria after seeing the results and then present the new rate as if the original test passed. If criteria change, label and recompute the analysis transparently.

03 | KEEP ONE ROW PER ATTEMPT

A trial is one complete attempt at the defined route. Fifty telemetry samples from that attempt are not fifty independent trials.

The complete script uses a compact CSV schema. Add temperature, battery voltage, start-pose and log references to your full trial record. Do not discard that context when reducing the data for analysis.

04 | KEEP FAILURES IN THE DENOMINATOR

Observed success rate is successful eligible trials divided by all eligible trials. Eight successes, one timeout and one mechanism fault give 8 / 10 = 80%, not 8 / 8 = 100%.

The script counts TIMEOUT, FAULT and CANCELLED as failures. If an external interruption should be excluded under a rule written before testing, mark it EXCLUDED and record the reason. Always report exclusion counts.

Equipment faults that are part of the robot’s real performance should not disappear as “bad tests”. Explain any boundary between setup error and system failure before the trial series.

05 | DO NOT HIDE UNKNOWN OUTCOMES

If a completed run has no heading or endpoint measurement, success may be unverified. The script labels that trial UNKNOWN unless another known criterion already proves failure.

Report confirmed successes, confirmed failures and unknown outcomes separately. For eight successes, one failure and one unknown, the verified-success fraction is 80%; the possible overall success fraction lies between 80% and 90% depending on the missing result.

Those missing-data bounds are not a confidence interval. The script suppresses its binary Wilson interval while unknown outcomes remain. Resolve the missing evidence or clearly report the limitation.

06 | A SMALL PERFECT SAMPLE STILL HAS UNCERTAINTY

A confidence interval describes uncertainty in the underlying success proportion under the statistical model. A small perfect sample still has a wide interval.

A 95% method has roughly 95% coverage across repeated samples under its assumptions. It does not mean this fixed interval contains the true value with a 95% probability, and it does not guarantee the next run.

The calculation assumes trials approximate independent binary outcomes with a stable success probability. Systematic placement error, changing conditions and correlated runs are not fixed by collecting more rows.

07 | COMPARE VERSIONS WITH MATCHED CONDITIONS

Version A tested cold on a fresh battery and version B tested after ten demanding runs is an unfair comparison. Both version and operating conditions changed.

Use matched blocks: test both versions with comparable placement, surface, load and initial thermal/battery conditions. Alternate or randomise order where practical, while still restoring the required starting setup.

Keep code versions and parameter values fixed within each comparison. If a repair or recalibration changes the robot, record it as a new condition rather than silently pooling the results.

The script lists outcomes by block descriptively. It does not perform a paired significance test. Overlapping confidence intervals alone do not prove equality, and a higher observed rate alone does not prove an improvement.

08 | REPORT ERRORS, TIME AND POINTS AS WELL

A binary pass rate compresses a lot of information. Report endpoint-error spread, heading error, duration and verified points alongside it.

Successful-runs-only timing can look fast because slow failures were removed. The script summarises all available eligible measurements and prints their count. Missing measurements can still bias that summary.

Inspect run order and condition groups. A route that degrades as motors warm needs investigation even if its pooled success rate looks acceptable.

09 | COMPLETE COMPUTER-BASED CSV ANALYSER

This script runs on a computer with Python 3, not on the V5 Brain. Save the code as analyze_trials.py. It uses only the Python standard library.

python analyze_trials.py
python analyze_trials.py trials.csv

Without a filename, it analyses labelled illustrative data. With a filename, it reads one row per physical trial using the header shown in DEMO_CSV. Empty, NA and UNKNOWN numeric entries are treated as missing.

# =============================================================================
# REPEATED-TRIAL ANALYSIS | RUN ON A COMPUTER WITH PYTHON 3, NOT THE V5 BRAIN
# =============================================================================
# python analyze_trials.py              -> illustrative built-in data
# python analyze_trials.py trials.csv   -> your one-row-per-trial dataset
import csv
import io
import math
import sys
from collections import Counter

# Freeze these criteria BEFORE collecting/comparing the trials.
MAX_DURATION_MS = 18000.0
MAX_POSITION_ERROR_MM = 20.0
MAX_HEADING_ERROR_DEG = 3.0
MIN_VERIFIED_POINTS = 5  # Illustrative target; adapt to current scoring rules.
REQUIRED = ('trial_id','version','block','battery_id','surface','load','outcome',
            'duration_ms','position_error_mm','heading_error_deg','verified_points','reason')
OUTCOMES = {'COMPLETE','TIMEOUT','FAULT','CANCELLED','EXCLUDED'}
DEMO_CSV = '''trial_id,version,block,battery_id,surface,load,outcome,duration_ms,position_error_mm,heading_error_deg,verified_points,reason
A01,A,1,B1,tiles,normal,COMPLETE,12000,10,1,5,
B01,B,1,B1,tiles,normal,COMPLETE,12800,8,1,5,
A02,A,2,B2,tiles,normal,COMPLETE,12500,12,2,5,
B02,B,2,B2,tiles,normal,COMPLETE,13000,9,1,5,
A03,A,3,B1,tiles,normal,TIMEOUT,18000,NA,NA,2,collection_timeout
B03,B,3,B1,tiles,normal,COMPLETE,12900,35,2,5,endpoint_miss
A04,A,4,B2,tiles,normal,COMPLETE,12300,15,2,5,
B04,B,4,B2,tiles,normal,COMPLETE,13100,7,1,5,
A05,A,5,B1,tiles,normal,COMPLETE,12400,11,1,5,
B05,B,5,B1,tiles,normal,COMPLETE,13050,10,1,5,
'''

def optional_number(text,field,row_id):
    text=text.strip()
    if text.upper() in ('','NA','UNKNOWN'):return None
    value=float(text)
    if not math.isfinite(value):raise ValueError(row_id+': non-finite '+field)
    if field!='heading_error_deg' and value<0:
        raise ValueError(row_id+': negative '+field)
    if field=='verified_points' and value!=int(value):
        raise ValueError(row_id+': points must be whole numbers')
    return value

def read_trials(stream):
    reader=csv.DictReader(stream)
    if (reader.fieldnames is None or not set(REQUIRED).issubset(reader.fieldnames)
            or len(set(reader.fieldnames))!=len(reader.fieldnames)):
        raise ValueError('Missing required CSV columns')
    rows=[]
    seen=set()
    for raw in reader:
        if None in raw or any(raw.get(k) is None for k in REQUIRED):
            raise ValueError('Malformed CSV row')
        r={k:raw[k].strip() for k in REQUIRED}
        if not r['trial_id'] or not r['version']:
            raise ValueError('Trial ID and version are required')
        if r['trial_id'] in seen:raise ValueError('Duplicate trial: '+r['trial_id'])
        seen.add(r['trial_id'])
        if r['outcome'] not in OUTCOMES:raise ValueError('Unknown outcome: '+r['outcome'])
        if r['outcome'] in ('TIMEOUT','FAULT','CANCELLED','EXCLUDED') and not r['reason']:
            raise ValueError(r['trial_id']+': record an outcome/exclusion reason')
        for field in ('duration_ms','position_error_mm','heading_error_deg','verified_points'):
            r[field]=optional_number(r[field],field,r['trial_id'])
        rows.append(r)
    return rows

def classify(r):
    if r['outcome']=='EXCLUDED':return 'EXCLUDED'
    if r['outcome']!='COMPLETE':return 'FAIL'
    values=[r['duration_ms'],r['position_error_mm'],r['heading_error_deg'],r['verified_points']]
    checks=[None if values[0] is None else values[0]<=MAX_DURATION_MS,
            None if values[1] is None else values[1]<=MAX_POSITION_ERROR_MM,
            None if values[2] is None else abs(values[2])<=MAX_HEADING_ERROR_DEG,
            None if values[3] is None else values[3]>=MIN_VERIFIED_POINTS]
    if any(c is False for c in checks):return 'FAIL'
    if any(c is None for c in checks):return 'UNKNOWN'
    return 'SUCCESS'

def wilson(successes,n,z=1.959963984540054):
    if not isinstance(n,int) or not isinstance(successes,int) or n<0 or not 0<=successes<=n:
        raise ValueError('Need integer 0 <= successes <= n')
    if n==0:return None
    p=successes/n
    d=1+z*z/n
    center=(p+z*z/(2*n))/d
    half=z*math.sqrt(p*(1-p)/n+z*z/(4*n*n))/d
    return max(0,center-half),min(1,center+half)

def median(values):
    v=sorted(values)
    if not v:return None
    mid=len(v)//2
    return v[mid] if len(v)%2 else (v[mid-1]+v[mid])/2

def summarize(rows):
    eligible=[r for r in rows if classify(r)!='EXCLUDED']
    counts=Counter(classify(r) for r in eligible)
    n=len(eligible)
    s,u=counts['SUCCESS'],counts['UNKNOWN']
    return {'n':n,'successes':s,'failures':counts['FAIL'],'unknown':u,
            'excluded':len(rows)-n,
            'verified_fraction':s/n if n else None,
            'missing_bounds':(s/n,(s+u)/n) if n else None,
            # Do not present missing outcomes as a complete binary sample.
            'wilson95':wilson(s,n) if n and not u else None,
            'reasons':Counter(r['reason'] or 'criteria_not_met' for r in eligible if classify(r)=='FAIL')}

def report(rows):
    for version in sorted({r['version'] for r in rows}):
        group=[r for r in rows if r['version']==version]
        summary=summarize(group)
        print('\nVERSION',version,summary)
        eligible=[r for r in group if classify(r)!='EXCLUDED']
        for field in ('duration_ms','position_error_mm','heading_error_deg','verified_points'):
            available=[r[field] for r in eligible if r[field] is not None]
            if field=='heading_error_deg':available=[abs(v) for v in available]
            print(field,'available n',len(available),'of',len(eligible),
                  'median',median(available),'min',min(available) if available else None,
                  'max',max(available) if available else None)
    print('\nMatched-block descriptive results (not a significance test):')
    for block in sorted({r['block'] for r in rows if r['block']}):
        print(block,[(r['version'],r['trial_id'],classify(r)) for r in rows if r['block']==block])
    print('\nCheck independence, stable conditions, missing data and the trial protocol before interpreting intervals.')

if __name__=='__main__':
    if len(sys.argv)>2:
        raise SystemExit('Usage: python analyze_trials.py [trials.csv]')
    if len(sys.argv)==2:
        with open(sys.argv[1],newline='',encoding='utf-8-sig') as stream:
            trials=read_trials(stream)
    else:
        print('ILLUSTRATIVE DATA ONLY; no physical robot trials are represented')
        trials=read_trials(io.StringIO(DEMO_CSV))
    report(trials)

The example versions both achieve 4 / 5 successes. A has a collection timeout; B has an endpoint miss. Their small samples support inspection, not a strong conclusion that either route is more reliable.

The script rejects duplicate trial IDs, invalid outcomes, malformed rows and non-finite numeric values. It cannot verify that the physical observations, exclusions or test protocol were honestly and accurately recorded.

10 | PLAN A TEST SERIES THAT CAN ANSWER THE QUESTION

1. State the route, operating band and desired reliability question.
2. Freeze success rules, exclusion rules, code and measurement methods.
3. Choose a fixed practical number of trials or a documented stopping rule.
4. Use matched conditions and planned version order.
5. Record every attempt and its conditions.
6. Analyse failures, unknowns and continuous measurements.
7. Investigate the most consequential failure mechanism.
8. Change one justified feature, then run a fresh validation series.

Do not stop solely because the latest few runs look good. Optional stopping and repeatedly checking for a favourable result can make a fixed-sample interval misleading.

There is no universal trial count that proves a VEX route is reliable. Required sample size depends on the reliability target, desired uncertainty, conditions and available time.

11 | TURN A FAILURE TABLE INTO AN ENGINEERING CHANGE

Classify observed failures consistently: collection, navigation, station verification, mechanism readiness, release, deadline, cancellation or feedback fault.

Use the trace to locate the first failure, then inspect the physical evidence. A collection timeout may come from alignment, object path, counting logic or a jam.

Choose a change linked to the evidence and predict what should improve. Retest under the same comparison protocol and preserve the old results. Keep a separate final validation set after choosing the configuration so tuning does not become its own proof.

12 | COMMON MISTAKES

• Reporting only the best run: include the full trial series.
• Counting telemetry rows as trials: use one outcome per attempt.
• Removing faults from the denominator: apply declared exclusion rules.
• Calling unknown evidence success: keep unverified outcomes visible.
• Interpreting 10 / 10 as certain reliability: report sample size and uncertainty.
• Comparing different conditions: match and document the setup.
• Pooling changing configurations: retain version and condition labels.
• Using the tuning data as final validation: test the chosen configuration again.

CHALLENGES | COUNT, COMPARE AND JUSTIFY

1. Define: write measurable success and exclusion rules for your route.
2. Record: prepare a one-row-per-trial CSV and link each trial to its telemetry.
3. Classify: show why COMPLETE with excessive endpoint error is a failure.
4. Missing data: remove one required successful-run measurement and inspect the UNKNOWN count and bounds.
5. Uncertainty: compare 10 / 10 and 20 / 20 Wilson intervals.
6. Compare: design matched A/B blocks with a documented order.
7. Investigate: connect the most common failure category to a testable engineering change.
8. Validate: run a fresh series and report all outcomes, measurement counts and limitations.

REFERENCES

• NIST: confidence intervals for proportions
• NIST: Wilson interval expressions and methods

The CSV analyser and trial-classification policy are independently written teaching examples. Statistical summaries depend on the stated assumptions and the quality of the trial record.