01 | A PROGRAM ASKING ANOTHER PROGRAM

A web API is an interface through which one program asks a service for data or an action. For example, a library app may request a catalogue page rather than scrape the words shown on a website.

HTTP describes requests and responses. A URL identifies the destination; an endpoint is a particular interface location. GET conventionally retrieves a representation. Your Python program is the client, and the responding program is the server.

Use documented APIs and respect their access terms. This lesson includes a tiny local practice server so the examples do not depend on a public service remaining available.

02 | READ THE RESPONSE IN LAYERS

A response has a status code, headers and a body. A status such as 200 indicates a successful HTTP response. 404 means the requested resource was not found; 401 concerns authentication, 403 refusal, 429 rate limiting and 5xx codes server-side problems.

JSON is a text format for structured data. response.json() decodes a JSON body into Python values such as dictionaries and lists. It does not certify that the fields have the meaning or shape your application expects.

Follow the layers: connection succeeds, HTTP status is acceptable, JSON decodes, and data meets your expected schema. A 200 response with broken JSON or wrong fields is still unusable for this application.

03 | PREPARE THE LOCAL API LAB

Download and extract the lab below. Activate a .venv and install requirements.txt from 07.2. In one terminal run server.py. Keep that terminal open while a second terminal runs client.py using the same environment.

The server listens only on 127.0.0.1, your own computer, on port 8765. The HTTP URL is deliberately loopback-only for this exercise; use HTTPS and normal certificate verification for public APIs. Do not publish this teaching server as a production service.

The main endpoint is /api/books. Other paths deliberately return an HTTP error, broken JSON or a wrong-shaped JSON value. Ctrl+C stops the server. If the port is already used, stop the conflicting practice process or change both server and client configuration.

TERMINAL | INSIDE THE EXTRACTED LAB

python -m pip install -r requirements.txt
python server.py

TERMINAL | SECOND TERMINAL, SAME ENVIRONMENT

python client.py

04 | A FIRST CHECKED GET REQUEST

Install requests before running this example and start the local server. params supplies query-string values without hand-building a URL. page=1 requests the first catalogue page.

timeout=(3, 10) supplies a connection timeout and a read timeout in seconds. The read value limits waiting for incoming data; it is not a strict total deadline for the whole request or paginated operation.

raise_for_status() raises for HTTP error responses such as 404 and 503. The context manager closes the response. The next sections add payload checks and error handling, because this short happy-path example assumes the lab’s normal response.

import requests

with requests.get("http://127.0.0.1:8765/api/books", params={"page": 1}, timeout=(3, 10)) as response:
    response.raise_for_status()
    data = response.json()

for book in data["items"]:
    print(book["id"], book["title"])

1 Python Adventures
2 Robots in Action

05 | VALIDATE THE SHAPE YOU USE

A schema is the expected structure and rules for data. This client expects a dictionary with items and next_page, a list of book records, positive integer IDs and non-blank text titles. next_page must be None or a positive integer.

Check types before indexing or looping over unknown structures. A list-shaped field is different from a string, even though both are iterable. Reject Boolean IDs under the exact-integer policy.

The lab client puts these rules in validate_page. Validation tells us a record fits the agreed shape; it does not prove the book title is truthful, the server is authorised or the information is up to date.

def validate_page(data):
    if type(data) is not dict or set(data) != {"items", "next_page"}:
        raise ValueError("Expected items and next_page fields.")
    if type(data["items"]) is not list:
        raise ValueError("items must be a list.")
    for item in data["items"]:
        if type(item) is not dict or set(item) != {"id", "title"}:
            raise ValueError("Each item needs id and title.")
        if type(item["id"]) is not int or item["id"] < 1:
            raise ValueError("Use positive integer IDs, not Booleans.")
        if type(item["title"]) is not str or not item["title"].strip():
            raise ValueError("Use non-blank title text.")
    next_page = data["next_page"]
    if next_page is not None and (type(next_page) is not int or next_page < 1):
        raise ValueError("next_page must be a positive integer or null.")
    return data

06 | HANDLE DIFFERENT FAILURES CLEARLY

RequestException covers many network and HTTP-related Requests failures. Requests’ JSONDecodeError also participates in that exception hierarchy, so catch it first if you want a separate malformed-JSON message.

Then catch other RequestException failures, followed by ValueError from your own shape checks. Distinct feedback helps users know whether a service was unavailable or its returned data was unsuitable.

The bundled main function follows this order. It prints a report only after the full collection passes validation, so a failed later page does not produce a misleading complete result.

PYTHON | EXCEPTION ORDER IN THE LAB

try:
    books = fetch_all(url)
except requests.exceptions.JSONDecodeError:
    print("The service returned invalid JSON")
except requests.exceptions.RequestException:
    print("The request or HTTP status failed")
except ValueError as error:
    print("Unexpected data:", error)

07 | FETCH MORE THAN ONE PAGE

Pagination divides a result collection into manageable pages. The lab gives two books on page one and one on page two. next_page=None marks completion. Do not assume the first page represents every record.

fetch_all follows the server’s next_page value, records visited pages and stops after a configured maximum. A repeated page would otherwise create a loop. It also rejects repeated item IDs so a duplicate record does not silently masquerade as another book.

Real APIs may use cursors, next links or different page rules. Read their contract rather than applying this page-number scheme universally. If following links, validate the destination before forwarding credentials.

"""Consume the controlled API. Install requests in your project environment."""
import requests


def validate_page(data):
    if type(data) is not dict or set(data) != {"items", "next_page"}:
        raise ValueError("Expected items and next_page fields.")
    if type(data["items"]) is not list:
        raise ValueError("items must be a list.")
    for item in data["items"]:
        if type(item) is not dict or set(item) != {"id", "title"}:
            raise ValueError("Each item needs id and title.")
        if type(item["id"]) is not int or item["id"] < 1:
            raise ValueError("Use positive integer IDs, not Booleans.")
        if type(item["title"]) is not str or not item["title"].strip():
            raise ValueError("Use non-blank title text.")
    next_page = data["next_page"]
    if next_page is not None and (type(next_page) is not int or next_page < 1):
        raise ValueError("next_page must be a positive integer or null.")
    return data


def fetch_page(url, page=1):
    with requests.get(url, params={"page": page}, timeout=(3, 10)) as response:
        response.raise_for_status()
        return validate_page(response.json())


def fetch_all(url, max_pages=10):
    if type(max_pages) is not int or max_pages < 1:
        raise ValueError("Use a positive page limit.")
    page = 1
    seen_pages = set()
    seen_ids = set()
    items = []
    while page is not None:
        if page in seen_pages or len(seen_pages) >= max_pages:
            raise ValueError("Repeated page or page limit exceeded.")
        seen_pages.add(page)
        data = fetch_page(url, page)
        for item in data["items"]:
            if item["id"] in seen_ids:
                raise ValueError("Duplicate item ID across the response pages.")
            seen_ids.add(item["id"])
            items.append(item)
        page = data["next_page"]
    return items

08 | FOLLOW A RESPONSE THROUGH THE CHECKS

This model shows where each reply fails. It uses fixed examples and makes no network request from this page. Compare successful HTTP delivery with usable application data.

09 | CREDENTIALS, RATE LIMITS AND RETRIES

Some services require an API key or token. Keep it outside public source and Git history, for example in a process environment variable. A secret embedded in downloadable browser JavaScript is visible to users. Never send a key to an unverified endpoint.

Rate limits protect services from excessive traffic. Cache where appropriate and follow documented limits and Retry-After instructions. Do not retry every failure in a tight loop.

A limited retry of an idempotent GET may be appropriate for a temporary failure. An action that creates or charges something requires greater care because repeating it may duplicate the action. This beginner lab reports failures rather than implementing a hidden retry policy.

10 | TEST WITHOUT DEPENDING ON THE INTERNET

The bundled local server deliberately supplies valid pages, 503 errors, malformed JSON and wrong-shaped data. Run those cases and record which layer rejects each one. The downloaded test_api.py also supplies 12 repeatable test methods; run python -m unittest discover -s . -v in the lab folder. Also stop the server and observe connection failure.

A fixed test response makes debugging reproducible. Tests should distinguish an empty valid list from an unavailable service and a missing field. Do not turn every failure into an empty list, because that tells the user the collection is genuinely empty.

Try it: run client.py, then point its URL at /api/error, /api/broken and /api/wrong-shape one at a time. Restore the working URL after testing.

PUT YOUR SKILLS TO WORK

Use the Section 7 challenges to practise these ideas independently. Work in your Python editor, predict results and keep a short test record.