Persisting and Sharing Data

Validating External Data

Learning Objectives

You can distinguish file-system, text-decoding, parsing, and application-data failures.
You can raise a specific exception when a data contract is violated.
You can validate shape and types before using external records.
You can reject an invalid collection without partially changing existing state.

Part 3 separated text conversion from domain validation. The same distinction applies to files. For the text formats in this part, the program must obtain the file, decode its bytes as text, and parse that text before checking its records.

A failure can happen at several boundaries:

locate and read file → decode text → parse format → validate records → calculate

Giving all of these failures the same explanation makes recovery harder.

Report a Violated Contract with `raise`

You have caught exceptions. A function can also deliberately signal that it cannot satisfy its contract:

No files opened Select a file to edit

The accepted duration is returned and printed as 20. Change the argument to 0 to see the rejected call, then restore 20 before adding the rejection check below.

type(...) is familiar from Part 1. Comparing its result with int checks the exact type: True, 1.0, and "1" are not integers under this contract. Check the type before applying the numeric rule.

raise ValueError(...) stops normal execution of this call and reports the stated failure to its caller. It is not a printed warning followed by continued calculation.

Use a returned result for accepted data and an exception when this function’s contract says to reject it. Do not return a plausible empty result merely because rejection is inconvenient.

In the editor above, add the following after the definition. Predict whether the call returns a value or reaches the handler. To check rejection, record whether the expected exception occurs:

try:
    require_positive(0)
except ValueError:
    rejected = True
else:
    rejected = False

assert rejected == True

An else block on try runs when the try block completes without an exception. Here it records unexpected success. It is different from an else attached to an if.

Arrange a value check

0 / 4 points

Arrange the blocks into a program whose checked(value) function returns a positive integer unchanged and raises ValueError for any other value. In particular, it must reject 0, negative integers, True, and text such as "3". The complete program prints 3.

Validate and Build a Known Record

We need to check the application contract separately. For this example:

the outer value is a list;
each item is a dictionary with "subject" and "minutes";
subject is a nonempty string after surrounding whitespace is removed;
minutes is a positive integer, not a boolean or a decimal value;
extra fields are ignored when creating the application’s two-field record.

These are requirements of this application, not universal JSON rules.

def checked_session(value):
    if type(value) != dict:
        raise ValueError("A session must be a record.")

    if "subject" not in value or "minutes" not in value:
        raise ValueError("A session needs subject and minutes.")

    if type(value["subject"]) != str:
        raise ValueError("Subject must be text.")

    subject = value["subject"].strip()
    if subject == "":
        raise ValueError("Subject must not be empty.")

    minutes = value["minutes"]
    if type(minutes) != int:
        raise ValueError("Minutes must be an integer.")
    if minutes <= 0:
        raise ValueError("Minutes must be positive.")

    return {"subject": subject, "minutes": minutes}

The same exact-type check also works with dict and str. These exact checks suit the plain data produced by the parsers in this part. The integer check deliberately rejects True: a boolean is not a session duration even though Python can use it in arithmetic.

The checks proceed before the operations that depend on them. We do not look up fields before knowing there is a dictionary, or strip a value before knowing it is a string.

The function builds a new record with a trimmed subject and valid minutes. The original dictionary stays unchanged.

original = {"subject": " Math ", "minutes": 20, "note": "optional"}
assert checked_session(original) == {"subject": "Math", "minutes": 20}
assert original["subject"] == " Math "

Apply the checks to a booking record. Decide which checks must succeed before you can use its fields.

Check a booking

0 / 20 points

Write checked_booking(value). Accept a dictionary containing a name string that is nonempty after stripping and a positive integer seats value. Booleans, decimal numbers, and numeric strings are not valid seats. Return a new dictionary with exactly the trimmed name and seats. Ignore extra fields. Reject any violation with ValueError, without printing or changing the input. Error message wording is your choice.

For {"name":" Ada ","seats":2,"note":"aisle"}, return {"name":"Ada","seats":2}. For {"name":"Ada","seats":0}, raise ValueError.

Edit main.py, the only supplied file, to implement checked_booking. This exercise uses ordinary Python values and needs no data files or keyboard input.

In the course editor, add the following assertion below the definitions and Run. A passing assertion produces no output; remove temporary calls before submitting.

assert checked_booking({"name": " Ada ", "seats": 2}) == {"name": "Ada", "seats": 2}

For local practice, create your own check.py beside main.py, put from main import checked_booking at the top, and add the assertion there. Run py check.py on Windows or python3 check.py on macOS/Linux from that folder. A failed assertion displays a traceback. The function implementation belongs in main.py; check.py only calls it to test the result. Use the course editor’s Submit to submit main.py for grading and read the feedback.

Pause Before Loading Collections

Before continuing, check that you can identify the first failing check for a missing subject, a text duration, and True as a duration. Explain why a rejected record must not be repaired by returning an empty dictionary.

Validate the Whole Collection Before Replacing State

def checked_sessions(data):
    if type(data) != list:
        raise ValueError("Sessions must be a list.")

    result = []
    for value in data:
        result.append(checked_session(value))

    return result

If a later record fails, this function raises instead of returning a partly validated result. It has not changed its input. The caller should replace its live state only after receiving a complete successful result.

Practise validating a complete list using the supplied checker for individual bookings:

Check booking data

0 / 15 points

Complete checked_bookings(data), using the supplied checked_booking to validate every record. Require a list; return a new list of independent normalized bookings in input order. Empty lists are accepted, and repeated names are separate bookings. If any record is invalid, raise ValueError without returning a partial result or changing any input record. Do not print. The supplied helper accepts a nonempty trimmed name and positive integer seats, rejecting Booleans, decimals, and text seats. Keep its behavior.

Edit main.py, the only supplied file, to implement checked_bookings. Keep the supplied checked_booking helper unchanged. This exercise uses ordinary Python values and needs no data files or keyboard input.

In the course editor, add the following assertion below the definitions and Run. A passing assertion produces no output; remove temporary calls before submitting.

assert checked_bookings([{"name": " Ada ", "seats": 2}]) == [{"name": "Ada", "seats": 2}]

For local practice, create your own check.py beside main.py, put from main import checked_bookings at the top, and add the assertion there. Run py check.py on Windows or python3 check.py on macOS/Linux from that folder. A failed assertion displays a traceback. The function implementation belongs in main.py; check.py only calls it to test the result. Use the course editor’s Submit to submit main.py for grading and read the feedback.

A useful load function can now parse and then validate:

import json


def load_checked_sessions(path):
    with open(path, "r", encoding="utf-8") as file:
        data = json.load(file)

    return checked_sessions(data)

It does not catch errors because it does not decide the user’s recovery action. The caller can distinguish a missing file, undecodable text, malformed JSON, and invalid records when that distinction matters.

Before the next exercise, follow a smaller failed collection check. Predict whether current receives the partly built result. Watch the call raise before its assignment can happen.

Code

Step 1 of 13

1 / 13

Formatting...

State

main

No variables

Objects

No objects yet

Output

No output yet

The temporary result belongs to the failed call. The existing current list remains the caller’s state.

Now connect JSON loading to the supplied booking validator. Let the caller decide how to handle a failed load.

Load booking data

0 / 15 points

Complete load_bookings(path) using the supplied checked_bookings. Read one UTF-8 JSON document, validate its complete contents, and return the resulting list. Do not print or change the source. Let file errors, JSON parsing errors, and validation ValueError propagate; do not replace them with an empty result.

The supplied checker accepts lists of bookings with trimmed nonempty names and positive integer seats, rejects wrong types and invalid records, and returns independent records. A document containing [] is valid empty data; a zero-byte file is not.

Work locally

Download this exercise with the Aalto OpenCS IDE plugin, or use Download starter code below and extract the ZIP into its own folder inside programming-practice. Open that exercise folder in your IDE. Keep each exercise in a separate folder.

File	Role
`main.py`	Complete `load_bookings(path)`; keep the supplied record and list validators unchanged.
`check.py`	Supplied assertions about loading and normalizing bookings; extend them.
`bookings.json`	Sample input containing one booking with surrounding name whitespace.

Save your changes, then open the IDE terminal in this exercise folder: dir on Windows or ls on macOS/Linux should show the files above. Relative paths in the checks start from this folder. Run:

System	Run the checks
Windows	`py check.py`
macOS/Linux	`python3 check.py`

Use python instead if that is your working Python command. Successful supplied checks print Checks passed.. Read any failed assertion or traceback, fix your code, and rerun the checks. You may extend check.py; use separate data files for extra cases so the supplied samples retain their contents.

The supplied check calls load_bookings("bookings.json") and expects [{"name": "Ada", "seats": 2}]. Create separate files containing [] and damaged JSON; check that the first loads and the second raises an error.

main.py defines functions; running it alone produces no output. check.py imports and calls those functions. Keep file operations inside the functions, with test calls in check.py, so importing main.py does not read or write data files or ask for input.

Submit your solution

Save your files. Submit using Submit Current Exercise in the IDE plugin, review the file list, and confirm Submit. Alternatively, create a ZIP from the exercise folder:

System	Command
Windows	`py -m zipfile -c submission.zip main.py check.py bookings.json`
macOS/Linux	`python3 -m zipfile -c submission.zip main.py check.py bookings.json`

Upload submission.zip below and press Submit. The listed files must be at the archive’s top level, without an enclosing folder. Include the supplied support and sample files; leave out generated outputs, .venv, and __pycache__. Read the grading feedback after submitting. The grader also uses its own inputs; passing the sample checks alone does not establish the whole contract.

Locate a File Failure

A loader can fail before it reaches the record checker. A missing file differs from a file containing a valid empty list. Use a specific exception handler only when the caller knows how to recover.

Step	Example failure	Exception
Open or read	Missing file or denied access	`FileNotFoundError` or another `OSError`
Decode text	Bytes are not valid UTF-8	`UnicodeDecodeError`
Parse JSON	Text is not a JSON document	`json.JSONDecodeError`
Validate records	A duration is negative	The checker’s `ValueError`

UnicodeDecodeError and json.JSONDecodeError are kinds of ValueError. When different messages are needed, put the more specific handler before the broader one. A loader can instead let all these failures reach its caller.

The following example deliberately uses a missing filename. Predict which line is skipped:

No files opened Select a file to edit

Changing a filename’s extension does not change its bytes, encoding, or format. Check the file’s source before trying another encoding.

Locate the Failed Step

A loader expects a UTF-8 JSON list of session records. Which step fails in each case, and which earlier steps succeed?

The requested file does not exist.
The file’s bytes are not valid UTF-8.
The file contains the UTF-8 text subject,minutes.
The file contains [{"subject": "Math", "minutes": -5}].

Why would returning [] hide each problem?

Apply the Same Boundary to CSV

CSV parsing produces text fields, so normalize their types before calling the same record checker:

import csv


def read_checked_session_csv(path):
    result = []

    with open(path, "r", encoding="utf-8", newline="") as file:
        reader = csv.DictReader(file, strict=True)

        if reader.fieldnames != ["subject", "minutes"]:
            raise ValueError("Expected the header subject,minutes.")

        for row in reader:
            if None in row:
                raise ValueError("A CSV row has extra fields.")
            if type(row["subject"]) != str or type(row["minutes"]) != str:
                raise ValueError("A CSV row has missing fields.")

            converted = {
                "subject": row["subject"],
                "minutes": int(row["minutes"])
            }
            result.append(checked_session(converted))

    return result

strict=True asks the CSV parser to report certain malformed CSV syntax rather than tolerate it. It does not check our header, types, or positive-minute rule. Parser failures use csv.Error; our contract checks use ValueError.

The exact header order is a requirement of this example. A header-only file is accepted; a file without a header is rejected. A failure rejects the whole load rather than silently dropping a row.

Read an inventory

0 / 25 points

Write read_inventory_csv(path) with csv.DictReader and strict=True. Open the file with encoding="utf-8" and newline="". Require the exact header product,quantity. Each data row must contain exactly two fields. Trim surrounding product whitespace and require a nonempty product. Convert quantity with int and require a nonnegative result. Preserve row order and repeated products. Return a list of dictionaries such as {"product": "Pencils", "quantity": 12}, with exactly those two keys. Header-only input returns []. Reject missing or extra fields, wrong headers, and invalid values with ValueError; let CSV parsing errors and file errors propagate. Reject the whole load if any row fails. Do not print or modify the file. Blank lines skipped by the CSV reader are not records.

Work locally

File	Role
`main.py`	Write `read_inventory_csv(path)` here.
`check.py`	Supplied assertions about valid inventory records; add invalid cases too.
`inventory.csv`	Sample input containing pen and paper quantities.

Save your changes, then open the IDE terminal in this exercise folder: dir on Windows or ls on macOS/Linux should show the files above. Relative paths in the checks start from this folder. Run:

System	Run the checks
Windows	`py check.py`
macOS/Linux	`python3 check.py`

The supplied check expects [{"product": "pen", "quantity": 0}, {"product": "paper", "quantity": 12}] from inventory.csv. Use separate CSV files to check a wrong header and an invalid row after a valid row.

Submit your solution

Save your files. Submit using Submit Current Exercise in the IDE plugin, review the file list, and confirm Submit. Alternatively, create a ZIP from the exercise folder:

System	Command
Windows	`py -m zipfile -c submission.zip main.py check.py inventory.csv`
macOS/Linux	`python3 -m zipfile -c submission.zip main.py check.py inventory.csv`

Check Your Understanding

How does a missing file differ from a valid empty dataset?
What does a deliberate raise ValueError(...) do to the current call?
Why must the record’s shape be checked before accessing and converting its fields?
Why is successful parsing insufficient to establish positive integer minutes?
Why should live application state change only after the complete load succeeds?
How does a UnicodeDecodeError differ from a json.JSONDecodeError?

← Structured Data with JSON

Building a Data Pipeline →

Validating External Data

Learning Objectives

Report a Violated Contract with raise

Arrange a value check

Validate and Build a Known Record

Check a booking

Pause Before Loading Collections

Validate the Whole Collection Before Replacing State

Check booking data

main

Load booking data

Work locally

Submit your solution

Locate a File Failure

Apply the Same Boundary to CSV

Read an inventory

Work locally

Submit your solution

Check Your Understanding

Report a Violated Contract with `raise`