Persisting and Sharing Data

Building a Data Pipeline

Learning Objectives

You can connect loading, validation, calculation, and output through explicit function boundaries.
You can test calculations separately from file-level behavior.
You can avoid replacing output before the source has been validated.
You can verify an AI claim about a library using documentation and a controlled experiment.

Suppose the input contains two Math sessions of 20 and 15 minutes and one Programming session of 30 minutes. The report should contain Math: 35 minutes and Programming: 30 minutes, in that order.

A file-processing program can divide its work into steps. Each function receives the result of the previous step:

CSV file
    → parsed and checked session records
    → grouped totals
    → report lines
    → text file

The calculation in the middle should not need to know where the records came from.

Keep the Stages Separate

Use read_checked_session_csv and its helper checked_session from the preceding chapter. They read the CSV file and check its records before the calculations begin. This is the boundary principle: the loader establishes the record contract where external data enters the program. Calculation functions can then work with those records under that contract. Any later changes must preserve the requirements that the next function relies on.

The summarizing function uses the Part 4 pattern:

def totals_by_subject(sessions):
    totals = {}

    for session in sessions:
        subject = session["subject"]
        if subject not in totals:
            totals[subject] = 0
        totals[subject] = totals[subject] + session["minutes"]

    return totals

It receives already-valid records. It has no open, input, or exception handler because none is needed for its stated job.

The next stage creates report text:

def report_lines(totals):
    lines = []

    for subject in sorted(totals):
        lines.append(f"{subject}: {totals[subject]} minutes")

    return lines

Both functions can be tested without a file:

sessions = [
    {"subject": "Math", "minutes": 20},
    {"subject": "Programming", "minutes": 30},
    {"subject": "Math", "minutes": 15}
]
assert totals_by_subject(sessions) == {"Math": 35, "Programming": 30}
assert report_lines(totals_by_subject(sessions)) == [
    "Math: 35 minutes", "Programming: 30 minutes"
]
assert report_lines({}) == []

Write Only After Reading and Validation Succeed

def write_report(path, lines):
    text = "\n".join(lines)
    if len(lines) > 0:
        text = text + "\n"

    with open(path, "w", encoding="utf-8", newline="") as file:
        file.write(text)

The writer separates report entries with newlines and adds a newline after the final entry. An empty list produces an empty file.

Now coordinate the stages:

def create_report(source_path, report_path):
    sessions = read_checked_session_csv(source_path)
    totals = totals_by_subject(sessions)
    lines = report_lines(totals)
    write_report(report_path, lines)

The input and output must be different files. For the course examples, use practice_sessions.csv and practice_report.txt in the disposable practice folder.

Opening the output before validating the source could erase an existing report even when the input is invalid. The order above avoids that particular failure.

It is not a guarantee against every possible write failure. A storage error or interrupted process can still leave a partly written output. Use practice files and keep originals; robust replacement strategies are beyond this introductory exercise.

Trace the stages and their execution order before running the complete program.

Run the Complete Pipeline

This editor includes the reader, validator, calculation, formatter, and writer. It also creates its input file, so it needs no earlier Run or copied definitions. First predict the report. Then change the last input duration from 15 to 0. Identify which function rejects the record and why write_report is not called.

No files opened Select a file to edit

Restore the valid duration afterwards before trying the recovery code below.

Recovery Belongs Where the User’s Action Is Known

A simple caller can report failure without claiming success:

try:
    create_report("practice_sessions.csv", "practice_report.txt")
except (OSError, ValueError, csv.Error) as error:
    print(f"Report not created: {error}")
else:
    print("Report created.")

To try this recovery code, replace the final create_report(...) call and the report-reading statements in the complete editor above with this block. Keep its definitions and input-file setup. The tuple of exception types names several failures that have the same recovery action here. as error binds the caught exception to the name error so its message can be displayed inside the handler. The else runs only after successful completion of the try block.

Errors of other types, such as a NameError from a misspelled variable, still produce a traceback. A programming mistake that raises one of the caught types is handled too, so keep checking where a failure originated.

An equipment desk needs a different report: which items have been checked out longer than a chosen number of days? The next exercise supplies the file and record operations. Connect them to produce the report.

Create a checkout report

0 / 20 points

An equipment desk records how many days each item has been checked out. It needs a report of the items whose day count is greater than a chosen limit. An item at exactly the limit does not belong in the report.

Complete create_checkout_report(source_path, report_path, limit) in main.py. The two paths refer to different files, and limit is a nonnegative integer. Use the four supplied functions and keep their behavior:

Function	Contract
`read_checkout_csv(path)`	Read and validate the complete CSV file. Return a list of records such as `{"item": "Camera", "days": 9}`.
`overdue_items(records, limit)`	Return the records whose `days` exceed `limit`, in their original order. Leave the input unchanged.
`checkout_lines(records)`	Return report text for the given records, in order: `Camera: 9 days`, for example. Use `1 day` for a day count of one. Leave the input unchanged.
`write_report(path, lines)`	Replace the file with the given text entries, each followed by `\n`. An empty list creates an empty file. Return `None`.

The loader expects the exact header item,days. Each row must have two fields: an item name that is nonempty after surrounding whitespace is stripped, and an integer day count of zero or more. It preserves repeated records and their order. Quoted commas, quotation marks, and line breaks inside names remain part of the name. Blank CSV rows are ignored. A header-only file is valid; a file without a header is not. The loader raises ValueError for invalid headers or values, and lets CSV parsing and file errors propagate.

Use the caller’s limit and the results returned by the supplied functions. Return None without printing or asking for input. Do not change the source file or the records and lists passed between the functions. Read and validate the complete source before opening the report for replacement. Let failures propagate so that a caller can decide how to report them. If reading or validation fails, preserve an existing report and do not create a new one. Recovery from failures during writing is outside this task.

For example, if the CSV contains Camera,9, Tablet,6, and Tripod,8, a limit of 6 produces:

Camera: 9 days
Tripod: 8 days

If nothing exceeds the limit, replace any previous report with an empty file.

Work locally

Download this exercise with the Aalto OpenCS IDE plugin, or use Download starter code below and extract the ZIP into its own folder inside programming-practice. Open that exercise folder in your IDE. Keep each exercise in a separate folder.

File	Role
`main.py`	Complete `create_checkout_report`; keep the four supplied helper functions unchanged.
`check.py`	Supplied checks for reports and rejected input; extend them.
`checkouts.csv`	Sample input with four equipment records, including a quoted comma.

Save your changes, then open the IDE terminal in this exercise folder: dir on Windows or ls on macOS/Linux should show the files above. Relative paths in the checks start from this folder. Run:

System	Run the checks
Windows	`py check.py`
macOS/Linux	`python3 check.py`

Use python instead if that is your working Python command. Successful supplied checks print Checks passed.. Read any failed assertion or traceback, fix your code, and rerun the checks. You may extend check.py; use separate data files for extra cases so the supplied samples retain their contents.

The checks call create_checkout_report("checkouts.csv", "practice_report.txt", 6), compare the complete report, and try other limits. They also create practice_bad_checkouts.csv and check that rejection preserves an existing report. After all checks, practice_report.txt contains "Keep this report.\n". Both generated files remain available to inspect.

main.py defines functions; running it alone produces no output. check.py imports and calls those functions. Keep file operations inside the functions, with test calls in check.py, so importing main.py does not read or write data files or ask for input.

Submit your solution

Save your files. Submit using Submit Current Exercise in the IDE plugin, review the file list, and confirm Submit. Alternatively, create a ZIP from the exercise folder:

System	Command
Windows	`py -m zipfile -c submission.zip main.py check.py checkouts.csv`
macOS/Linux	`python3 -m zipfile -c submission.zip main.py check.py checkouts.csv`

Upload submission.zip below and press Submit. The listed files must be at the archive’s top level, without an enclosing folder. Include the supplied support and sample files; leave out generated outputs, .venv, and __pycache__. Read the grading feedback after submitting. The grader also uses its own inputs; passing the sample checks alone does not establish the whole contract.

Review where failures should be reported and which state a failed stage may leave behind.

A first-run booking application has a different recovery rule: a missing save starts a new list, while a damaged save must still be reported. Repair that distinction in the next exercise.

Start a booking list

0 / 15 points

Repair load_or_empty(path). A first-run booking program starts with [] only when its save file does not exist. Otherwise return the list produced by the supplied load_bookings. Invalid JSON, invalid booking data, and other file errors must propagate. Do not print, write files, or turn damaged data into an empty list. The starter supplies a complete loader and record checkers. Keep their behavior and investigate the recovery function.

Work locally

File	Role
`main.py`	Repair `load_or_empty(path)`; keep the supplied loader and validators unchanged.
`check.py`	Supplied check for an existing valid save; add missing and damaged file cases.
`bookings.json`	Sample existing save containing one valid booking.

Save your changes, then open the IDE terminal in this exercise folder: dir on Windows or ls on macOS/Linux should show the files above. Relative paths in the checks start from this folder. Run:

System	Run the checks
Windows	`py check.py`
macOS/Linux	`python3 check.py`

The supplied check only covers load_or_empty("bookings.json") returning [{"name": "Ada", "seats": 2}]. It passes even before the faulty recovery rule is repaired. Add checks with a filename that does not exist and a separate damaged JSON file: only the missing file should produce [].

Submit your solution

Save your files. Submit using Submit Current Exercise in the IDE plugin, review the file list, and confirm Submit. Alternatively, create a ZIP from the exercise folder:

System	Command
Windows	`py -m zipfile -c submission.zip main.py check.py bookings.json`
macOS/Linux	`python3 -m zipfile -c submission.zip main.py check.py bookings.json`

Make Small Files for Testing

You have tested calculations by passing them small lists. You can test a file loader in the same way: create a small file, call the loader, and check its result or the exception it raises. The next exercise supplies an inventory loader and a check file; you will create CSV files with product names and quantities to test the loader.

Keep different faults in different files. If a row has both an extra field and an invalid quantity, the first error may prevent you from checking the second. A file containing only the header also has a useful role: it checks that the loader accepts an inventory with no products.

When you have rows as lists of fields, use csv.writer. Unlike DictWriter, it receives a list for each row, including the header:

import csv

with open("practice_inventory.csv", "w", encoding="utf-8", newline="") as file:
    writer = csv.writer(file)
    writer.writerow(["product", "quantity"])
    writer.writerow(["Tape, blue", "3"])

The writer handles the comma inside the product name. You can write rows of different lengths when making a test file with a missing or extra field.

In the downloaded exercise, complete create_inventory_probes in main.py. Run py check.py on Windows or python3 check.py on macOS/Linux from the exercise folder. The supplied checks create the files and try the loader on each one. Open the five generated CSV files in your IDE afterward to compare their contents.

Create inventory test files

0 / 15 points

An inventory loader can work on one file and fail on another. Write create_inventory_probes(folder) to create five small files for checking its behavior. The supplied folder is an existing directory, given as a string. It may be a relative or an absolute path.

Each file starts with the header product,quantity. Write the following data rows after that header:

Filename	Data rows, written here as lists of CSV fields
`ordinary.csv`	`["Pencils", "12"]`, then `["Paper", "0"]`
`quoted.csv`	`["Tape, blue", "3"]`, then `["Mäki's notebook", "2"]`
`fields.csv`	`["Pencils", "12"]`, then `["Tape", "3", "spare"]`
`quantity.csv`	`["Pencils", "12"]`, then `["Markers", "many"]`
`empty.csv`	No data rows; keep the header.

Use csv.writer to write the fields, with encoding="utf-8" and newline="" when opening the file. The lists in the table describe fields; do not write their Python brackets into the CSV files. In quoted.csv, the comma belongs inside the product name. The other two unusual files each test a different problem: fields.csv has an extra field, and quantity.csv has a quantity that cannot be converted to an integer.

Create all five files inside folder, replacing their previous contents if they exist. Leave other files unchanged. Return a dictionary whose keys are the five filenames and whose values are strings giving the paths to those files. For example, the value under "ordinary.csv" must point to the ordinary file you just wrote. Use os.path.join(folder, filename) to build a path from a folder and a filename. Do not print or ask for input.

Work locally

File	Role
`main.py`	Complete `create_inventory_probes(folder)`; keep the supplied inventory loader unchanged.
`check.py`	Supplied checks that create all five CSV files and try the loader on them; extend them.

Save your changes, then open the IDE terminal in this exercise folder: dir on Windows or ls on macOS/Linux should show the files above. Relative paths in the checks start from this folder. Run:

System	Run the checks
Windows	`py check.py`
macOS/Linux	`python3 check.py`

The checks call create_inventory_probes("."), load ordinary.csv, quoted.csv, and empty.csv, and expect errors for fields.csv and quantity.csv. These five generated files remain in the exercise folder after the checks; open them to compare the valid and invalid data.

Submit your solution

Save your files. Submit using Submit Current Exercise in the IDE plugin, review the file list, and confirm Submit. Alternatively, create a ZIP from the exercise folder:

System	Command
Windows	`py -m zipfile -c submission.zip main.py check.py`
macOS/Linux	`python3 -m zipfile -c submission.zip main.py check.py`

Verify a Library Claim with Evidence

An AI assistant suggests using another CSV option, skipinitialspace=True, to remove surrounding whitespace from every field.

Use the documentation and small files to investigate that claim. Choose cases that test what the option changes, and inspect the row dictionaries before applying your own normalization. Record each file, the operation you ran, and its result.

Investigate a CSV claim

0 / 10 points

An assistant claims: “Using csv.DictReader(file, skipinitialspace=True) removes whitespace from both ends of every CSV field, including text inside quotation marks.”

Investigate this option using the official description of skipinitialspace and at least two small CSV files chosen to test different parts of the claim. Use the header subject,minutes and inspect the row dictionaries before calling .strip() or converting minutes.

Report each file’s contents, the operation you ran, and the observed fields. Compare the observations with the documented guarantee. Explain whether the option replaces the normalization needed by an application that requires nonempty subjects after trimming and positive integer minutes. Include one valid session row and its expected normalized record.

Do not rely only on another assistant response or on one successful ordinary load.

Check Your Understanding

Which function should know about CSV headers?
Why can the grouping calculation be tested without the filesystem?
Why does the source get read and checked before the report is opened for writing?
What separate evidence do documentation and the controlled experiment provide?

← Validating External Data

Files and Games →