Defensive Programming with Files and External Data

Files come from outside your program. You wrote them, or an earlier run of your program wrote them, or someone else handed them to you. The contents are not guaranteed to be what you expect, and the file system is not guaranteed to be in the state you assume.

The Boundary Principle

When data comes in from outside — a file, the user, a network request — the right time to check it and clean it up is at the boundary: the moment it enters your program. Once it is past the boundary, the rest of your code can assume it is well-formed.

For instance, if we would like to read sessions from a CSV file, we can write a function load_sessions that reads the file, checks for problems, and returns a clean list of sessions. The rest of the program can then work with that list without worrying about malformed rows, non-numeric minutes, or any of the other things that can go wrong in a file:

This first version focuses on malformed data rows. Later in the chapter, we also handle missing and empty files.

def load_sessions(path):
    sessions = []
    with open(path, encoding="utf-8") as f:
        reader = csv.reader(f)
        next(reader)        # header
        for row in reader:
            if len(row) != 3:
                continue    # skip malformed rows
            try:
                minutes = int(row[1])
            except ValueError:
                continue    # skip rows where minutes is not a number
            sessions.append([row[0], minutes, row[2]])
    return sessions

Here, load_sessions is the boundary. It deals with problems like short rows and non-numeric minutes, and returns a clean list. The rest of the program can then work with sessions without checking every row every time it uses one.

The boundary principle is how real software works. Input/output operations should be pushed to the edges of the program, and the core logic should be able to assume that the data it is working with is valid.

Messy Text

Real files contain things that clean code examples do not show. Some of them have nothing to do with malicious data and everything to do with the fact that humans, editors, and other programs write files in slightly different ways:

• Trailing whitespace. Lines that end with one or more spaces before the newline. "Math,45 \n" instead of "Math,45\n". Strip it.
• Blank lines in the middle of a file. An empty line, or a line with only whitespace, between data rows. Skip blanks.
• Inconsistent casing. "Math", "math", and "MATH" are different strings, even though they probably mean the same subject. Lowercase before comparing, if that matches your intent.
• Empty vs. missing fields. A row with three fields where one of them is the empty string — Math,,2026-05-10 — vs. a row with only two fields. Both arrive as rows from csv.reader, but they are different shapes. Decide what you do about each.

None of this requires new syntax. The new part is building a habit of expecting these cases:

for row in reader:
    if len(row) == 0:
        continue            # blank line
    if len(row) != 3:
        print(f"Skipping malformed row: {row}")
        continue
    subject = row[0].strip()
    if subject == "":
        continue            # empty subject
    try:
        minutes = int(row[1].strip())
    except ValueError:
        print(f"Bad minutes value in row: {row}")
        continue
    sessions.append([subject.lower(), minutes, row[2].strip()])

The above is more cautious than the example in the CSV chapter. It anticipates several kinds of dirty input and decides what to do about each. The decisions are what makes it defensive.

Missing vs. Empty vs. Corrupt

Three things can be wrong with a file you are trying to read, and they need different responses:

• The file is missing. FileNotFoundError. The file does not exist at the path you asked for. For the project’s sessions.csv, this is the first-run case — not an error, just “no data yet”.
• The file is empty. It exists, but it contains no bytes. csv.reader returns no rows; f.read() returns "". Usually you treat this the same way as a header-only file: zero sessions, nothing to do.
• The file is corrupt. It exists, has content, but the content is not what you expected. A CSV with broken quoting. A header that says subject,minutes,date but data rows with four fields. Bytes that do not decode as UTF-8.

These are three different problems, and treating them the same is a bug. The first is often expected (no file means no data yet); the second is usually fine (empty file means no data this time); the third is usually a real problem that the user needs to know about. Conflating them — catching all three with one except Exception: print("error") — hides the cases the user actually needs to see.

For a CSV file with a header, the empty-file case often appears when you try to skip the header:

reader = csv.reader(f)
try:
    next(reader)        # skip header
except StopIteration:
    return []           # empty file: no rows to load

In the CSV chapter, we used plain next(reader) because the example assumed a normal file with a header. Defensive code cannot assume that. If an empty file should mean “no data yet”, catch StopIteration and return an empty result deliberately.

The rule from the last part for catching exceptions applies also here: catch specifically, do something meaningful, and never silently swallow:

try:
    sessions = load_sessions("sessions.csv")
except FileNotFoundError:
    print("No sessions file yet -- starting fresh.")
    sessions = []
except UnicodeDecodeError:
    print("sessions.csv is not valid UTF-8 -- check the file.")
    raise   # don't try to recover from this

The first except recovers. The second reports and re-raises — the program cannot meaningfully continue with a corrupt file, and the user needs to see the failure.

name,days

name,days
Ada,5

Linus,3
Grace,not-yet
,7
Too,Many,Fields
  Katherine  ,  8

[["Ada", 5], ["Linus", 3], ["Katherine", 8]]

Paths Across Operating Systems

One subtle source of problems is paths. macOS and Linux use forward slashes as the separator (folder/subfolder/file.txt); Windows uses backslashes (folder\subfolder\file.txt). Python is reasonably forgiving, but code that builds paths by hand can still break in surprising ways when moved between systems:

path = "data" + "/" + "sessions.csv"  # works on macOS/Linux
                                      # works on most Windows setups
                                      # not portable in general

The robust answer is pathlib, a standard-library module for paths. The shape is small:

from pathlib import Path

path = Path("data") / "sessions.csv"
print(path)        # data/sessions.csv  (or data\sessions.csv on Windows)

with open(path, encoding="utf-8") as f:
    ...

Path(...) takes a string. The / operator (overloaded for Path — it does not divide anything) joins path pieces using whatever separator the operating system uses. The resulting Path works almost everywhere a string path does: open(path, ...) accepts it, file functions accept it. You do not have to think about which slash to use.

This course does not lean heavily on pathlib, but if you find yourself building paths by hand, this is the better tool. The rule is small: do not hardcode / or \ in your paths.

Path construction is usually just the first step. Once you can point to the file portably, the same boundary principles still apply: the file might be missing, empty, malformed, or full of values that need conversion before the rest of the program can trust them.

Encoding Errors

The reading text files chapter introduced encoding="utf-8". The chapter on data formats explained why it matters: a file in text mode is interpreted byte-by-byte as characters in some encoding, and if the bytes do not match the encoding, the read fails.

The specific exception is UnicodeDecodeError:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

This means: “I tried to read this file as UTF-8, but byte 0x89 at position 0 is not a valid start of a UTF-8 character.” The two most common causes:

• The file is in a different text encoding (often latin-1 or windows-1252 for older files from European or American systems). You can pass encoding="latin-1" to read it.
• The file is not text at all — it is a PNG, ZIP, or other binary file. No text encoding will make it decode correctly.

For the study log project, both cases are unlikely — your code wrote the file, in UTF-8, and your code reads it back, in UTF-8. The error mostly arises with files from elsewhere. When it does, the message itself is informative: it tells you which byte failed and at which position, which is often enough to spot the cause.

Fail Loudly, or Recover Deliberately

Never silently swallow an actual problem. Either recover from it in a way that you can defend (skip the row, print a warning, use a default), or fail loudly so the problem reaches a human.

The opposite is the swallow:

try:
    sessions = load_sessions("sessions.csv")
except:
    sessions = []

The bare except: here catches everything. Every problem produces the same response, which is “pretend nothing was there.” The user never sees what went wrong. Their data may be gone, and they will not know until they try to retrieve it.

Failing loudly preserves the original problem. Recovering deliberately makes the choice explicit. Swallowing exceptions hides both.

try:
    rows = load_attendance("attendance.csv")
except:
    rows = []

Check Your Understanding

1. What is the boundary principle? Where in your code should you validate external data?
2. Give three examples of “messy text” that real files contain.
3. Missing, empty, and corrupt are three different things that can be wrong with a file. Why is it a bug to treat them all the same way?
4. Why is hardcoding a / in a file path not portable? What is the alternative?
5. What does a UnicodeDecodeError mean?