Plausible but Wrong: Detecting AI Code That Runs but Does Not Work

This is the AI-collaboration chapter for the part. The previous three parts focused on reading, verifying, and specifying code. This part adds a fourth skill: detecting plausible-but-wrong code — code that runs without errors but is still wrong, often in ways that hide for a while.

”It Runs” Is Not “It Works”

A program runs when Python is able to execute it without raising an unhandled exception. That is a much weaker statement than “the program does what it is supposed to do.” A program can run and be wrong:

• It can read the file successfully and produce wrong output, because the parsing logic is subtly off.
• It can write the file successfully and erase data, because the wrong mode was used.
• It can handle an error successfully — by ignoring it — and produce silently corrupt results.
• It can work on the happy path and crash on the unhappy one, where “unhappy” means anything you did not test.

This gap — between “it runs” and “it works” — is wide because so much can go wrong externally without raising any exception in your own program.

A Gallery of Plausible-but-Wrong File Code

What follows is a tour through eight pieces of file code, each of which would happily run on the example you first try, and each of which is broken in some way. Reading carefully through them is one of the best ways to learn to spot the failure modes by sight.

1. Opening without with

f = open("sessions.csv")
data = f.read()

This runs. It reads the file. It also leaves the file open when the function ends — the program never calls .close(), and Python only gets around to closing it eventually, when the operating system reclaims the resource. For a small script reading one file, the program ends quickly enough that no harm comes of it. In a longer-running program, or one that opens many files, this leaks file handles, which can eventually cause the program to fail to open more files.

The fix is the with open(...) as f: shape. It closes the file as soon as the block ends, regardless of how the block ends.

2. Bare except: and except: pass

try:
    with open("sessions.csv") as f:
        data = f.read()
except:
    pass

Although the errors are caught, they are not handled, and the result is exactly the opposite of robust. Any error — missing file, broken file, bug in your own code, even keyboard interrupt — is caught and discarded. The program continues with data undefined. The next line that tries to use data fails with NameError, and the original cause is gone.

The fix is to catch specific exceptions and do meaningful recoveries.

3. "w" where "a" was meant

def add_session(subject, minutes):
    with open("sessions.csv", "w") as f:
        f.write(f"{subject},{minutes}\n")

This runs. It writes one row to the file. It also erases every previous session, because "w" empties the file before writing. The user’s previous data is gone, silently, and the program reports no error.

The fix is one character: "a" instead of "w".

4. Hand-.split(",") on quoted data

with open("sessions.csv") as f:
    for line in f:
        parts = line.strip().split(",")
        subject = parts[0]
        minutes = int(parts[1])

This works on Math,45. It fails on "History, ancient",60 — .split(",") gives back three pieces ("History, ancient", 60), parts[0] is the partial string "History, and int(parts[1]) raises ValueError because ancient" is not a number. The error type makes the cause look like a data issue (“bad minutes value”), but the actual cause is that the parser is wrong.

The fix is using csv.reader, not .split(",").

5. Hardcoded path separators

with open("data/sessions.csv") as f:
    ...

This works on macOS and Linux. It also works on most Windows setups, because Windows is forgiving of forward slashes in many places — but not all. The issue is often only visible “in the wild” and not in the developer’s environment, which happens to have a setup where it works.

The fix is Path("data") / "sessions.csv".

6. No encoding specified

with open("sessions.csv") as f:
    data = f.read()

This relies on whatever default text encoding the operating system happens to use. On most modern systems that is UTF-8, and the code works. On older systems, or systems configured differently, the code fails.

The fix is explicit: encoding="utf-8".

7. Missing newline="" on CSV write

with open("sessions.csv", "w") as f:
    writer = csv.writer(f)
    writer.writerow(["Math", "45"])
    writer.writerow(["Physics", "90"])

This works on macOS and Linux. On Windows, it produces a file with blank lines between rows — because Python adds its own newline translation on top of the one the csv module already added, and the result is two newlines for every row. The file opens in spreadsheet programs, just with blank rows between every row of data.

The fix is a tiny detail: pass newline="" when opening a file for CSV writing.

8. Assuming the file exists and is well-formed

def load_sessions():
    with open("sessions.csv") as f:
        reader = csv.reader(f)
        next(reader)
        return [(row[0], int(row[1])) for row in reader]

Take a moment to count the assumptions: the file exists, has a header row, has at least two columns per row, and the second column converts cleanly to an integer. Any one of these can fail in real use. The first run of the project (no file yet) crashes here. A user who edits the CSV by hand and accidentally introduces a blank line crashes here. A user whose subject name contains a comma and was saved without proper CSV quoting crashes here.

This is the defensive programming chapter’s point in microcosm: the boundary code has to assume that the data may not be what was promised, and behave reasonably when it is not.

The Common Thread

Each of the eight examples feature a different bug — a resource leak, a swallowed exception, a destroyed file, a wrong parser, a path issue, an encoding issue, a platform issue, and an assumption. All of them share a property: each one runs cleanly on the inputs you would naturally try first. A small file you wrote yourself. A file that already exists. A simple value with no commas. The developer’s own machine.

The bugs only appear when reality strays from those defaults — when the file is missing, when one field contains the separator, when the system is different, when the file is empty. That is what makes this category of bug both common and durable. The code passes the obvious tests because the obvious tests are the happy path. The bugs hide on the unhappy path, which the obvious tests do not visit.

The skill is to ask, every time, “what does the unhappy path look like?” and then to try the unhappy path before declaring the code working.

A Checklist for Probing File Code

When reviewing a piece of file-handling code — whether you wrote it, an AI assistant wrote it, or a colleague did — the following checklist catches most of the gallery above.

For any file code:

1. Does the file open with with? If not, where is it closed?
2. Is the encoding specified? Is it "utf-8"?
3. What happens if the file does not exist? Try it.
4. What happens if the file is empty? Try it.

For reading code, additionally:

5. What happens if a row has the wrong number of fields? Try it.
6. What happens if a numeric field has a non-numeric value? Try it.
7. What happens if a field contains the separator character? Try it.

For writing code, additionally:

8. Which mode is used — "w" or "a"? Is that the intended one?
9. For CSV writes, is newline="" present?
10. If you write the file twice, what does the file look like after the second write?

For exception handling, additionally:

11. Is every except for a specific exception type?
12. Does every except block do something meaningful, or does it just pass?
13. After the except block runs, is the program in a sensible state, or is it limping along with a problem nobody knows about?

This is more than you would run through for every piece of file code you read, but the few minutes are well spent the first time you trust a piece of generated code. Once you have run through the checklist a few times, the items become reflex — you start spotting f = open(...) without with, or a bare except:, the moment the code appears on screen.

item,count
Pencils,12
Markers,not known
,4
"Tape, blue",3

[["Pencils", 12], ["Tape, blue", 3]]

The Probing Habit

The checklist is what to look for. The probing habit is what you really want to learn.

Every time you have file code in front of you, run it on inputs that are not the happy path. The unhappy path is what catches the gallery above. It is also what catches most other bugs in this category.

A useful set to keep in mind, every time:

• The missing file (run before the file exists).
• The empty file (create the file, leave it blank, run).
• A malformed row (open the file in an editor, mangle one row, run).
• A non-ASCII value (add 庭には二羽の鶏がいる or 石室诗士施氏，嗜狮，誓食十狮 somewhere, run).
• A field with the separator inside it (try a subject called "History, ancient", run).
• The second run (run, then run again — does the file look right? did the first run survive?).

You do not need to try all of these every time. Two minutes of probing on the obvious unhappy inputs catches the majority of file-code bugs at the moment they were introduced, which is the cheapest possible time to catch them.

print(create_inventory_probe_files("inventory-probes"))

Creating probe files is the practical side of the habit. The other side is critique: when a loader appears to work, read it against the probes you would actually try. Look for the assumptions before the bug has a chance to hide behind a successful first run.

def load_inventory(path):
    items = []
    with open(path) as f:
        for line in f:
            name, count = line.strip().split(",")
            items.append([name, int(count)])
    return items

Pencils,12
Markers,5

Where You Are in the Nine-Skill Arc

Four skills are now in place:

Check Your Understanding

1. What does it mean to test the happy path and the unhappy path? Why is the unhappy path important?
2. Why is file-handling code especially prone to plausible-but-wrong AI output?
3. From the gallery in this chapter, name three things that can be wrong with a piece of file code that still runs without raising an exception.
4. What is the difference between "w" and "a" mode, and why does using "w" when you meant "a" show up as a plausible-but-wrong bug?
5. Give three concrete inputs you would try when probing a file-reading function.