Summarizing Data with pandas

You already know what summaries are. A summary turns many rows into a smaller fact: total minutes, average session length, minutes by subject, the longest sessions, recent sessions.

Pandas gives names to these operations. Start with a small DataFrame:

import pandas as pd

sessions = [
    {"subject": "math", "minutes": 45, "date": "2026-05-10"},
    {"subject": "history", "minutes": 30, "date": "2026-05-11"},
    {"subject": "math", "minutes": 60, "date": "2026-05-12"},
]

df = pd.DataFrame(sessions)

Overall summaries come from a column:

print(df["minutes"].sum())
print(df["minutes"].mean())
print(df["minutes"].max())

The next exercise sequence practices the routine behind these lines: select the minutes column, summarize it, and decide what to return for an empty DataFrame.

Summaries by Subject

The main new operation is groupby:

totals = df.groupby("subject")["minutes"].sum()
print(totals)

Read this from left to right:

1. Start with the DataFrame.
2. Group rows by the subject column.
3. In each group, look at the minutes column.
4. Sum the minutes.

The result is a Series whose labels are subjects and whose values are totals.

averages = df.groupby("subject")["minutes"].mean()
counts = df.groupby("subject")["minutes"].count()

This is much shorter than the nested-loop version from Summarizing Your Data, but it is the same idea: collect rows with the same subject and summarize each collection.

The same grouped-summary routine works for averages as well as totals.

sessions = [
    {"subject": "math", "minutes": 45, "date": "2026-05-10"},
    {"subject": "history", "minutes": 30, "date": "2026-05-11"},
    {"subject": "math", "minutes": 60, "date": "2026-05-12"},
]

{"history": 30.0, "math": 52.5}

Split, Apply, Combine

You will often see groupby explained as split, apply, combine.

For:

df.groupby("subject")["minutes"].sum()

pandas:

1. splits the rows into groups with the same subject,
2. applies the sum operation to the minutes in each group,
3. and combines the results into one summary object.

This mental model helps when the expression gets longer. If the output is strange, ask which step is wrong. Did the split use the right column? Did the apply step summarize the right values? Did the combined result need to be converted before printing?

You can also ask pandas for more than one summary:

summary = df.groupby("subject")["minutes"].agg(["count", "sum", "mean"])
print(summary)

This produces a small table with one row per subject and columns for count, sum, and mean. The project checkpoint keeps the required helpers simpler, but this is the same family of operation.

Worked Summary: Rebuilding an Earlier Question

In Summarizing Your Data, a typical question was: “How many minutes have I studied per subject, from most to least?” By the end of Structuring Your Data, the study log answered that question by looping over Session objects.

With loops, the core looked like this:

totals = {}
for session in sessions:
    subject = session.subject
    minutes = session.minutes
    totals[subject] = totals.get(subject, 0) + minutes

For this sample:

sessions = [
    Session("math", 45, "2026-05-10"),
    Session("history", 30, "2026-05-11"),
    Session("math", 60, "2026-05-12"),
    Session("programming", 25, "2026-05-12"),
]

the loop result is:

{"math": 105, "history": 30, "programming": 25}

The pandas version starts by converting those sessions to records:

records = []
for session in sessions:
    records.append(session.to_dict())

df = pd.DataFrame(records)

Then split by subject and sum the minutes:

totals = df.groupby("subject")["minutes"].sum()
print(totals)

One possible printed output is:

subject
history         30
math           105
programming     25
Name: minutes, dtype: int64

This is not a dictionary yet. It is a pandas Series with subject labels. If the dashboard wants the subjects from most minutes to least minutes, sort the values:

totals = totals.sort_values(ascending=False)
print(totals)

Output:

subject
math           105
history         30
programming     25
Name: minutes, dtype: int64

Finally, if the rest of your code wants a regular dictionary:

totals_dict = totals.to_dict()
print(totals_dict)

Output:

{'math': 105, 'history': 30, 'programming': 25}

That is the full path: records to DataFrame, DataFrame to grouped Series, grouped Series to sorted Series, sorted Series to plain dictionary. Writing it in pieces makes each shape visible.

Sorting Rows Versus Sorting Summaries

Sorting rows answers a different question from sorting a summary.

This asks “Which individual sessions were longest?”

longest_rows = df.sort_values("minutes", ascending=False)
print(longest_rows.head(3))

This asks “Which subjects have the most total minutes?”

subject_totals = df.groupby("subject")["minutes"].sum()
subject_totals = subject_totals.sort_values(ascending=False)
print(subject_totals)

Both use sort_values, but the object being sorted is different. In the first version, each row is still one study session. In the second version, each row of the result is one subject summary.

When choosing a pandas operation, name the question first. “Longest sessions” points to sorting the original rows. “Most studied subjects” points to grouping first, then sorting the grouped totals.

A Second CSV: Subject Information

Sometimes the information you need is split across two tables. The session table records what happened:

subject,minutes,date
math,45,2026-05-01
history,30,2026-05-02
programming,60,2026-05-02

A second table can describe the subjects themselves. Create a file named subjects.csv in the same folder as study_sessions.csv. Copy the CSV text below into that file:

subject,category,weekly_goal_minutes
math,quantitative,180
history,humanities,120
programming,technical,240
biology,science,150
writing,communication,120

Now you can read both files:

import pandas as pd

sessions = pd.read_csv("study_sessions.csv")
subjects = pd.read_csv("subjects.csv")

The column that connects the files is subject. Pandas calls this operation a merge:

joined = sessions.merge(subjects, on="subject", how="left")
print(joined.head())

The joined DataFrame still has one row per study session, but each row now also has the subject’s category and weekly goal. .head() shows the first five rows:

       subject  minutes        date       category  weekly_goal_minutes
0         math       45  2026-05-01   quantitative                  180
1      history       30  2026-05-02     humanities                  120
2  programming       60  2026-05-02      technical                  240
3         math       25  2026-05-03   quantitative                  180
4      writing       40  2026-05-04  communication                  120

The how="left" part means “keep every row from the sessions table, and add matching information from the subjects table when it exists.” For this sample data, every subject has a matching row in subjects.csv.

After the merge, you can summarize by a column that did not exist in the original session file:

category_totals = joined.groupby("category")["minutes"].sum()
print(category_totals.sort_values(ascending=False))

This is a small example of why pandas is useful in real work. One table rarely contains every useful detail. Being able to combine tables lets you ask questions like “How much time did I spend on technical subjects?” without manually copying category names into every session row.

Practice that routine before moving on: join the session table to the subject table, then summarize the joined data.

Date Summaries

Dates should be converted before date-based operations. Store the converted values in a local Series so the original table keeps its date strings:

dates = pd.to_datetime(df["date"])

Then comparisons work across the whole column:

today = pd.Timestamp("2026-05-15")
start = today - pd.Timedelta(days=6)

recent_mask = (dates >= start) & (dates <= today)
recent = df[recent_mask]
print(recent)

pd.Timestamp is pandas’ date-and-time value, and pd.Timedelta is pandas’ duration value. They play the same roles here as date and timedelta did in the standard-library date chapter, but they work naturally with pandas columns. The parentheses make each comparison happen before the two masks are combined.

df = pd.DataFrame([
    {"subject": "math", "minutes": 45, "date": "2026-05-08"},
    {"subject": "history", "minutes": 30, "date": "2026-05-09"},
    {"subject": "math", "minutes": 60, "date": "2026-05-15"},
    {"subject": "programming", "minutes": 25, "date": "2026-05-16"},
])

Common Mistake: and Instead of &

This looks reasonable, but it does not work with pandas Series:

recent_mask = dates >= start and dates <= today

Python’s and expects one truth value on the left and one truth value on the right. But dates >= start produces one Boolean value per row. Pandas does not know how to collapse that whole Series of Booleans into one yes-or-no answer.

Use & to combine masks element by element:

recent_mask = (dates >= start) & (dates <= today)

The parentheses are not decoration. They make each comparison happen before the masks are combined.

df = pd.DataFrame([
    {"subject": "math", "minutes": 45, "date": "2026-05-08"},
    {"subject": "history", "minutes": 30, "date": "2026-05-09"},
    {"subject": "math", "minutes": 60, "date": "2026-05-15"},
    {"subject": "programming", "minutes": 25, "date": "2026-05-16"},
])

Comparing the Loop Version and the pandas Version

The loop version makes every step visible. The pandas version names a common operation. Both are valuable.

Loop version:

totals = {}
for session in sessions:
    subject = session.subject
    minutes = session.minutes
    totals[subject] = totals.get(subject, 0) + minutes

This is the class-based shape from Structuring Your Data: sessions is a list of Session objects.

Pandas version, starting from a DataFrame that has already been created:

totals = df.groupby("subject")["minutes"].sum().to_dict()

The pandas version is not magic. It is a compact expression of the same summary idea. You should be able to explain the result in ordinary language before relying on it.

totals = df.groupby("subject")["minutes"].sum().to_dict()

A Debugging Pattern for Summaries

When a pandas summary looks wrong, debug it in layers:

print(df.head())
print(df["subject"])
print(df["minutes"])

Then inspect the grouped result before converting it:

totals = df.groupby("subject")["minutes"].sum()
print(totals)

Only after that, convert or sort:

totals = totals.sort_values(ascending=False)
print(totals.to_dict())

This is slower to type than one perfect chain, but it mirrors how you debugged loops earlier in the course. First inspect the input, then inspect the intermediate result, then inspect the final result.

If the grouped result is wrong, the bug is before .to_dict(). If the grouped result is right but the printed dashboard is wrong, the bug is after the summary helper. Separating those layers keeps pandas from becoming a black box.

Check Your Understanding

1. What does df.groupby("subject")["minutes"].sum() compute?
2. Why might you sort grouped totals after computing them?
3. Why does pandas use & instead of and for combining masks?
4. Why might a helper return totals.to_dict() instead of returning the pandas Series?