Handling missing values

Finding NaN, deciding what to do about it, and what the choice costs.

What you'll be able to do

Locate missing values
Choose between dropping and filling
Understand how NaN affects calculations

Finding them

df.isnull().sum()                    # count per column
df.isnull().sum().sum()              # total
df[df['revenue'].isnull()]           # rows where revenue is missing
df.isnull().mean().round(3)          # proportion missing per column

That last one is the useful one: 2% missing and 60% missing are entirely different problems.

How NaN behaves

Operation	Result with NaN
`df['a'].sum()`	Skips NaN
`df['a'].mean()`	Skips NaN — divides by non-null count
`df['a'] + df['b']`	NaN if either is NaN
`df['a'] == NaN`	Always False — use `.isnull()`
`df['a'].count()`	Non-null count only

The mean is the one that misleads. A column of 100 values with 40 missing averages over 60, which may or may not be what you intended.

Dropping

df.dropna()                          # any row with any NaN — usually too aggressive
df.dropna(subset=['revenue'])        # only where revenue is missing
df.dropna(axis=1, thresh=len(df)*0.5) # drop columns over half empty

Check what you lose

df.dropna() on a wide table can silently remove most of your rows, because it only takes one missing value anywhere in a row. Compare len(df) before and after, every time.

Filling

df['revenue'] = df['revenue'].fillna(0)
df['revenue'] = df['revenue'].fillna(df['revenue'].median())
df['region']  = df['region'].fillna('Unknown')
df['price']   = df['price'].ffill()      # carry the last value forward

Each choice is an assumption. Filling revenue with 0 says the sale was zero; filling with the median says it was typical. Those produce different answers, so pick deliberately and write down which you chose.

Forward fill suits time series with genuine gaps, and is wrong almost everywhere else.

Key points

mean() divides by the non-null count, not the row count
df.dropna() removes a row for a single missing value anywhere in it
Filling with 0 versus the median are different claims about the data

Check yourself