Browse cheat sheets · Python
Inspect a pandas DataFrame
Inspect a pandas DataFrame quickly and safely to understand its shape, columns, data types, nulls, duplicates, distributions, and suspicious values before transforming data.
On this page
- Problem
- Quick answer
- Code / commands
- 1. Check shape and column names
- 2. Inspect schema and memory usage
- 3. View the first and last rows
- 4. Sample rows randomly
- 5. Inspect data types
- 6. Count nulls by column
- 7. Check duplicate rows
- 8. Check duplicates by business key
- 9. Summarize numeric columns
- 10. Summarize all column types
- 11. Count unique values
- 12. Inspect categorical values
- 13. Inspect multiple categorical columns safely
- 14. Inspect numeric ranges
- 15. Check date ranges
- 16. Find suspicious whitespace in text columns
- 17. Find empty strings separately from nulls
- 18. Find common null-like string values
- 19. Inspect one record by business key
- 20. Create a reusable compact inspection function
- What the result means
- What to look for
- Numeric column loaded as object or string
- Date column loaded as text
- Very high null percentage
- Unexpectedly high cardinality in a category column
- Unexpectedly low cardinality
- Very wide min/max range
- Duplicate business keys
- Production notes
- Inspection operations can be expensive
- df.info(memory_usage="deep") can be slow
- pandas data types need interpretation
- Do not mutate while inspecting
- Samples can hide rare problems
- Production pipelines need assertions or quality rules
- Common mistakes
- 1. Trusting head() as representative
- 2. Calling drop_duplicates() immediately
- 3. Filling all nulls with zero
- 4. Assuming pandas inferred the right dtype
- 5. Printing every unique value
- 6. Converting bad data before inspecting it
- 7. Profiling every column on a huge DataFrame
- Next step
- Related deep dives
Problem
You loaded a CSV, Parquet file, API response, or query result into pandas and need to understand what is actually in the data before transforming it.
The goal is to answer quickly:
- How many rows and columns are there?
- What are the column names and data types?
- Which columns contain nulls?
- Are there duplicate rows?
- What values dominate categorical columns?
- Are numeric ranges plausible?
- Are date columns actually parsed as dates?
- Are there suspicious strings, mixed types, or unexpected values?
Do this before joins, aggregations, filtering, or writing data back out.
Quick answer
Start with a compact inspection pass:
import pandas as pd
df = pd.read_parquet("data.parquet")
print(df.shape)
print(df.columns.tolist())
df.info()
print(df.head())
print(df.isna().sum().sort_values(ascending=False).head(20))
print(df.duplicated().sum())
print(df.describe(include="all").T)
This gives you:
- dimensions
- column names
- inferred data types
- null counts
- duplicate count
- sample rows
- numeric and non-numeric summaries
Do not start transforming data until the schema and data quality look plausible.
Code / commands
1. Check shape and column names
print(df.shape)
print(df.columns.tolist())
Use this immediately after loading data.
Example:
(125000, 18)
means:
- 125,000 rows
- 18 columns
If the shape is unexpectedly small or large, verify the source before doing anything else.
2. Inspect schema and memory usage
df.info()
For deeper memory visibility:
df.info(memory_usage="deep")
Use this to inspect:
- column names
- non-null counts
- dtypes
- approximate memory usage
memory_usage="deep" is more accurate for object/string-heavy DataFrames, but it can take longer on large data.
3. View the first and last rows
print(df.head())
print(df.tail())
Choose a specific number when useful:
print(df.head(10))
Use this to verify:
- delimiter/parse behavior
- column alignment
- obvious nulls
- unexpected headers
- date formatting
- suspicious values
Do not rely on head() alone for data-quality conclusions.
4. Sample rows randomly
print(df.sample(10, random_state=42))
A random sample is often more representative than the first rows.
Use a fixed random_state when you want reproducible inspection.
5. Inspect data types
print(df.dtypes)
Filter by type:
numeric_cols = df.select_dtypes(include="number").columns
text_cols = df.select_dtypes(include=["object", "string"]).columns
date_cols = df.select_dtypes(include=["datetime", "datetimetz"]).columns
Use this to detect problems such as:
- numeric columns loaded as strings
- dates loaded as strings
- identifiers accidentally inferred as numbers
6. Count nulls by column
null_counts = df.isna().sum().sort_values(ascending=False)
print(null_counts)
Add percentage:
null_summary = (
pd.DataFrame(
{
"null_count": df.isna().sum(),
"null_pct": df.isna().mean().mul(100),
}
)
.sort_values("null_pct", ascending=False)
)
print(null_summary)
Use percentages because raw null counts are hard to interpret without row counts.
7. Check duplicate rows
duplicate_count = df.duplicated().sum()
print(duplicate_count)
Inspect duplicates:
duplicates = df[df.duplicated(keep=False)]
print(duplicates.head(20))
Do not automatically drop duplicates until you know whether duplicates are invalid.
Repeated rows may be legitimate events.
8. Check duplicates by business key
Whole-row duplicate checks can miss the duplicates that actually matter.
Example:
key_cols = ["customer_id", "order_id"]
duplicate_keys = df[
df.duplicated(subset=key_cols, keep=False)
].sort_values(key_cols)
print(duplicate_keys.head(50))
Use the columns that should uniquely identify a business entity or event.
9. Summarize numeric columns
print(df.describe().T)
Useful fields include:
- count
- mean
- standard deviation
- minimum
- quartiles
- maximum
This is a fast way to spot:
- impossible negative values
- extreme maxima
- suspicious zeros
- unexpected scale
10. Summarize all column types
summary = df.describe(include="all").T
print(summary)
This can show:
- unique value counts
- most frequent values
- numeric distribution
- date ranges where supported
On very wide or high-cardinality data, prefer targeted summaries instead of blindly profiling everything.
11. Count unique values
unique_counts = df.nunique(dropna=False).sort_values()
print(unique_counts)
Useful for detecting:
- constant columns
- near-unique IDs
- unexpectedly high-cardinality categories
- booleans encoded as strings or integers
Example:
constant_cols = df.columns[df.nunique(dropna=False) <= 1]
print(constant_cols.tolist())
12. Inspect categorical values
print(df["status"].value_counts(dropna=False))
Show percentages:
print(
df["status"]
.value_counts(dropna=False, normalize=True)
.mul(100)
.round(2)
)
Use this to detect:
- spelling variations
- unexpected categories
- null-like strings such as
"NULL"or"N/A" - dominant values
13. Inspect multiple categorical columns safely
for col in ["status", "country", "channel"]:
print(f"\n--- {col} ---")
print(df[col].value_counts(dropna=False).head(20))
Avoid printing every unique value for high-cardinality columns.
Limit output.
14. Inspect numeric ranges
numeric_summary = df.select_dtypes(include="number").agg(
["min", "max", "mean", "median"]
).T
print(numeric_summary)
Use this to validate domain expectations.
Example questions:
- Can quantity be negative?
- Should percentage exceed 100?
- Can age be 250?
- Should revenue ever be zero?
15. Check date ranges
If a date column is already parsed:
print(df["event_time"].min())
print(df["event_time"].max())
If it is still text:
parsed = pd.to_datetime(df["event_time"], errors="coerce")
print(parsed.min())
print(parsed.max())
print(parsed.isna().sum())
errors="coerce" is useful during inspection because invalid values become NaT.
Do not overwrite the original column until you understand what failed to parse.
16. Find suspicious whitespace in text columns
col = "customer_name"
leading_or_trailing_space = df[
df[col].notna()
& (df[col] != df[col].str.strip())
]
print(leading_or_trailing_space.head(20))
Whitespace problems can break:
- joins
- deduplication
- grouping
- equality comparisons
17. Find empty strings separately from nulls
empty_string_counts = (
df.select_dtypes(include=["object", "string"])
.apply(lambda s: s.eq("").sum())
.sort_values(ascending=False)
)
print(empty_string_counts)
An empty string is not the same as NaN or None.
Treat them according to the source system’s semantics.
18. Find common null-like string values
null_like_values = {"", "null", "none", "n/a", "na", "unknown"}
for col in df.select_dtypes(include=["object", "string"]).columns:
s = df[col].astype("string").str.strip().str.lower()
count = s.isin(null_like_values).sum()
if count:
print(col, count)
Use this as a diagnostic only.
Do not globally convert these strings to null without confirming their business meaning.
19. Inspect one record by business key
row = df.loc[df["customer_id"].eq(12345)]
print(row)
For multiple conditions:
row = df.loc[
df["customer_id"].eq(12345)
& df["order_id"].eq("A1007")
]
print(row)
This is often the fastest way to validate a known problematic record.
20. Create a reusable compact inspection function
def inspect_dataframe(df: pd.DataFrame) -> None:
print("Shape:", df.shape)
print("\nDtypes:")
print(df.dtypes)
print("\nNulls:")
print(df.isna().sum().sort_values(ascending=False).head(20))
print("\nUnique values:")
print(df.nunique(dropna=False).sort_values().head(20))
print("\nDuplicate rows:", df.duplicated().sum())
print("\nSample:")
print(df.sample(min(5, len(df)), random_state=42))
Use this for quick local analysis.
For production pipelines, prefer structured validation and logging rather than print-based inspection.
What the result means
| Signal | Why it matters |
|---|---|
df.shape |
Confirms whether ingestion volume is roughly what you expected. |
df.dtypes |
Shows how pandas interpreted each column. Wrong inference can break calculations and joins. |
| non-null counts | Reveal partially populated columns. |
| null percentage | Helps distinguish occasional missing values from systematically missing fields. |
nunique() |
Helps identify IDs, constants, categories, and suspicious cardinality. |
| duplicates | Signal repeated records, but only after you define what “duplicate” means. |
| min/max | Quickly expose impossible or suspicious numeric/date values. |
value_counts() |
Shows category distribution and unexpected values. |
| memory usage | Helps predict whether later operations such as joins or copies may become expensive. |
What to look for
Numeric column loaded as object or string
Possible causes:
- currency symbols
- commas
- mixed values
- invalid tokens
- source-format inconsistencies
Do not immediately call astype(float).
Inspect the bad values first.
Date column loaded as text
Check:
- multiple date formats
- timezone markers
- invalid values
- day/month ambiguity
Parse explicitly.
Very high null percentage
Ask whether the column is:
- optional
- source-specific
- recently introduced
- broken upstream
Do not automatically fill nulls with zero or empty string.
Unexpectedly high cardinality in a category column
Possible causes:
- whitespace
- capitalization differences
- IDs stored in the wrong field
- dirty source data
Unexpectedly low cardinality
A supposedly unique identifier with only a few values is a major signal.
Verify the source and transformation logic.
Very wide min/max range
Investigate:
- unit inconsistencies
- sign errors
- currency conversion
- malformed parsing
- outliers
Duplicate business keys
This may indicate:
- source duplication
- one-to-many relationships
- reprocessed events
- incomplete key definition
Do not drop until the business grain is clear.
Production notes
Inspection operations can be expensive
Some operations scan every row:
isna().sum()nunique()duplicated()describe()value_counts()
On very large DataFrames, avoid running every profiling operation blindly.
Prefer targeted columns or sampled exploration first.
df.info(memory_usage="deep") can be slow
Deep memory inspection examines Python object/string memory more accurately.
Use it when memory diagnosis matters, not automatically in every pipeline.
pandas data types need interpretation
Examples:
objectmay contain strings, mixed Python objects, or inconsistent values- nullable pandas dtypes such as
Int64,boolean, andstringbehave differently from NumPy primitives - timezone-aware datetimes differ from naive datetimes
Do not infer business semantics from dtype alone.
Do not mutate while inspecting
Prefer non-destructive checks.
Bad pattern:
df["amount"] = pd.to_numeric(df["amount"], errors="coerce")
during initial inspection.
Safer:
parsed_amount = pd.to_numeric(df["amount"], errors="coerce")
Then compare failures before replacing the original.
Samples can hide rare problems
head() and sample() are useful but incomplete.
Use full-column checks for:
- nulls
- duplicates
- key integrity
- range validation
when accuracy matters.
Production pipelines need assertions or quality rules
Manual inspection is for exploration and troubleshooting.
Production pipelines should eventually encode expectations such as:
assert df["order_id"].notna().all()
assert not df["order_id"].duplicated().any()
For larger systems, use a proper validation framework or pipeline-level quality checks.
Common mistakes
1. Trusting head() as representative
The first rows may all come from the same partition, date, customer, or sort order.
Use random samples and aggregate checks.
2. Calling drop_duplicates() immediately
Duplicates may be legitimate.
First define the business key and event grain.
3. Filling all nulls with zero
Zero and missing are not the same thing.
This can silently corrupt meaning.
4. Assuming pandas inferred the right dtype
Always inspect dtypes after ingestion.
CSV data in particular is vulnerable to inference problems.
5. Printing every unique value
This can flood logs or notebooks on high-cardinality columns.
Use:
value_counts().head()
or nunique() first.
6. Converting bad data before inspecting it
Using errors="coerce" directly on the source column can hide the original invalid values.
Compare first, then clean deliberately.
7. Profiling every column on a huge DataFrame
Some inspection operations are full scans.
Start with:
- shape
- dtypes
- targeted nulls
- sample
- known business-critical columns
Then expand only where needed.
Was this helpful?
What could be better?
Thanks — that helps me improve Javi Builds.
Stay in the loop
Get the Javi Builds updates you actually care about.
Check your inbox
I sent you a confirmation link. Your subscription will be active after you confirm it.