Interview preparation

Python Interview Questions and Answers

Practical Python questions for data engineers, from core language fundamentals to reliable data processing and production debugging.

  • 20 questions
  • Beginner → Senior
  • Production-focused

Python interviews for data roles care less about language trivia and more about whether your code survives real data: large files, missing values, failures halfway through a load, memory that keeps growing. Each answer is written to be spoken in your own words: a short answer, why it matters in production, what a strong answer usually demonstrates, the traps to avoid, and where to go deeper.

Esta guía está disponible en inglés.

Python interview preparation workspace with pandas, generators, file processing, testing, logging and memory profiling.

At a glance

How the levels work

Q01BeginnerData Structures

What is the difference between a list, tuple, set and dictionary in Python?

Short answer

A list is an ordered, mutable sequence, my default for collecting records in order. A tuple is ordered but immutable, so it suits fixed records like a composite key. Because it can’t change, it can also be used as a dictionary key. A set holds unique, unordered items with fast membership checks, which is great for deduplicating IDs. A dictionary maps unique keys to values with fast lookup by key, like a lookup table from customer ID to name, and it keeps insertion order.

Why it matters in production

Choosing the right structure changes both correctness and speed. Checking if id in seen against a list of a million IDs is a linear scan each time; against a set it is a hash lookup. Using a list where duplicates must not exist lets duplicates slip into a load.

What the interviewer is looking for

A strong answer usually demonstrates the properties that drive the choice (order, mutability, uniqueness, lookup by key), with realistic data-engineering uses:

  • list: rows read from a file, a batch of records to insert;
  • tuple: (source_system, business_key) as a composite key, rows from a database cursor;
  • set: IDs already processed, allowed status values, comparing two key sets with - or &;
  • dict: mappings and lookups, configuration, counting with collections.Counter, grouping with defaultdict(list).

Watch out

Reducing the answer to syntax ([], (), {}). Also: an empty {} is a dictionary, not a set (use set()); set iteration order is not something to rely on; and dictionary insertion order has been guaranteed by the language only since Python 3.7.

Back to all questions ↑

Q02BeginnerData Structures

What does mutable vs immutable mean in Python?

Short answer

A mutable object can change its contents after it’s created; an immutable one can’t. Lists, dictionaries and sets are mutable. Integers, floats, strings, tuples and frozensets are immutable: an operation that looks like a change actually creates a new object. This matters because variables hold references. If two names point to the same list, a change through one is visible through the other, including inside functions. Dictionary keys and set members must be hashable, which in practice means immutable.

Why it matters in production

Shared mutable state causes quiet bugs: a function that appends to a list it received changes the caller’s data; a configuration dictionary modified in one place affects every job using it; a list stored in a cache is later mutated by accident.

What the interviewer is looking for

A strong answer usually demonstrates the reference model (names point to objects; assignment does not copy) and the practical consequences: functions should not mutate their inputs unless that is their documented purpose, and you copy when you need an independent value.

Watch out

  • The mutable default argument trap: def add(row, batch=[]) reuses the same list across calls. Use batch=None and create the list inside.
  • Thinking a tuple is “deeply” immutable: a tuple containing a list cannot be reassigned, but the list inside it can still change, and such a tuple is not hashable.
  • Thinking x += 1 modifies an integer in place: it rebinds x to a new object.

Example

def add_row(row, batch=None):
    batch = [] if batch is None else batch   # a fresh list per call
    batch.append(row)
    return batch

Back to all questions ↑

Q03BeginnerPython Fundamentals

What is the difference between == and is?

Short answer

== compares values: are these two objects equal? It calls the type’s equality method. is compares identity: are these the same object in memory? Two separate lists with the same contents are equal but not identical. In practice I use is almost only for singletons, especially x is None, and == for comparing data. Using is to compare numbers or strings is a bug, even if it sometimes seems to work.

Why it matters in production

Code like if status is "done" can pass in testing and fail in production, because whether two equal strings are the same object is an interpreter implementation detail. Value comparisons must use ==.

What the interviewer is looking for

A strong answer usually demonstrates the value versus identity distinction, the None convention and why it exists (None is a singleton, and is cannot be overridden by a class’s __eq__), and awareness that apparent “identity” of small integers or short strings is a CPython optimization, not language behavior.

Watch out

  • == None instead of is None. It usually works, but objects can define __eq__ in surprising ways, and libraries such as pandas and NumPy overload == to return arrays, not booleans.
  • In pandas, neither is None nor == None finds missing values in a column; use isna().
  • Using is for numbers, strings or tuples.

Back to all questions ↑

Q04BeginnerFunctions & Expressions

What are list comprehensions, and when should you avoid them?

Short answer

A list comprehension builds a new list by transforming and optionally filtering items in one expression, like [r["id"] for r in rows if r["active"]]. It’s concise and readable for simple mappings and filters, and usually a little faster than an equivalent loop with append. I avoid it when the logic gets complex, with nested loops, several conditions or multi-step transformations, because a plain loop or a small named function reads better. I also avoid it when I’m running it for side effects rather than to build a list.

Why it matters in production

Readable code is what the next person can debug at 3 a.m. A three-level nested comprehension with conditions saves lines but costs understanding. And a comprehension over millions of records builds the whole list in memory, when a generator would stream it.

What the interviewer is looking for

A strong answer usually demonstrates judgment about readability and memory, and knowledge of the related forms: dict and set comprehensions, and generator expressions (with parentheses) that produce items lazily instead of building a list.

Watch out

  • Saying comprehensions are always faster or better. The speed difference is usually small, and clarity matters more.
  • Using a comprehension only for side effects, such as [print(x) for x in rows] or [db.insert(r) for r in rows], which builds a throwaway list. Use a loop.
  • Building a huge list just to pass it to sum(), any() or a writer: sum(r["amount"] for r in rows) streams instead.

Back to all questions ↑

Q05BeginnerFunctions & Expressions

What are *args and **kwargs?

Short answer

In a function definition, *args collects extra positional arguments into a tuple, and **kwargs collects extra keyword arguments into a dictionary. The names are just a convention; the asterisks are what matter. In a call, * and ** do the reverse: they unpack a sequence or a dictionary into arguments. They’re useful for flexible interfaces, like wrappers, decorators or functions that pass options through to a library. But for normal functions, explicit parameters are clearer.

Why it matters in production

Explicit signatures document themselves, get checked by editors and type checkers, and fail fast on typos. A function that accepts **kwargs silently ignores or forwards a misspelled option, which can turn into a setting that never took effect.

What the interviewer is looking for

A strong answer usually demonstrates both collecting (in definitions) and unpacking (in calls), a realistic use such as forwarding options to pandas.read_csv or wrapping a function with retry logic, and restraint about using them.

Watch out

  • Using **kwargs to avoid designing a proper interface.
  • Forgetting that keyword-only parameters can be declared after a bare * (def load(path, *, chunk_size=10_000)), which is often what you actually want: optional settings that must be named.
  • Mutating kwargs you received and expecting the caller not to notice when the same dictionary was unpacked from their side.

Example

def read_source(path, **read_options):
    # Forward pandas options without listing each one, but keep the important ones explicit elsewhere.
    return pd.read_csv(path, **read_options)

read_source("orders.csv", usecols=["order_id", "total"], dtype={"order_id": "string"})

Back to all questions ↑

Q06BeginnerData Structures

What is the difference between a shallow copy and a deep copy?

Short answer

A shallow copy creates a new outer container but keeps references to the same inner objects. A deep copy recursively copies the inner objects too, so the result is fully independent. With a list of dictionaries, a shallow copy gives me a new list, but changing a dictionary inside it changes the original as well. copy.copy, list.copy() or slicing make shallow copies, and copy.deepcopy makes deep ones. Deep copies cost time and memory, so I only use them when I really need independence.

Why it matters in production

Copy bugs are subtle: a “copied” configuration template gets its nested settings modified for one job, and every later job inherits the change. On the other side, deep-copying large structures on every call can quietly double memory use and slow a pipeline down.

What the interviewer is looking for

A strong answer usually demonstrates the outer versus nested distinction, how to make each copy, and the cost trade-off. Often the better design avoids mutation entirely, building new structures instead of copying and editing.

Watch out

  • Assuming new = old copies anything; it is just another name for the same object.
  • Assuming dict.copy() or list(old) copies nested objects.
  • In pandas, chained indexing and views versus copies have their own rules (and pandas is moving to copy-on-write behavior). Use .copy() explicitly when you need an independent DataFrame slice, and do not reason about it from Python’s copy module alone.

Example

import copy

template = {"name": "orders", "options": {"delimiter": ","}}
shallow = copy.copy(template)
shallow["options"]["delimiter"] = ";"      # also changes template["options"]
deep = copy.deepcopy(template)             # fully independent

Back to all questions ↑

Q07WorkingIterators & Generators

What is the difference between an iterable, iterator and generator?

Short answer

An iterable is anything you can loop over: lists, files, dictionaries. Calling iter() on it gives you an iterator. An iterator is the object that keeps the position. Each next() returns the next item until it raises StopIteration, and once it’s exhausted, it’s done. A generator is a convenient way to create an iterator, by writing a function with yield or a generator expression. So a list is iterable but not an iterator; a file object and a generator are both iterators, which is why you can only loop over them once.

Why it matters in production

The “only once” behavior causes real bugs: a generator of records consumed by a validation step is empty when the load step tries to use it, and nothing errors, it just writes zero rows.

What the interviewer is looking for

A strong answer usually demonstrates the protocol (an iterable implements __iter__; an iterator implements __iter__, returning itself, and __next__), the single-pass nature of iterators, and why generators matter: they produce items lazily, one at a time, without building the whole collection.

Watch out

  • Looping twice over a generator or file and expecting data the second time.
  • Calling len() on a generator (it has no length) or converting it to a list “just to check”, which loads everything into memory.
  • Confusing “iterable” with “sequence”: sets and dictionaries are iterable but not indexable.

Example

rows = (line.rstrip("\n") for line in open("orders.csv"))   # a generator: single pass
first = next(rows)          # header
count = sum(1 for _ in rows) # consumes the rest
list(rows)                  # [] — already exhausted

Back to all questions ↑

Q08WorkingIterators & Generators

What does yield do, and why is it useful in data processing?

Short answer

yield turns a function into a generator. When called, it returns a generator object without running the body. Each time the caller asks for the next item, the function runs until the next yield, hands back that value, and pauses with its local state intact, then resumes from there next time. In data processing, that lets me stream records one at a time, or in batches, instead of building huge lists. I can also chain small steps like read, parse, filter and batch into a memory-efficient pipeline.

Why it matters in production

A pipeline built from lists holds every intermediate result in memory at once; a pipeline built from generators holds roughly one record or batch per step. That is often the difference between a job that runs on a modest machine and one that crashes as files grow.

What the interviewer is looking for

A strong answer usually demonstrates lazy, resumable execution and a practical pattern: generator functions for reading, cleaning and batching records, with the final consumer (a writer or loader) pulling items through the chain.

Watch out

  • Expecting code in a generator to run when the function is called. Nothing runs until the first next(), including validation at the top of the function.
  • Consuming a generator twice.
  • Resources: a generator that opens a file inside a with block keeps it open until it is exhausted or closed. Make sure generators are fully consumed or explicitly closed.

Example

def read_records(path):
    with open(path, newline="") as f:
        for row in csv.DictReader(f):
            yield row

def batches(records, size=5_000):
    batch = []
    for record in records:
        batch.append(record)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch

for batch in batches(r for r in read_records("orders.csv") if r["status"] != "test"):
    load(batch)

Back to all questions ↑

Q09WorkingFiles & Memory

How would you process a very large file without loading it entirely into memory?

Short answer

I’d stream it instead of reading it all at once. For text and CSV files, I iterate over the file object line by line, or use csv.reader, and process records through generators. For tabular work in pandas, I use read_csv with chunksize and aggregate or write each chunk. For JSON, a streaming or line-delimited parser. For columnar files like Parquet, I read only the needed columns or row groups. Output goes out incrementally too, appended or written per chunk, so memory stays flat as the file grows.

Why it matters in production

f.read() or readlines() on a 20 GB file needs at least that much memory, often several times more once parsed into Python objects. Streaming keeps memory proportional to one chunk, not to the file.

What the interviewer is looking for

A strong answer usually demonstrates streaming end to end: input, transformation and output. Plus the practical details:

  • iterate the file object directly (for line in f), which reads buffered lines lazily;
  • choose a chunk size by measuring memory and throughput, not by habit;
  • with pandas chunks, keep running aggregates instead of concatenating all chunks back into one DataFrame;
  • read only the columns you need (usecols) and set dtypes up front;
  • handle the edge cases: encoding, the header, a malformed line in the middle, and a final partial batch.

Watch out

  • pd.concat(list_of_chunks) at the end, which rebuilds the whole file in memory.
  • Accumulating results in a list “to write later”.
  • Assuming a JSON array file can be streamed with json.load; it cannot. Prefer JSON Lines, or a streaming parser.

Example

totals = {}
for chunk in pd.read_csv("sales.csv", usecols=["region", "amount"], chunksize=500_000):
    for region, amount in chunk.groupby("region")["amount"].sum().items():
        totals[region] = totals.get(region, 0) + amount

Back to all questions ↑

Q10WorkingExceptions & Resource Management

How should exception handling be designed in production Python code?

Short answer

Catch specific exceptions, only where you can do something useful: retry, add context, clean up, or turn the error into a clear failure. Let everything else propagate. When I re-raise, I add context, such as which file, record or batch failed, and chain the original with raise ... from err. I retry only failures that are actually transient, like timeouts or throttling, with limits and backoff. I log once, at the level that handles the error. Cleanup goes in with blocks or finally, and errors are never swallowed silently.

Why it matters in production

A swallowed exception turns a failed load into a “successful” run with missing data. An overly broad retry hammers a source that is down, or retries a bug forever. Errors without context make every incident start with “which file was it?”

What the interviewer is looking for

A strong answer usually demonstrates a policy, not just syntax:

  • Specific exceptions (FileNotFoundError, KeyError, a library’s timeout error) rather than bare except: or except Exception in the middle of code.
  • Context and chaining: raise LoadError(f"batch {batch_id} failed") from err keeps the original traceback as the cause.
  • Retry classification: transient (timeouts, throttling, temporary network errors) versus permanent (bad data, permission denied, bugs).
  • One place to log: usually the top-level job handler records the failure with the run’s context, instead of every layer logging the same error.
  • Clear exit status: a job that fails should exit non-zero, so the orchestrator knows.

Watch out

  • except Exception: pass, or logging and continuing as if nothing happened.
  • Catching exceptions to return None, which moves the failure somewhere harder to debug.
  • except Exception is acceptable at the very top of a job, to log with context and exit with failure, not deep inside processing loops.

Back to all questions ↑

Q11WorkingExceptions & Resource Management

What is a context manager, and why is with useful?

Short answer

A context manager is an object that sets something up when a with block starts and guarantees cleanup when the block ends, whether it finishes normally or raises an exception. Files are the classic case: with open(path) as f: closes the file no matter what. The same pattern fits database connections and transactions, locks, temporary directories, and timers. I can write my own with a class that has __enter__ and __exit__, or more simply with contextlib.contextmanager and a generator.

Why it matters in production

Leaked resources are slow-burning incidents: files left open until the process runs out of handles, connections never returned to a pool, transactions left open and holding locks after an exception. with makes the cleanup path impossible to forget.

What the interviewer is looking for

A strong answer usually demonstrates deterministic cleanup on both paths (success and exception), realistic uses (a transaction that commits on success and rolls back on error), and how to write a simple custom one.

Watch out

  • Opening resources without with and relying on garbage collection to close them; the timing is not guaranteed across Python implementations.
  • Returning True from __exit__, which suppresses the exception, usually unintentionally.
  • Many related resources in one block: contextlib.ExitStack manages a variable number of them cleanly.

Example

from contextlib import contextmanager

@contextmanager
def transaction(conn):
    try:
        yield conn
        conn.commit()
    except Exception:
        conn.rollback()
        raise

with transaction(conn):
    load_batch(conn, rows)

Back to all questions ↑

Q12Workingpandas & Data Processing

When would you use pandas instead of plain Python?

Short answer

I’d use pandas when the data is tabular and the work is column-oriented: filtering, joins, group-bys, reshaping, time series, or reading and writing CSV and Parquet. Its operations run on whole columns in optimized native code, so they’re both shorter and much faster than looping over dictionaries in Python. Plain Python is better for small or irregular data, record-by-record streaming, or simple scripts where pandas adds a dependency and memory overhead. And pandas works in memory, so very large data needs chunking or a different engine.

Why it matters in production

pandas makes complex transformations short and readable, but loading a whole dataset into a DataFrame costs several times the raw file size in memory. Choosing it for a 50-line file transfer adds overhead; choosing plain Python for a multi-join transformation adds bugs.

What the interviewer is looking for

A strong answer usually demonstrates fit for the shape of the work: tabular, set-based operations favor pandas; streaming, nested or irregular records favor plain Python or generators; data larger than memory favors chunking, a database, or a distributed engine such as Spark.

Watch out

  • Saying pandas replaces Python collections. Lookups, configuration, small sets of IDs and record streams are still better as dicts, sets and generators.
  • Using pandas and then iterating rows with iterrows(), which throws away its main advantage.
  • Forgetting that pandas infers types: IDs with leading zeros, mixed columns and dates often need explicit dtype and parsing.

Back to all questions ↑

Q13Workingpandas & Data Processing

Why are vectorized pandas operations usually better than row-by-row loops?

Short answer

A vectorized operation applies to a whole column at once, like df["total"] = df["price"] * df["qty"], and the loop runs in optimized native code instead of in Python. A row-by-row loop with iterrows() or a Python function pays interpreter overhead and object creation for every row, so it’s often tens to hundreds of times slower. Vectorized code is also shorter and clearer. Not everything vectorizes cleanly, and some operations create large temporary arrays, but column operations are the default I reach for first.

Why it matters in production

A transformation that takes seconds vectorized can take hours as a loop on millions of rows, which turns into missed load windows and bigger machines bought for the wrong reason.

What the interviewer is looking for

A strong answer usually demonstrates the vectorized toolbox and an honest view of the trade-offs:

  • arithmetic and comparisons on columns; boolean masks for filtering;
  • np.where or np.select for conditional logic; Series.map with a dictionary for lookups;
  • the .str and .dt accessors for strings and dates;
  • groupby with built-in aggregations, and merge instead of looking up values row by row;
  • awareness of memory: each intermediate expression can create a full-size temporary column.

Watch out

  • Calling apply() “vectorized”. df.apply(f, axis=1) still calls a Python function for every row; it is a loop with nicer syntax.
  • Optimizing a tiny DataFrame where the loop is perfectly fast enough.
  • Treating .str methods on object-dtype columns as fully native; string operations are improved with the newer string dtypes but are still slower than numeric ones.

Back to all questions ↑

Q14Workingpandas & Data Processing

How do you handle missing values safely in pandas?

Short answer

First I measure: isna().sum() per column, and look at why values are missing. Then I decide per column, based on what missing means for the business. I might drop rows where a required key is missing, fill with a documented default where it’s safe, keep the gap and flag it, or quarantine the records. I keep missing distinct from zero or an empty string, because they mean different things. And I watch dtypes: classic NumPy integer columns become floats when they contain NaN, so I use pandas’ nullable types when that matters.

Why it matters in production

Filling missing amounts with 0 changes totals and averages; dropping rows with any null can silently remove a large share of the data; a key column that turned into floats (1001.0) breaks joins downstream.

What the interviewer is looking for

A strong answer usually demonstrates decisions driven by meaning, not habit:

  • detect with isna(), and also catch disguised missing values: empty strings, "N/A", "null", sentinel values like -1 or 1900-01-01, often handled at read time with na_values;
  • choose per column: dropna(subset=[...]) for required fields, fillna only where a default is correct, and explicit flags where analysts need to know;
  • use nullable dtypes (Int64, boolean, string) to keep integers and booleans intact alongside missing values (pd.NA);
  • validate after cleaning: counts of rows dropped or filled, recorded in logs or quality results.

Watch out

  • df.fillna(0) across the whole DataFrame.
  • dropna() with no subset.
  • Comparing with == None or == np.nan (NaN is not equal to itself); use isna().

Back to all questions ↑

Q15Seniorpandas & Data Processing

A pandas job works with 100 MB but crashes with 10 GB. What would you change?

Short answer

I’d first measure where memory goes, then shrink and stream the work rather than buying RAM. I read only the columns I need, set compact dtypes up front (categories for repeated strings, smaller numeric types), and switch from CSV to Parquet when I can. If the result doesn’t need everything at once, I process in chunks or incrementally and avoid unnecessary copies and big intermediate merges. If the work is really a large join or aggregation, I push it to a database or a distributed engine like Spark, which is built for that scale.

Why it matters in production

A DataFrame often needs several times the size of the source file in memory, and operations such as merges and string handling create temporary copies on top. A job that fits comfortably at 100 MB can need tens of gigabytes at 10 GB.

What the interviewer is looking for

A strong answer usually demonstrates evidence first (df.memory_usage(deep=True), tracemalloc or a memory profiler, peak usage per step), then the levers in order:

  • Read less: usecols, filters at the source, only the needed date range.
  • Store compactly: explicit dtypes, category for low-cardinality text, downcast numerics, nullable or Arrow-backed types where they reduce memory.
  • Better formats: Parquet reads only the requested columns and keeps types; CSV must be parsed in full.
  • Stream: chunksize with running aggregates, or incremental processing of only new data.
  • Avoid copies: drop intermediates with del, avoid chained operations that create full copies, be careful with wide merges that multiply rows.
  • Change engine: push joins and aggregations into SQL, or use Spark or another out-of-core tool when the data truly exceeds one machine.

Watch out

“Just add RAM.” It may work once, but cost grows with the data, and it does not fix a merge that multiplies rows or a job that reloads full history every run.

Back to all questions ↑

Q16WorkingTesting & Reliability

How would you structure reusable transformation code in Python?

Short answer

I separate I/O from logic. Reading and writing, to files, databases or APIs, lives at the edges. Transformations are small functions that take data in and return data out, without side effects, so each one is easy to understand, reuse and test. Inputs and outputs are explicit, with type hints and clear names, and configuration like paths, dates and thresholds is passed in rather than hard-coded. Then a thin orchestration layer composes the steps: read, transform, validate, write.

Why it matters in production

When reading, transforming and writing are tangled in one long script, every change is risky, nothing can be tested without real files and credentials, and the same logic gets copied into the next job with small differences.

What the interviewer is looking for

A strong answer usually demonstrates design for testability and reuse:

  • pure-ish functions for transformations, so the same input gives the same output;
  • explicit interfaces: parameters and return values instead of globals; type hints that document the expected shapes;
  • configuration outside the code: environment variables, config files or job parameters, validated at start-up;
  • composition: small steps assembled by a pipeline function, so each step can be reused, reordered and tested on its own;
  • packaging: shared code in a proper module or package with pinned dependencies, not copied between notebooks.

Watch out

  • One giant function that does everything.
  • Functions that read global state or environment variables deep inside the logic.
  • Over-engineering: class hierarchies and plugin frameworks for a job with three steps. Simple functions usually go a long way.

Back to all questions ↑

Q17WorkingTesting & Reliability

How do you test data-processing code?

Short answer

Mostly with fast unit tests on small, hand-built inputs. A few rows per test, each designed to exercise one rule, with the expected output written out explicitly. I deliberately include the awkward cases: nulls, duplicates, empty input, unexpected types, boundary dates, and malformed records. I check the schema, the row counts and the values, not just that the function ran. Then a smaller number of integration tests run the real reads and writes against a test environment, and data quality checks in the pipeline catch what tests can’t predict.

Why it matters in production

Data code fails on the data you did not imagine. Tests that only use clean, happy-path rows give confidence right up to the first file with a missing column or a duplicated key.

What the interviewer is looking for

A strong answer usually demonstrates a layered strategy:

  • Unit tests for transformation functions with small fixtures (a few dicts, or a tiny DataFrame), compared with pandas.testing.assert_frame_equal or plain assertions.
  • Edge cases as first-class tests: nulls, duplicates, empty input, extreme values, timezone and date boundaries, bad encodings.
  • Contract checks: output columns, dtypes, uniqueness of keys, expected row counts.
  • Determinism: fixed dates, seeds and sorted outputs, so tests do not flake.
  • Integration tests for the I/O edges, run less often.
  • Runtime data quality checks in the pipeline itself for problems tests cannot foresee.

Watch out

  • Testing only happy-path values.
  • Tests that depend on production data or the current date.
  • Asserting only “no exception” or only the row count, without checking values.

Back to all questions ↑

Q18SeniorProduction Python

How would you design logging for a production Python data pipeline?

Short answer

I use the standard logging module, configured once at the entry point, with structured output, usually JSON, so logs can be searched and aggregated. Every line carries a run or correlation ID, the job and step, a timestamp and a level. Key events log the facts I’ll need later: the source window, row counts read, written and rejected, durations, retries and errors with tracebacks. Levels are used deliberately, and secrets and personal data never get logged. For run-level facts, I also write an execution log table.

Why it matters in production

When a job fails at 2 a.m. or the numbers are wrong next week, logs are the evidence. Free-text print statements without IDs or counts cannot answer “which run, which file, how many rows, what failed”.

What the interviewer is looking for

A strong answer usually demonstrates logs designed for investigation:

  • Configuration once: handlers and format set at the entry point; libraries and modules use logging.getLogger(__name__).
  • Structured fields: run ID, correlation ID, job, step, environment, source window, counts, duration, error type.
  • Levels with meaning: INFO for milestones and counts, WARNING for recoverable anomalies (rows quarantined, retry), ERROR with traceback (logger.exception) when a step fails, DEBUG off in production by default.
  • Volume control: one summary line per batch, not one line per record.
  • Safety: no credentials, tokens or personal data, including in exception messages and logged payloads.
  • Run-level record: an execution log table for status, window and counts, which logs complement rather than replace.

Watch out

  • print() in production code.
  • Logging the same error at every layer.
  • Building messages eagerly with f-strings inside very hot loops; pass arguments (logger.debug("row %s", row_id)) so disabled levels cost little.
  • Logging whole records or connection strings.

Back to all questions ↑

Q19SeniorDebugging & Performance

A Python data job suddenly uses much more memory than before. How do you investigate it?

Short answer

Something changed, either the data, the code or the environment, so I compare with a good run instead of guessing. First the data: more rows, wider columns, a new text column, or a join key that’s now duplicated and multiplies rows. Then I measure where memory peaks, with tracemalloc, a memory profiler, or memory_usage(deep=True) on DataFrames, step by step. Typical culprits are full materialization where a generator used to stream, results accumulating in lists or caches, extra DataFrame copies, type changes, and library upgrades.

Why it matters in production

Memory growth usually ends in an out-of-memory kill with no Python traceback, often only in production, where the data is largest. Adding memory without finding the cause just moves the failure date.

What the interviewer is looking for

A strong answer usually demonstrates an evidence-driven approach:

  1. Compare runs: input sizes and row counts from the execution log, peak memory per run, the first run where it grew.
  2. Check the data: volume growth, new or wider columns, high-cardinality text, a join that now produces more rows than its inputs.
  3. Profile: tracemalloc snapshots compared between steps, or a line-level profiler; df.memory_usage(deep=True) for DataFrames; peak resident memory from the platform.
  4. Look for accumulation: lists or dicts that grow for the whole run, caches without limits, references kept to old DataFrames.
  5. Look for materialization: list(generator), pd.concat of all chunks, read() of whole files, a toPandas() on a larger Spark result.
  6. Check types: numeric columns parsed as object because of a bad value, categories lost, strings duplicated.
  7. Check changes: recent code changes, library upgrades (pandas, NumPy, pyarrow), Python version, configuration.

Watch out

Assuming freed memory always returns to the operating system. In CPython, memory released by objects is often kept by the allocator for reuse, so a process can stay large after a peak; measure the peak and the allocation pattern, not only the final size.

Back to all questions ↑

Q20SeniorProduction Python

How would you design a reliable Python ingestion job for millions of records?

Short answer

I’d design it to survive failure and reruns, not just the happy path. It reads a bounded window or a known set of files, streamed in chunks so memory stays flat. Each batch is validated, and bad records go to a quarantine path with the reason instead of failing everything. Writes are idempotent, using upserts on a business key or replacing a partition, and committed per batch. Progress is checkpointed, and the watermark moves only after a successful write, so a restart resumes safely. Transient errors get bounded retries, and structured logs and metrics record every run.

Why it matters in production

At millions of records, something will fail mid-run: a network timeout, a malformed row, a full disk. The question is whether the job resumes cleanly or leaves duplicates, gaps and a manual cleanup.

What the interviewer is looking for

A strong answer usually demonstrates failure recovery and trade-offs, independent of any specific technology:

  • Bounded input: an explicit window (watermark range, business date, file list) recorded at start, so reruns cover the same data.
  • Streaming and batching: chunked reads and batched writes; batch size balances throughput, memory, transaction size and how much is redone after a failure.
  • Validation: schema and key checks per batch; quarantine (a dead-letter table or folder with the error reason) for bad records, with a threshold that fails the run if too many are rejected.
  • Idempotent writes: upsert by key, or write to staging and swap or replace partitions, so reprocessing a batch cannot duplicate data.
  • Checkpoints and watermarks: progress stored after each committed batch; the watermark advances only after the final successful write.
  • Retries: only for transient errors, with exponential backoff and a limit; permanent errors fail fast with context.
  • Observability: structured logs with run ID, counts read, written and rejected, durations, plus an execution log row per run and alerts on failure or anomalies.
  • Testing: unit tests for parsing and validation with ugly records, an end-to-end test on a small fixture, and a rerun test that proves no duplicates.
  • Resource control: bounded memory, connection pooling, and rate limits that respect the source.

Watch out

Loading everything into memory and inserting in one transaction, retrying every exception, advancing the watermark before the write commits, and treating a run that quarantined half the records as a success.

Back to all questions ↑

Keep preparing

New questions are added as they are written. Interview questions for other topics live on the interview questions hub. The best preparation is still hands-on: start from the Python hub, practise with the pandas DataFrame lessons, and keep the Python cheat sheets close for quick reference.