Microsoft Fabric

Microsoft Fabric Notebook Tutorial: From Your First Notebook to Production

A practical Microsoft Fabric notebook tutorial covering Lakehouse attachment, PySpark and SQL, Delta tables, parameters, pipelines, Git, monitoring, performance, and production readiness.

By JaviPublished 18 min readIntermediate

On this page

A Microsoft Fabric notebook can start as a few exploratory cells and quickly become part of a production data platform.

That transition is where many teams struggle.

Creating a notebook is easy. Building one that is parameterized, observable, repeatable, source-controlled, and safe to run from a pipeline requires more deliberate engineering.

This tutorial walks through that full path.

The goal is not only to show how to execute PySpark in Fabric. It is to show how a notebook fits into a real data-engineering workflow:

  1. Workspace
  2. Lakehouse
  3. Notebook
  4. Transform
  5. Delta tables
  6. Pipeline
  7. Monitoring
  8. Git / deployment
Where a notebook fits in a data-engineering workflow

By the end, you should understand how to move from an interactive notebook to something you can operate as part of a production data platform.

What a Microsoft Fabric notebook is

A Fabric notebook is an interactive development surface for Apache Spark workloads inside Microsoft Fabric.

You can use it to:

  • explore data;
  • read and write Lakehouse data;
  • transform large datasets with PySpark;
  • execute Spark SQL;
  • work with Delta tables;
  • parameterize processing;
  • run notebooks from Data Factory pipelines;
  • integrate notebook source with Git;
  • operationalize recurring data transformations.

Fabric notebooks are especially useful when the transformation needs Spark, distributed processing, or a code-first workflow.

A notebook is not automatically the best tool for every task. Simple copy operations may belong in a pipeline activity. T-SQL transformations may fit better in a Warehouse. Small orchestration steps may not need Spark at all.

The notebook is one tool in the platform.

Prerequisites

For the exercises in this tutorial, you need:

  • access to a Microsoft Fabric tenant;
  • a workspace assigned to Fabric capacity or an eligible trial;
  • permission to create items in that workspace;
  • a Lakehouse;
  • a notebook.

A practical development workspace might look like:

Workspace: data-platform-dev
Lakehouse: lh_sales
Notebook: nb_transform_sales

For production environments, use separate workspaces for development, test, and production rather than treating one workspace as every environment.

Step 1: Create your first Microsoft Fabric notebook

Inside a Fabric workspace, select New item, search for Notebook, select it, and give the notebook a meaningful name such as demo_javibuilds.

Microsoft Fabric New item panel showing Notebook as a selectable item

Create a Notebook from the Fabric workspace’s New item experience.

Step 2: Understand the notebook workspace

A new Fabric notebook starts with a code cell. The cell-language menu supports PySpark (Python), Spark (Scala), Spark SQL, SparkR (R), and HTML. This tutorial focuses on PySpark for DataFrame transformations and Python logic, and Spark SQL when SQL is the clearest expression of a transformation.

Blank Microsoft Fabric notebook showing the cell-language menu with PySpark, Spark SQL, Scala, R, and HTML options

A Fabric notebook can run PySpark, Spark SQL, Scala, R, and HTML cells.

Step 3: Create or attach a Lakehouse

The Lakehouse stores the Delta tables used in this tutorial. In Data items, select Add data items and attach an existing Lakehouse or create one named demo. Once attached, Explorer exposes Tables and Files, which notebook code can read and write directly.

Microsoft Fabric notebook showing the New lakehouse dialog with the demo Lakehouse name

Create or attach a Lakehouse from the notebook’s Data items pane.

Step 4: How the notebook, Lakehouse, and SQL analytics endpoint fit together

Before writing more code, it helps to see how the items you just created relate to each other.

A Fabric workspace is a container for Fabric items. In this tutorial it holds the Lakehouse and the notebook you created, plus one item that Fabric created for you:

Workspace
  ├─ Lakehouse (demo)
  │   ├─ Tables
  │   ├─ Files
  │   └─ SQL analytics endpoint
  └─ Notebook (demo_javibuilds)
      └─ reads/writes Delta tables in the Lakehouse
  • The Lakehouse is a Fabric item that stores data in OneLake, as tables and as files.
  • When you create a Lakehouse, Fabric automatically creates a SQL analytics endpoint for it. The endpoint is not a second copy of the data: it exposes the Lakehouse’s Delta tables to SQL and reads the same tables in OneLake.
  • The notebook is a separate item in the same workspace. It does not store the data itself; once the Lakehouse is attached, notebook code reads and writes the Lakehouse’s Delta tables and files.
  • The SQL analytics endpoint lets SQL users and tools query those same Lakehouse tables with SQL, without opening a notebook. Step 11 shows it in action.

Microsoft Fabric workspace showing a Lakehouse, its SQL analytics endpoint, and a Notebook in the same workspace

A Fabric workspace can contain the Lakehouse, its SQL analytics endpoint, and the Notebook that processes its data.

Step 5: Tables and Files inside a Fabric Lakehouse

The Lakehouse Explorer shows two areas: Tables and Files. Both live in OneLake, but they serve different purposes.

Tables

  • managed Delta tables that you can query;
  • visible to Spark by name, for example spark.table("demo_customers");
  • queryable through the SQL analytics endpoint;
  • appropriate for structured, curated data such as demo_customers and demo_sales.

Files

  • general-purpose file storage in OneLake;
  • useful for CSV, JSON, Parquet, and other formats;
  • often used for raw landing or staging data;
  • a file in Files does not automatically become a managed Lakehouse table: you load it into a table, for example with saveAsTable().
Tables Files
Holds Managed Delta tables Files in any format (CSV, JSON, Parquet, …)
Read from Spark spark.table("demo_customers") spark.read.csv("Files/raw/customers.csv")
SQL analytics endpoint Queryable as tables Not exposed as tables
Typical use Structured, curated data Raw landing and staging data

The screenshot below shows the Lakehouse after the next steps: demo_customers and demo_sales under Tables, and a raw folder under Files.

Microsoft Fabric Lakehouse showing demo_customers and demo_sales under Tables and a raw folder under Files

A Fabric Lakehouse separates managed tables from general file storage while keeping both in OneLake.

Step 6: Create a Delta table with PySpark

Create a small Spark DataFrame and save it as a managed Delta table in the attached Lakehouse. (How Delta tables differ from SQL Server tables covers what a Delta table is under the hood.)

PySpark
from pyspark.sql import Row

data = [
    Row(customer_id=1, customer_name="Alice", country="USA"),
    Row(customer_id=2, customer_name="Bob", country="Canada"),
    Row(customer_id=3, customer_name="Carlos", country="Spain"),
]

df = spark.createDataFrame(data)

df.write \
    .format("delta") \
    .mode("overwrite") \
    .saveAsTable("demo_customers")
PySpark14 linesRead-only example cell. Nothing on this page runs code.

Verify the table:

spark.table("demo_customers").show()

createDataFrame() creates the Spark DataFrame, format("delta") selects Delta format, and saveAsTable("demo_customers") registers the table under Tables in the attached Lakehouse.

Microsoft Fabric notebook running PySpark to create the demo_customers Delta table, with the table visible in the Lakehouse Explorer

PySpark creates demo_customers as a Delta table in the attached Lakehouse.

Step 7: Create and query the table with Spark SQL

You can create the same Delta table with Spark SQL. This is Spark SQL running on the notebook Spark session—not the Fabric Warehouse SQL endpoint or SQL Server T-SQL. Step 11 shows the Lakehouse’s own SQL analytics endpoint.

%%sql

CREATE OR REPLACE TABLE demo_customers
USING DELTA
AS
SELECT 1 AS customer_id, 'Alice' AS customer_name, 'USA' AS country
UNION ALL
SELECT 2, 'Bob', 'Canada'
UNION ALL
SELECT 3, 'Carlos', 'Spain';
Spark SQL10 linesRead-only example cell. Nothing on this page runs code.

Then query it:

%%sql

SELECT *
FROM demo_customers;
Spark SQL4 linesRead-only example cell. Nothing on this page runs code.

Microsoft Fabric notebook using Spark SQL to create and query the demo_customers Delta table and display the results

Spark SQL can create and query the same Delta table without leaving the notebook.

PySpark vs. Spark SQL: Both execute on Spark and can work with the same Delta tables. Use PySpark when transformations are easier to express with DataFrames and Python logic; use Spark SQL when SQL is clearest.

Step 8: Create sample sales data

Create a second Delta table so the tutorial has realistic data to transform:

%%sql

CREATE OR REPLACE TABLE demo_sales
USING DELTA
AS
SELECT 1001 AS order_id, 1 AS customer_id, 125.50 AS amount, 'USA' AS country
UNION ALL
SELECT 1002, 2, 89.99, 'Canada'
UNION ALL
SELECT 1003, 1, 210.00, 'USA';
Spark SQL10 linesRead-only example cell. Nothing on this page runs code.

Microsoft Fabric Spark SQL cell creating the demo_sales Delta table in the attached Lakehouse

A second Delta table gives us data we can transform with PySpark.

Step 9: Transform Lakehouse data with PySpark

Read demo_sales, add a derived business classification, and inspect the result:

PySpark
from pyspark.sql import functions as F

sales_df = spark.table("demo_sales")

enriched_df = (
    sales_df
    .withColumn(
        "order_size",
        F.when(F.col("amount") >= 150, "Large")
         .when(F.col("amount") >= 100, "Medium")
         .otherwise("Small")
    )
)

display(enriched_df)
PySpark15 linesRead-only example cell. Nothing on this page runs code.

The code reads a Delta table from the attached Lakehouse, adds order_size, and displays the transformed DataFrame interactively.

Microsoft Fabric notebook using PySpark to read demo_sales, add an order_size classification, and display the transformed rows

Read a Lakehouse Delta table with PySpark, add a derived column, and inspect the result interactively.

The attached Lakehouse stays visible in the notebook’s Explorer while you work, so you can see the Delta tables the code reads, such as demo_sales, next to the transformed rows:

Microsoft Fabric Notebook with the demo Lakehouse attached, a PySpark transformation over demo_sales, and displayed results

With the Lakehouse attached, PySpark can read Delta tables directly and display transformed results in the same notebook.

Aggregate the result by country:

PySpark
country_summary_df = (
    enriched_df
    .groupBy("country")
    .agg(
        F.count("*").alias("order_count"),
        F.round(F.sum("amount"), 2).alias("total_sales"),
        F.round(F.avg("amount"), 2).alias("avg_order_amount")
    )
    .orderBy(F.desc("total_sales"))
)

display(country_summary_df)
PySpark12 linesRead-only example cell. Nothing on this page runs code.

Persist the reusable summary in the Lakehouse:

PySpark
(
    country_summary_df.write
    .format("delta")
    .mode("overwrite")
    .saveAsTable("demo_sales_by_country")
)
PySpark6 linesRead-only example cell. Nothing on this page runs code.

overwrite is convenient for a tutorial, but production workloads may need append, partition replacement, MERGE, incremental processing, or idempotent updates. Always ask what happens when the notebook runs twice.

Step 10: Land raw files in Files/raw

Real pipelines rarely start from hand-typed rows. Source systems usually deliver files, and the Lakehouse Files area is a natural place to land them before they become Delta tables.

One possible organization:

Files/raw/
  customers.csv
  products.csv
  purchase_orders.csv
  sales_order_lines.csv
  sales_orders.csv

This is an example layout, not a Fabric requirement. Choose folders that match your sources and retention needs, for example by source system, by load date, or by tenant.

Microsoft Fabric Lakehouse Files area showing a raw folder with CSV source files

The Files area is a natural place to land raw source files before transforming them into Delta tables.

Read one of the files with PySpark:

PySpark
raw_customers_df = (
    spark.read
    .option("header", True)
    .option("inferSchema", True)
    .csv("Files/raw/customers.csv")
)

display(raw_customers_df)
PySpark8 linesRead-only example cell. Nothing on this page runs code.

inferSchema is convenient for exploring a file. For production ingestion, declare an explicit schema with .schema(...) so that a type change in the source is caught instead of silently inferred.

Then load it into a managed Delta table:

PySpark
(
    raw_customers_df.write
    .format("delta")
    .mode("overwrite")
    .saveAsTable("demo_customers_from_file")
)
PySpark6 linesRead-only example cell. Nothing on this page runs code.

After this write, demo_customers_from_file appears under Tables, and the CSV file stays in Files/raw/. The file itself did not become a table: the notebook created a new Delta table from it.

Step 11: Query Lakehouse tables through the SQL analytics endpoint

The Delta tables you created are not only for Spark. Open the Lakehouse’s SQL analytics endpoint from the workspace, or switch the Lakehouse view to SQL analytics endpoint, and query the tables with SQL:

SQL query: Step 11: Query Lakehouse tables through the SQL analytics endpointT-SQL (Fabric Warehouse)
SELECT TOP (100)
    customer_id,
    customer_name,
    country
FROM dbo.demo_customers;
T-SQL (Fabric Warehouse)5 linesRead-only query display. It is not connected to a database.

Microsoft Fabric SQL analytics endpoint querying the demo_customers Lakehouse table and showing the results

The Lakehouse SQL analytics endpoint lets SQL users query the same Lakehouse tables without opening a notebook.

This is a different experience from the %%sql cells in Step 7. Spark SQL runs through Spark inside the notebook; the SQL analytics endpoint provides a SQL query experience outside the notebook, over the same Lakehouse tables.

Spark SQL in Notebook SQL analytics endpoint
Runs through Spark SQL endpoint experience
Used inside notebooks Used from the SQL query editor and SQL tools
Good for mixed PySpark + SQL workflows Good for SQL-oriented querying
Works through Spark execution Exposes Lakehouse tables through SQL
Can write Delta tables Read-only for Lakehouse table data

Both read the same Delta tables in OneLake; nothing is copied between them. Tables the notebook creates, such as demo_sales_by_country in Step 9, become queryable from the endpoint as well, typically after a short metadata sync.

Step 12: Parameterize the notebook

Avoid hardcoded run values:

process_date = "2026-10-09"
tenant_id = 42

Fabric notebooks support a designated parameter cell.

A parameter cell can define defaults such as:

PySpark
process_date = "2026-10-09"
tenant_id = 0
full_reload = False
PySpark3 linesRead-only example cell. Nothing on this page runs code.

When a Fabric pipeline invokes the notebook, the Notebook activity can provide parameter values.

Notebook parameters support simple types such as string, int, float, and bool.

A useful shape is:

  1. 1Parameters
  2. 2Validate parameters
  3. 3Resolve configuration
  4. 4Execute transformation
  5. 5Write result
A parameterized notebook

Step 13: Run the notebook from a Fabric pipeline

Create a Fabric Data Factory pipeline and add a Notebook activity. The pipeline owns scheduling, retries and branching while the notebook owns the transformation; choosing between pipelines, notebooks, Dataflow Gen2 and Copy jobs explains that split.

The pipeline can select the notebook and pass base parameters.

  1. Pipeline
  2. Notebook activity
  3. base parameters
    nb_transform_sales
    • process_date
    • tenant_id
Running the notebook from a pipeline

The pipeline should usually own schedules, dependencies, retries, sequencing, and high-level execution state.

The notebook should usually own transformation logic, validation, data-quality checks, and writes.

Step 14: Add execution logging

A production notebook should let you answer:

Which run failed?
Which tenant?
Which process date?
When did it start?
When did it end?
How many rows were read?
How many were written?
What error occurred?

At minimum, establish a run context:

PySpark
from datetime import datetime, timezone
import uuid

run_id = str(uuid.uuid4())
started_at = datetime.now(timezone.utc)

print(f"run_id={run_id}")
print(f"tenant_id={tenant_id}")
print(f"process_date={process_date}")
print(f"started_at={started_at.isoformat()}")
PySpark10 linesRead-only example cell. Nothing on this page runs code.

For a real platform, persist run information to an execution-log table.

Useful fields often include:

run_id
parent_run_id
pipeline_name
notebook_name
tenant_id
process_date
status
started_at
ended_at
rows_read
rows_written
rows_rejected
error_code
error_message

Step 15: Fail deliberately when validation fails

A dangerous notebook detects a problem and continues:

if invalid_count > 0:
    print("Warning: invalid rows found")

If the output should not be trusted, fail explicitly:

PySpark
if invalid_count > 0:
    raise ValueError(
        f"Validation failed: {invalid_count} invalid records found."
    )
PySpark4 linesRead-only example cell. Nothing on this page runs code.

That allows the orchestration layer to detect failure.

Step 16: Think about retry safety

If a pipeline retries your notebook, what happens?

A production notebook should not duplicate data because the first attempt wrote successfully before the activity failed.

Typical patterns include:

  • stable business keys;
  • Delta MERGE;
  • partition replacement;
  • idempotent append;
  • processed-file tracking;
  • watermark/checkpoint state.

The principle is:

A pipeline retry should not corrupt the destination.

Incremental Loads in Microsoft Fabric walks through a concrete failure, a notebook that commits Silver but dies before recording its watermark, and shows why the rerun stays safe.

Step 17: Source-control the notebook

Fabric notebooks support Git integration as part of Fabric lifecycle management.

Source control gives you:

  • version history;
  • branches;
  • code review;
  • visibility into changes;
  • a path for promotion across environments.

A healthier model than editing directly in production is:

  1. Developer change
  2. Git branch
  3. Review
  4. DEV
  5. validation
    TEST / QA
  6. PROD
Promoting a notebook change

Your exact deployment mechanism can use Fabric Git integration, deployment pipelines, or CI/CD around Fabric APIs.

Step 18: Separate code from environment configuration

A notebook that contains workspace IDs, Lakehouse IDs, production endpoints, or secrets throughout its cells becomes difficult to promote.

Prefer environment-specific configuration outside the core transformation logic.

This becomes especially important when moving notebooks across:

DEV
QA
PROD

Step 19: Understand what Git does and does not deploy

Source control and deployment manage supported item metadata and code, not your business data itself.

A useful separation is:

Git
→ definitions / code / metadata

OneLake
→ actual data

Treat data migration and application deployment as related but separate concerns.

Step 20: Monitor notebook execution

A notebook is not finished when it succeeds once.

Track at least:

  • duration;
  • success/failure;
  • row counts;
  • input size;
  • output size;
  • retries;
  • rejected records;
  • repeated slowdowns.

A notebook that normally finishes in 4 minutes and suddenly takes 25 minutes may still technically succeed, but operationally it deserves investigation.

Step 21: Performance basics

Start with:

Where is the time actually going?

Possibilities include:

Spark session startup
data read
shuffle
join
serialization
write
small files
source latency
orchestration wait

Do not optimize Python syntax when the workload spends most of its time shuffling data in Spark.

Avoid unnecessary actions

Actions trigger execution:

PySpark
df.count()
display(df)
df.collect()
PySpark3 linesRead-only example cell. Nothing on this page runs code.

Use them deliberately.

Avoid large driver collections

rows = df.collect()

can overwhelm the driver for large dataframes.

Watch joins

Before joining large datasets, understand:

  • row counts;
  • key cardinality;
  • skew;
  • duplicate keys;
  • expected output cardinality.

Step 22: Keep notebook cells production-friendly

A good structure is:

1. Parameters
2. Imports
3. Configuration
4. Run context / logging
5. Read
6. Validate input
7. Transform
8. Validate output
9. Write
10. Final metrics

Production execution should not depend on manually running cells in a hidden order.

Step 23: Avoid hidden notebook state

A notebook can appear to work because a variable was created in a cell executed earlier.

Before considering it ready, test:

fresh session
→ Run all
→ success

If that fails, the notebook is not reproducible yet.

Step 24: Treat the notebook as a component

One notebook should not necessarily own ingestion, all transformations, orchestration, logging, notifications, and serving.

A cleaner boundary might be:

Pipeline
├── Copy raw data
├── Notebook: validate + standardize
├── Notebook: business transformation
├── Data-quality activity
└── Publish / notify

The boundary should reflect operational responsibilities.

Production-ready checklist

Data

  • Is the input contract explicit?
  • Are important keys validated?
  • Are duplicates handled?
  • Are invalid records accounted for?

Repeatability

  • Does it run from a fresh session?
  • Is it safe to retry?
  • Are writes idempotent?

Configuration

  • Are environment-specific values externalized?
  • Are run values parameters rather than hardcoded?

Operations

  • Does every run have an identity?
  • Are start/end/status recorded?
  • Are row counts captured?
  • Does a critical validation issue fail the activity?

Performance

  • Have large joins been checked?
  • Are unnecessary Spark actions removed?
  • Is driver collection avoided?
  • Is the write strategy appropriate?

Lifecycle

  • Is the notebook in source control?
  • Can it move through DEV/TEST/PROD?
  • Are dependent Lakehouses and environments understood?

From first notebook to production

  1. 1Notebook works interactively
  2. 2Notebook runs from a clean session
  3. 3Inputs and outputs are validated
  4. 4Parameters replace hardcoded values
  5. 5Writes become retry-safe
  6. 6Pipeline owns orchestration
  7. 7Execution is logged
  8. 8Git owns change history
  9. 9Deployment promotes across environments
  10. 10Production monitoring measures behavior
From first notebook to production

That is the difference between using a notebook and engineering a notebook workload.

Final thoughts

Microsoft Fabric makes it easy to create a notebook, connect it to a Lakehouse, and start transforming data.

But the engineering work starts after the first successful run.

A production notebook needs clear dependencies, parameters, validation, retry-safe writes, execution logging, source control, deployment discipline, and monitoring.

The goal is not to remove the flexibility that makes notebooks useful.

The goal is to keep that flexibility while adding enough structure that the notebook can be trusted in production.

Official Microsoft references

Tags

  • Microsoft Fabric
  • Fabric Notebook
  • PySpark