Microsoft Fabric

Create Your First Fabric Notebook

Create a Microsoft Fabric notebook, attach it to a Lakehouse, build a small dataset with PySpark, write it as a Delta table and query it with Spark SQL, with every step explained.

By JaviPublished 11 min readBeginner

A Lakehouse with no tables is just structure. In this tutorial you put the first data into it: you create a notebook, attach it to lh_learning, build a tiny sales dataset with PySpark, write it as a Delta table called bronze_sales, and query it with Spark SQL.

The dataset is created in code, so you do not need to download or upload anything. The point is not the five rows. It is to see each step clearly: what a Spark session is, what a DataFrame is, what “write as Delta” actually produces, and how the same table becomes visible to SQL.

This tutorial continues from Create Your First Fabric Lakehouse. You need the jb-fabric-dev workspace and the lh_learning Lakehouse from there.

What a Fabric notebook is

A Fabric notebook is an interactive document made of cells. Code cells run against a Spark session, Markdown cells explain what the code does, and results appear under each cell as tables, charts or text. Notebooks are items in a workspace: they are saved automatically, can be shared, versioned in Git, scheduled, and run from pipelines.

A few terms you will meet immediately:

  • Spark session. The connection between your notebook and a Spark cluster. It starts when you run the first cell and stays alive while you work, so later cells reuse it. In code, the session is the spark object, created for you.
  • PySpark. The Python API for Spark, and the default language of a Fabric notebook. You write Python; Spark distributes the work.
  • Spark SQL. SQL that runs on the same Spark engine against the same tables. You can switch a single cell to SQL with the %%sql magic.
  • Attached Lakehouse. A notebook can be attached to one or more Lakehouses. One of them is the default Lakehouse: table names without a prefix resolve to it, and relative paths such as Files/landing point to it.

Why notebooks are useful

  • Fast feedback. Run a cell, look at the result, adjust. Exploring data this way is much faster than editing and redeploying a job.
  • Code and explanation together. Markdown cells document the reasoning next to the code that implements it.
  • Real engineering power. PySpark handles parsing, joins, deduplication and MERGE logic that visual tools struggle with, and scales from a few rows to billions.
  • A path to production. The same notebook can later be parameterized, run from a pipeline and promoted through environments. Microsoft Fabric Notebook Parameters covers that step.

Create the notebook and attach the Lakehouse

  1. Open the jb-fabric-dev workspace and open lh_learning.

  2. From the Lakehouse, choose the option to open a new notebook.

    Creating the notebook from the Lakehouse attaches lh_learning automatically as its default Lakehouse. You see it in the notebook’s explorer pane, with its Tables and Files.

  3. Rename the notebook to nb_first_sales.

  4. Confirm that the notebook language is PySpark (Python), the default.

If you create the notebook from the workspace instead, select or attach lh_learning from the notebook’s explorer pane. If several Lakehouses are attached, the pinned one is the default. Restart the session after changing the default Lakehouse.

Step 1: create a DataFrame

Paste this into the first cell and run it with Shift+Enter:

PySpark
from datetime import date
from decimal import Decimal

from pyspark.sql.types import (
    DateType,
    DecimalType,
    IntegerType,
    StringType,
    StructField,
    StructType,
)

schema = StructType([
    StructField("order_id", IntegerType(), nullable=False),
    StructField("customer", StringType(), nullable=False),
    StructField("amount", DecimalType(10, 2), nullable=False),
    StructField("order_date", DateType(), nullable=False),
])

rows = [
    (1001, "Contoso",   Decimal("250.00"), date(2026, 10, 1)),
    (1002, "Fabrikam",  Decimal("99.90"),  date(2026, 10, 1)),
    (1003, "Contoso",   Decimal("40.10"),  date(2026, 10, 2)),
    (1004, "Northwind", Decimal("310.75"), date(2026, 10, 2)),
    (1005, "Fabrikam",  Decimal("12.50"),  date(2026, 10, 3)),
]

orders_df = spark.createDataFrame(rows, schema)
PySpark28 linesRead-only example cell. Nothing on this page runs code.

What this does:

  • The schema is explicit. Each column has a name, a type and a nullability rule. amount is a DECIMAL(10,2), not a floating-point number, because money should not suffer rounding errors.
  • rows is plain Python. A list of tuples in the driver’s memory, the same as in any Python script.
  • spark.createDataFrame() turns that list into a Spark DataFrame: a distributed, typed table that Spark can process in parallel.

The cell finishes almost immediately and prints nothing. Spark is lazy: it records what you asked for and only does the work when an action, such as displaying or writing, needs a result.

Step 2: display it

display(orders_df)

display() renders an interactive table under the cell, where you can sort columns and switch to a chart view. It is Fabric’s notebook tool for looking at data. For a plain-text result, orders_df.show() prints the first rows instead.

You should see five orders, from 1001 to 1005.

Step 3: inspect the schema

orders_df.printSchema()
print(f"Rows: {orders_df.count()}")

Output:

root
 |-- order_id: integer (nullable = false)
 |-- customer: string (nullable = false)
 |-- amount: decimal(10,2) (nullable = false)
 |-- order_date: date (nullable = false)

Rows: 5

Checking the schema is a habit worth building now. When data comes from files instead of code, wrong types are the most common source of quiet bugs: an amount read as a string, a date read as text, an ID that lost its leading zeros. Note that count() is an action: it makes Spark run the work and return a number.

Step 4: a simple transformation

Bronze data should stay close to the source, so the transformation you persist is small: add load metadata that answers “when and from where did this row arrive?”.

PySpark
from pyspark.sql import functions as F

bronze_df = (
    orders_df
    .withColumn("source_system", F.lit("tutorial_inline"))
    .withColumn("ingested_at", F.current_timestamp())
)

display(bronze_df)
PySpark9 linesRead-only example cell. Nothing on this page runs code.
  • withColumn() returns a new DataFrame with one more column. DataFrames are immutable: orders_df is unchanged.
  • F.lit() adds a constant value. F.current_timestamp() adds the time of the run.

Now a transformation you only look at, without saving it: daily totals.

PySpark
daily_df = (
    orders_df
    .groupBy("order_date")
    .agg(
        F.count("*").alias("orders"),
        F.sum("amount").alias("revenue"),
    )
    .orderBy("order_date")
)

display(daily_df)
PySpark11 linesRead-only example cell. Nothing on this page runs code.

You should see three days, with revenue of 349.90, 350.85 and 12.50. This is the kind of logic that will later build gold_daily_sales. Keeping it out of Bronze is deliberate: Bronze records what arrived, and business logic belongs in later layers.

Step 5: write it as a Delta table

PySpark
(
    bronze_df.write
    .format("delta")
    .mode("overwrite")
    .saveAsTable("bronze_sales")
)
PySpark6 linesRead-only example cell. Nothing on this page runs code.

What happens when this runs:

  • format("delta") writes Delta Lake. It is already the default for tables in Fabric; writing it explicitly makes the intent obvious to the next reader.
  • mode("overwrite") replaces the table if it exists, so you can run the notebook again and get the same result. That makes sense for a tutorial. A real Bronze load usually appends new batches instead, so history is never lost.
  • saveAsTable("bronze_sales") writes the data and registers the table. Because the name has no schema prefix, it goes to the dbo schema of the default Lakehouse. The full name is lh_learning.dbo.bronze_sales. To write to another schema, use a prefix such as saveAsTable("sales.bronze_orders").

Now look at the Lakehouse in the Explorer pane. Refresh Tables → dbo and bronze_sales appears. Select it to preview the rows.

Step 6: query it with Spark SQL

Add a new cell, put the %%sql magic on its first line, and run:

%%sql
SELECT *
FROM bronze_sales
Spark SQL3 linesRead-only example cell. Nothing on this page runs code.

The same five rows come back, this time from the table rather than from the DataFrame in memory. The %%sql magic switches only this cell to Spark SQL; the rest of the notebook stays PySpark.

SQL is also convenient for quick aggregations:

%%sql
SELECT order_date,
       COUNT(*)    AS orders,
       SUM(amount) AS revenue
FROM bronze_sales
GROUP BY order_date
ORDER BY order_date
Spark SQL7 linesRead-only example cell. Nothing on this page runs code.

You can mix the two languages from Python as well. spark.sql() returns a DataFrame you can keep working with:

PySpark
large_orders_df = spark.sql("""
    SELECT order_id, customer, amount
    FROM bronze_sales
    WHERE amount >= 100
""")

display(large_orders_df)
PySpark7 linesRead-only example cell. Nothing on this page runs code.

What happened

  1. Python / PySparkList of tuples in the driver
  2. Spark DataFrameTyped, distributed, lazy
  3. Delta filesParquet plus _delta_log
  4. Lakehouse tablelh_learning.dbo.bronze_sales
From Python objects to a Lakehouse table. The DataFrame lives in the Spark session; the Delta table lives in OneLake and outlives the session.
  1. Python held five tuples in memory.
  2. createDataFrame() turned them into a Spark DataFrame with an explicit schema. Nothing was stored yet.
  3. saveAsTable() started a Spark job that wrote Parquet files and a Delta transaction log into the table’s folder in OneLake, under the Lakehouse’s Tables/dbo area.
  4. The Lakehouse registered the table, so Spark SQL can query it by name.
  5. The SQL analytics endpoint picks up the new table through its metadata sync, so T-SQL users can query it too, with no copy.

The important shift: the DataFrame disappears when the session ends; the table does not. Close the notebook, come back tomorrow, and SELECT * FROM bronze_sales still works. You can confirm it from the SQL analytics endpoint of lh_learning:

SQL query: What happenedT-SQL (Fabric Warehouse)
SELECT TOP (10) *
FROM dbo.bronze_sales
ORDER BY order_id;
T-SQL (Fabric Warehouse)3 linesRead-only query display. It is not connected to a database.

PySpark vs SQL inside a notebook

You do not have to choose one language per notebook. Use each where it reads best.

Task Convenient choice Why
Creating data, reading files, calling APIs PySpark Python handles paths, options, loops and libraries
Exploring a table quickly Spark SQL SELECT … GROUP BY is short and familiar
Multi-step transformations PySpark Each step is a named DataFrame you can test and reuse
Joins and aggregations Either SQL for readability, PySpark when logic is built dynamically
Parameterized or reusable logic PySpark Functions, parameters and conditionals
Sharing a query with SQL users Spark SQL The logic transfers to the SQL analytics endpoint with few changes

A good rule: PySpark for the pipeline, SQL for the questions. Load and transform data in PySpark, and use %%sql cells to check the results the way a SQL user would. The PySpark select and filter cheat sheet and the Delta read and write cheat sheet are useful next to this notebook.

Stop the session and commit the notebook

When you finish, stop the Spark session from the notebook. An idle session expires on its own after a timeout, but until then it holds capacity that other work could use.

Then, if your workspace is connected to Git:

  1. Open the workspace’s source control panel. nb_first_sales is listed as a new change.
  2. Commit it with the message Add first sales notebook.

The repository now has a folder for nb_first_sales under fabric/, containing the notebook’s source code. The bronze_sales table is not in Git; it lives in OneLake. Rerunning the notebook in another workspace rebuilds it, which is exactly why code belongs in Git and data does not.

Common mistakes

  • No default Lakehouse. Without one, saveAsTable("bronze_sales") has nowhere sensible to go. Always check the Explorer pane before writing.
  • Forgetting that Spark is lazy. A cell that “ran instantly” may not have done the work yet; errors can appear later, at the first action.
  • Displaying or collecting large data. display() and collect() bring results to the notebook. Filter or aggregate first, and avoid collect() on large DataFrames.
  • Relying on inferred types. inferSchema is convenient for exploration, but production loads should declare schemas, as you did here.
  • Using overwrite for real Bronze loads. It is right for a tutorial and wrong for history. Real ingestion appends or merges.
  • Leaving sessions running. Idle sessions consume capacity until they time out.
  • Hard-coding secrets or another workspace’s paths in cells. Both end up in Git and break when the notebook moves between environments.

What you learned

  • A Fabric notebook runs PySpark, Spark SQL and other languages against a Spark session, with a default Lakehouse for table names and relative paths.
  • You created nb_first_sales from lh_learning, so the Lakehouse was attached as the default.
  • You built a DataFrame with an explicit schema, displayed it, inspected its schema and added load metadata.
  • saveAsTable("bronze_sales") wrote a Delta table into the dbo schema of the default Lakehouse, as Parquet files plus a transaction log in OneLake.
  • %%sql cells and spark.sql() query the same table with SQL, and the SQL analytics endpoint sees it without a copy.
  • PySpark suits loading and multi-step transformations; SQL suits quick questions and logic shared with SQL users.

Next: Explore Data with PySpark in Microsoft Fabric

You now have a workspace, a Git connection, a Lakehouse and a Delta table written from code. The next step is the everyday DataFrame toolkit: select, filter, groupBy, agg, orderBy and joins, used to explore and shape real sales data: Explore Data with PySpark in Microsoft Fabric. After that, step 5 explains what the Delta table you just wrote actually is (files, log, versions, and what the SQL endpoint can do with it): Delta Tables in Microsoft Fabric for SQL Server Engineers.

Previous step: Create Your First Fabric Lakehouse. When notebooks start to feel slow, read How to Make Microsoft Fabric Notebooks Faster.

Tags

  • Microsoft Fabric
  • Fabric Notebook
  • PySpark
  • Spark SQL
  • Delta Lake