Microsoft Fabric

Microsoft Fabric Real-Time Intelligence: Eventstream, Eventhouse and KQL Hands-On Guide

A hands-on guide to Microsoft Fabric Real-Time Intelligence: send service telemetry through an eventstream into an Eventhouse, query and window it with KQL, compare Eventstream, KQL and Spark Structured Streaming, and break and fix the flow on purpose.

Por JaviPublicado 24 min de lecturaIntermedio

Batch pipelines answer “what happened yesterday?”. Real-Time Intelligence answers “what is happening now?”, and it changes the tools: events arrive continuously, time becomes the main dimension of every question, and the query language is KQL rather than SQL or PySpark.

This guide builds one small, realistic flow: service telemetry monitoring. Events from an application go through an eventstream into an Eventhouse, where you query them with KQL, group them into time windows, find failing and slow services, and then break the flow on purpose to practise diagnosing it.

How to use this guide

Work in this order. Each step builds on the one before it:

  1. Learn basic KQL with the KQL for Microsoft Fabric Cheat Sheet.
  2. Query static sample data first: the cheat sheet generates a practice table in any KQL database.
  3. Learn time filtering and aggregation (ago(), summarize, bin()).
  4. Create an eventstream (Step 2).
  5. Route events to an Eventhouse (Step 4).
  6. Query live data (Steps 5–9).
  7. Add windowing, in KQL and in the eventstream (Windowing).
  8. Break and fix one failure (Break / fix).
  9. Compare Eventstream and KQL with Spark Structured Streaming (Choose a streaming engine).
  10. Revisit your DP-700 readiness checks if you’re preparing for the exam.

The mental model

  1. 1Event sourceApp, IoT, CDC, Kafka, sample data
  2. 2EventstreamIngest, filter, window, route
  3. ingest
    3EventhouseKQL database tables
  4. query
    4KQLFilter, summarize, bin, join
  5. 5ActDashboard, Activator, downstream
Real-Time Intelligence, end to end. Events flow from the source to an action. The eventstream moves and optionally transforms them in flight; the Eventhouse stores them; KQL turns them into answers, dashboards and alerts.

The pieces

Piece Role Think of it as
Eventstream Captures events from a source, optionally transforms them with no-code operators (Filter, Manage fields, Aggregate, Group by, Union, Expand, Join) or a SQL operator (preview), and routes them to one or more destinations The conveyor belt, not the warehouse: it keeps events for a limited time and doesn’t answer queries
Eventhouse A container for one or more KQL databases, sharing capacity across them, designed for time-based event data Where events live and get queried; creating an Eventhouse also creates a default KQL database with the same name
KQL database Holds tables, functions, materialized views and update policies; each has an embedded KQL queryset for exploration The database you actually query
KQL queryset A saved place to write and run KQL against a database Your query editor
KQL The read-only query language of the Eventhouse, built as a pipeline of operators with first-class time handling The language that turns events into time-windowed answers
Spark Structured Streaming Spark’s streaming engine: a stream treated as a table that keeps growing, processed in micro-batches, usually written to Delta tables in a Lakehouse Code-first stream processing for engineering teams already using Spark

Two Microsoft facts shape the design. An eventstream stores event data for at most 90 days and delivers events at least once, so downstream logic must tolerate duplicates. And eventstreams need at least an F4 capacity.

Choose a streaming engine

There isn’t one right engine. The question is where the work belongs: in flight, in the store, or in code.

Need Likely fit Why
Low-code ingestion and routing from a supported source Eventstream Built-in connectors, no-code operators, several destinations from one stream
Simple filtering, field cleanup or routing before storage Eventstream Filter and Manage fields run in flight; a derived stream can feed several destinations
KQL-first real-time analytics on high-volume, time-oriented telemetry Eventstream + Eventhouse The Eventhouse is designed for time-based events; KQL handles filtering, aggregation and windows at query time
Reshaping data inside the store as it arrives Eventhouse (KQL update policies, materialized views) Transformations defined once in KQL run on ingestion, without another engine
Fixed-window aggregates computed before storage Eventstream Group by or KQL Group by gives tumbling, hopping, sliding, session and snapshot windows in flight; KQL bin() gives fixed buckets at query time. Both are valid; choose by whether you need the raw events too
Complex transformations, custom stateful logic, joins with Delta tables, Python libraries Spark Structured Streaming Full Spark and Python, checkpointed state, Delta Lake as the sink
A team that already writes and tests PySpark Spark Structured Streaming, possibly fed by an eventstream Reuses skills, code review and testing; an eventstream can still be the front door
Results needed hourly or daily, not within seconds Batch (pipeline or notebook) Streaming adds always-on cost and operations; latency should justify it

A common production shape uses more than one: an eventstream ingests and routes, the Eventhouse serves real-time queries and dashboards, and a Lakehouse (through an eventstream destination or Spark) keeps the history for engineering and batch analytics. ETL vs ELT vs CDC vs Streaming covers where streaming fits among the other integration patterns.

Prerequisites

  • A workspace on a Fabric capacity of at least F4, or a trial capacity, where you are a Contributor or higher.
  • A Fabric notebook to send test events (Step 3). It installs the Azure Event Hubs Python package.
  • Basic KQL: sections 1–6 of the KQL cheat sheet.

The guide uses these names: workspace ws_rti_lab, Eventhouse eh_telemetry, eventstream es_telemetry, custom endpoint src_telemetry_app, table TelemetryEvents.

Hands-on: service telemetry monitoring

Each event describes one request to a service:

{
  "Timestamp": "2026-11-09T10:15:42.123+00:00",
  "DeviceId": "dev-17",
  "Service": "payments",
  "Region": "westeurope",
  "EventType": "request",
  "DurationMs": 212,
  "Status": "Succeeded",
  "Value": 48.31,
  "Properties": { "version": "1.4.2", "tier": "gold" }
}

Step 1: Create the Eventhouse

  1. In ws_rti_lab, select New item, search for Eventhouse, name it eh_telemetry and select Create.

Expected result: the Eventhouse opens on its system overview, and a KQL database named eh_telemetry exists inside it.

Step 2: Create the eventstream and a custom endpoint

  1. Create a new Eventstream item named es_telemetry.
  2. On the get-started page, select Use custom endpoint, name the source src_telemetry_app and select Add.
  3. Select Publish.
  4. In Live view, select the custom endpoint and open its Details pane. On the Event Hub tab, open SAS Key Authentication and copy Connection string-primary key.

Expected result: a published eventstream with a custom endpoint source, and a connection string that ends with EntityPath=es_.... Treat it as a secret: anyone with it can send events.

Step 3: Send sample events

Microsoft’s Show sample code button gives Java samples. The equivalent below is my own code, using the Azure Event Hubs SDK for Python, which works with the custom endpoint’s Event Hub connection string. Run it from a notebook.

%pip install azure-eventhub
Python
import json
import random
import time
from datetime import datetime, timezone

from azure.eventhub import EventData, EventHubProducerClient

CONNECTION_STRING = "<Connection string-primary key>"  # lab only; use Key Vault in real code
SERVICES = ["checkout", "payments", "search", "catalog", "auth"]
REGIONS = ["westeurope", "eastus", "southeastasia"]


def make_event():
    service = random.choice(SERVICES)
    failed = random.random() < (0.12 if service == "payments" else 0.03)
    return {
        "Timestamp": datetime.now(timezone.utc).isoformat(),
        "DeviceId": f"dev-{random.randrange(40)}",
        "Service": service,
        "Region": random.choice(REGIONS),
        "EventType": "error" if failed else "request",
        "DurationMs": 40 + random.randrange(300) + (400 if service == "search" else 0),
        "Status": "Failed" if failed else "Succeeded",
        "Value": round(random.random() * 100, 2),
        "Properties": {"version": random.choice(["1.4.2", "1.5.0"]), "tier": random.choice(["gold", ""])},
    }


producer = EventHubProducerClient.from_connection_string(CONNECTION_STRING)
with producer:
    for _ in range(60):  # about 10 minutes
        batch = producer.create_batch()
        for _ in range(20):
            batch.add(EventData(json.dumps(make_event())))
        producer.send_batch(batch)
        time.sleep(10)
Python36 líneasCelda de ejemplo de solo lectura. Nada en esta página ejecuta código.

The data has the same shape as the cheat sheet’s practice table: payments fails about four times more often than the others, and search is about 400 ms slower.

Expected result: the eventstream’s data preview shows JSON events arriving, about two per second.

If you can’t run code, add the Sample data source instead (for example Bicycles). The steps are the same, but you’ll need to adapt the queries to its columns.

Step 4: Route events to the Eventhouse

  1. Switch the eventstream to Edit mode, select Add destination, then Eventhouse.
  2. Choose Direct ingestion, enter a destination name, the workspace and eh_telemetry, and select Save. Connect the destination to the stream if it isn’t connected, then select Publish.
  3. In Live view, select Configure on the Eventhouse destination. The Eventhouse Get data screen opens.
  4. Select New table, name it TelemetryEvents, keep or rename the data connection and select Next.
  5. On Inspect the data, choose the JSON format. Use Edit columns if a type was inferred wrongly: Timestamp should be datetime, DurationMs long, Value real, Properties dynamic.
  6. Select Finish, review the Summary and select Close.

Direct ingestion writes events to the Eventhouse unchanged. The other mode, Event processing before ingestion, is the one to use when operators such as Filter or Group by sit in front of the destination (Windowing).

Expected result: the Eventhouse destination shows as active in Live view, and TelemetryEvents appears in the KQL database within a few minutes.

Step 5: Inspect the schema

Open the KQL database’s queryset and run:

TelemetryEvents
| getschema
TelemetryEvents
| take 10

Expected result: nine columns with the types set in Step 4, and rows that match the JSON events. Fix types now: every later query depends on them, and a Timestamp stored as string breaks every time filter.

Step 6: Query recent data

TelemetryEvents
| where Timestamp > ago(15m)
| summarize Events = count(), Services = dcount(Service), FirstEvent = min(Timestamp), LastEvent = max(Timestamp)

Expected result: roughly 120 events per minute since the sender started, five services, and a LastEvent within seconds of now.

Step 7: Aggregate by time window

TelemetryEvents
| where Timestamp > ago(30m)
| summarize Requests = count(), P95Ms = percentile(DurationMs, 95)
    by bin(Timestamp, 1m), Service
| render timechart

Expected result: five lines of about 24 requests per minute each, with search clearly higher on latency if you chart P95Ms.

Step 8: Identify failures

TelemetryEvents
| where Timestamp > ago(30m)
| summarize Events = count(), Failures = countif(Status == "Failed") by Service
| extend FailureRate = round(100.0 * Failures / Events, 1)
| order by FailureRate desc

Expected result: payments near 12%, the others near 3%.

Step 9: Identify high-latency services

Create a small reference table with each service’s latency target, then compare:

.set-or-append ServiceCatalog <|
datatable(Service: string, Team: string, Tier: string, SloMs: long)
[
    "checkout", "Commerce", "Tier 1", 400,
    "payments", "Commerce", "Tier 1", 400,
    "search",   "Discovery", "Tier 2", 500,
    "catalog",  "Discovery", "Tier 2", 600
]
TelemetryEvents
| where Timestamp > ago(30m)
| summarize P95Ms = percentile(DurationMs, 95) by Service
| lookup kind=leftouter ServiceCatalog on Service
| extend OverSlo = P95Ms > SloMs
| order by P95Ms desc

Expected result: search over its SLO (p95 around 700 ms against 500 ms), and auth with an empty team and SLO because the catalog doesn’t list it. That empty row is a finding too: a service nobody has set a target for.

Step 10: Decide what you’d monitor in production

Write down, for this flow:

  • Freshness: how old is the newest event? Alert when the gap exceeds a threshold.
  • Volume: events per minute against a baseline. A sudden drop is often a broken source, not a quiet system.
  • Errors: failure rate per service, not raw failure counts.
  • Latency: p95 against the SLO per service.
  • Schema health: the share of rows with empty or null critical fields (Break 1).

A freshness check to start from:

TelemetryEvents
| summarize LastEvent = max(Timestamp)
| extend MinutesSinceLastEvent = datetime_diff("minute", now(), LastEvent)

Queries like this become Real-Time Dashboard tiles or Activator rules. Monitoring Fabric pipelines and notebooks covers the batch side of the same discipline.

Windowing, three ways

A window turns an endless stream into finite groups you can count and average. Fabric gives you three places to do it, and the same words don’t always mean the same thing.

  • Eventstream Group byIn flight, before storage
    1. Tumbling
    2. Hopping
    3. Sliding
    4. Session
    5. Snapshot
  • KQL in the EventhouseAt query time, over stored events
    1. bin()Fixed buckets, like tumbling
    2. make-seriesFixed buckets, gaps filled
    3. Materialized viewKept up to date as data arrives
  • Spark Structured StreamingIn your job, state in a checkpoint
    1. window(ts, size)Tumbling
    2. window(ts, size, slide)Sliding / hopping
    3. withWatermark()How long to wait for late events
Where the window is computed. The same idea, three engines. Eventstream windows run in flight and store only the result; KQL bins run at query time over stored events; Spark windows run in your streaming job and keep state in a checkpoint.
Window Meaning Typical question
Tumbling Fixed, non-overlapping intervals; every event in exactly one window Requests per minute
Hopping Fixed size, moving by a smaller hop; windows overlap Requests in the last 5 minutes, every minute
Sliding Output when the content of the window changes Alert as soon as 5 failures happen within 1 minute
Session Groups events separated by less than a timeout Activity bursts per device
Snapshot Groups events with the same timestamp Events that arrived together

Exercise: the same window in the eventstream and in KQL

  1. In es_telemetry, switch to Edit mode. Add a Group by operator from the stream: group by Service, aggregate the average of DurationMs, window type Tumbling, duration 1 minute.
  2. Add an Eventhouse destination after the operator. Event processing before ingestion is preselected. Send the output to a new table TelemetryPerMinute, then Publish.
  3. After a few minutes, compare the eventstream result with KQL computed over the raw events:
TelemetryEvents
| where Timestamp > ago(10m)
| summarize AvgMs = avg(DurationMs) by bin(Timestamp, 1m), Service
| order by Timestamp desc, Service asc
  1. Keep the same aggregate in the Eventhouse without another engine, as a materialized view:
.create materialized-view TelemetryPerMinuteMV on table TelemetryEvents
{
    TelemetryEvents
    | summarize Requests = count(), Failures = countif(Status == "Failed"), AvgMs = avg(DurationMs)
        by Service, bin(Timestamp, 1m)
}

Expected result: similar per-minute averages from both paths, but not always identical at window edges. The eventstream computes each window in flight as events pass through and emits only the aggregate; the KQL query computes windows over every stored event, using the Timestamp column, including events that arrived late. Write down which one you’d trust for an alert, and which for a report.

Things to notice: the eventstream path stores only one row per service per minute, which is cheap but can’t be re-sliced later; the KQL path keeps every event and lets you change the window afterwards. That’s the core design trade-off.

Spark Structured Streaming in brief

Structured Streaming treats a stream as a table that new rows keep being appended to. You write a normal DataFrame query, and Spark runs it continuously in micro-batches.

Concept What it means
Streaming DataFrame Created with spark.readStream instead of spark.read
Source Where events come from: files, Kafka, Azure Event Hubs, or an eventstream
Sink Where results go; in Fabric usually a Delta table in a Lakehouse (writeStream ... toTable())
Checkpoint A folder where Spark records progress and state. It makes restarts resume where they stopped instead of reprocessing or skipping data; one per query, never shared
Output mode append (only new final rows, the usual choice for Delta), complete (rewrite the whole result) or update (changed rows only)
Watermark How long to wait for late events before a window is final
Trigger When micro-batches run: continuously, every N minutes, or availableNow to process what’s there and stop

A minimal, self-contained example on a Lakehouse: land a few JSON files, then aggregate them into 1-minute windows with a stream.

PySpark
import random
from datetime import datetime, timedelta, timezone

from pyspark.sql import functions as F
from pyspark.sql.types import LongType, StringType, StructField, StructType, TimestampType

schema = StructType([
    StructField("Timestamp", TimestampType()),
    StructField("Service", StringType()),
    StructField("Status", StringType()),
    StructField("DurationMs", LongType()),
])

# Land a batch of test events as JSON files (run this cell again to add more)
now = datetime.now(timezone.utc)
rows = [
    (now - timedelta(seconds=9 * i),
     random.choice(["checkout", "payments", "search"]),
     "Failed" if random.random() < 0.05 else "Succeeded",
     40 + random.randrange(300))
    for i in range(500)
]
spark.createDataFrame(rows, schema).write.mode("append").json("Files/telemetry/landing")
PySpark23 líneasCelda de ejemplo de solo lectura. Nada en esta página ejecuta código.
PySpark
events = spark.readStream.schema(schema).json("Files/telemetry/landing")

per_minute = (
    events
    .withWatermark("Timestamp", "2 minutes")
    .groupBy(F.window("Timestamp", "1 minute"), "Service")
    .agg(
        F.count("*").alias("Requests"),
        F.sum(F.when(F.col("Status") == "Failed", 1).otherwise(0)).alias("Failures"),
    )
    .select(F.col("window.start").alias("WindowStart"), "Service", "Requests", "Failures")
)

query = (
    per_minute.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "Files/checkpoints/telemetry_per_minute")
    .trigger(availableNow=True)
    .toTable("telemetry_per_minute")
)
query.awaitTermination()
PySpark22 líneasCelda de ejemplo de solo lectura. Nada en esta página ejecuta código.

Things to try, one at a time:

  • Run the second cell twice without new files. Thanks to the checkpoint, the second run should add nothing.
  • Add files, run it again. Only new windows should be written, and in append mode the newest windows appear only once the watermark has passed them.
  • Change the window to F.window("Timestamp", "5 minutes", "1 minute"): a sliding (hopping) window, where each event lands in five windows.
  • Delete the checkpoint folder and rerun: the job starts from scratch and can write duplicate windows. That’s why the checkpoint is part of the job, not a temporary file.

Microsoft’s Lakehouse streaming guidance adds what this example leaves out: an Event Hubs source, Optimized Write, trigger intervals for well-sized files, and running production streams as Spark job definitions with a retry policy rather than in a notebook. An eventstream can also hand events to Spark through its Spark Notebook destination (preview).

One example isn’t Structured Streaming mastery. State, late data and recovery are where streaming jobs fail in practice, and they need practice of their own.

Break and fix

Do each break in the lab workspace, find the evidence before you fix it, then write one line each for symptom, evidence, cause and fix.

Break 1: schema mismatch

Break it. In the sender, rename one field, as a producer team might in a new release: change the "DurationMs" key to "LatencyMs" in make_event() and run it for a few minutes.

Expected result: no error anywhere. The ingestion mapping created in Step 4 still points at DurationMs, so the new property isn’t mapped and new rows arrive with an empty DurationMs. The renamed field is visible in the eventstream’s data preview.

Find the evidence:

TelemetryEvents
| where Timestamp > ago(30m)
| summarize Rows = count(), MissingDuration = countif(isnull(DurationMs)) by bin(Timestamp, 1m)
| order by Timestamp desc

The minute the rename shipped shows up as the point where MissingDuration jumps from zero to every row. Check the eventstream’s data preview for the field name actually being sent. If rows are missing entirely rather than arriving with empty values, look for ingestion failures in the KQL database as well.

Fix it. Correct the sender. If you can’t change the producer quickly, rename the field in flight with a Manage fields operator, which means the destination needs Event processing before ingestion. Then decide what to do with the rows already ingested without a duration.

Break 2: wrong time filter

Break it. In the sender, shift the timestamp two hours into the past to simulate a producer that sends local time labelled as UTC: change the Timestamp line to (datetime.now(timezone.utc) - timedelta(hours=2)).isoformat() and add timedelta to the datetime import.

Expected result: a dashboard query such as where Timestamp > ago(15m) shows no new events, although the eventstream shows events arriving.

Find the evidence: compare the event time with the time the Eventhouse ingested the row.

TelemetryEvents
| where ingestion_time() > ago(15m)
| summarize Rows = count(), NewestEventTime = max(Timestamp), NewestIngestion = max(ingestion_time())
| extend SkewMinutes = datetime_diff("minute", NewestIngestion, NewestEventTime)

A skew of about 120 minutes says the events are fresh but their timestamps aren’t. ingestion_time() is available on tables with the ingestion time policy, which is on by default.

Fix it. Send UTC timestamps with an explicit offset. Decide whether dashboards filter on event time (what happened when) or ingestion time (what arrived when): both are legitimate, but they answer different questions.

Break 3: the wrong window size

Break it. Make one service fail for two minutes. In the sender, add burst_until = time.time() + 120 just before producer = ..., and change the failed = ... line in make_event() to:

failed = (service == "payments" and time.time() < burst_until) or random.random() < 0.03

Run it for a few minutes, then chart failures per hour:

TelemetryEvents
| where Timestamp > ago(3h)
| summarize Failures = countif(Status == "Failed") by bin(Timestamp, 1h), Service
| render timechart

Expected result: the burst disappears into the hourly total. payments looks slightly worse than usual, not broken.

Find the evidence and fix it: rerun with bin(Timestamp, 1m) and with make-series ... step 1m default = 0. The burst becomes obvious. Choose the window for the decision it supports: one minute for alerting, one hour for trend reports, and say which one a chart uses.

Cleanup

  1. Stop the sender cell.
  2. Delete es_telemetry. This stops ingestion and frees the eventstream’s capacity use.
  3. Delete eh_telemetry to remove its databases, tables and the materialized view, or keep it to practise more KQL.
  4. Delete Files/telemetry and Files/checkpoints in the Lakehouse, and the telemetry_per_minute table, if you ran the Spark example.

Next study step

  • Practise the queries until you can write them without notes: the KQL cheat sheet has seven exercises on the same data.
  • Compare designs, not just tools: for one source, write down which work belongs in the eventstream, in the Eventhouse and in Spark, and why.
  • If you’re preparing for DP-700, check the streaming items in the readiness checklist.

Sources

Last verified against Microsoft Learn: November 2026.

Etiquetas

  • Microsoft Fabric
  • Ingesta de datos
  • Arquitectura de datos
  • Monitorización
  • PySpark