Fabric Medallion Architecture1 prévus

  • PrévuMicrosoft FabricIntermédiaire

    Designing a Gold Layer for Reporting and APIs

    How to design the Gold layer that reports, semantic models and APIs consume: modelling choices, stable contracts, serving engines and how to change it without breaking consumers.

    Sujets prévus

    The curated layer is the contract with consumers. Its design determines how easy it is to change the platform underneath.

    • what belongs in the curated layer
    • dimensional vs wide tables
    • naming and data types as a contract
    • versioning changes
    • serving through SQL endpoint or Warehouse
    • serving APIs: latency, concurrency, pagination and when to use a different store
    • data quality checks
    • Gold
    • Microsoft Fabric
    • Data Architecture
    • Medallion Architecture

Fabric Performance at Scale1 prévus

  • PrévuDelta LakeIntermédiaire

    Partitioning Delta Tables in Microsoft Fabric: When, How and Liquid Clustering

    When to partition a Delta table at all, how to choose the column, the mistakes that create small files, how partitioning differs across Medallion layers, and when liquid clustering is the better choice.

    Sujets prévus

    Partitioning helps only when a table is large and queries filter on the partition column; otherwise it creates small files that are expensive to undo. This guide covers when to partition, how to choose the column, how the answer changes across Medallion layers and when clustering is the better tool.

    • when not to partition at all
    • choosing a column: data volume per partition, cardinality and common filters
    • write and MERGE patterns, date columns and granularity
    • mistakes that create small files, and their symptoms in file counts and query plans
    • partitioning by Medallion layer: load date in Bronze, large Silver facts, unpartitioned Gold
    • Hive-style Parquet directories under a Delta table
    • liquid clustering as an alternative, and what Fabric currently supports
    • repartitioning an existing table and checking that pruning happens
    • Delta Lake
    • Microsoft Fabric
    • Partitioning
    • Performance

Microsoft Fabric Notebook Performance1 prévus

  • PrévuMicrosoft FabricIntermédiaire

    Common Performance Mistakes in PySpark Notebooks

    Diagnose repeated actions, driver collection, shuffle-heavy logic, skew, inappropriate caching and output anti-patterns in Fabric notebooks.

    Sujets prévus

    Convenient interactive patterns can trigger repeated jobs, driver pressure, shuffles, skew and small-file output when moved unchanged into production.

    • Reading more data than needed: column projection, early filters and partition pruning
    • Repeated actions and long lineage
    • `collect`, `toPandas`, Python UDFs and driver limits
    • Join, aggregation, repartition and skew mistakes
    • Cache lifecycle and plan-driven debugging
    • Microsoft Fabric
    • Fabric Notebook
    • Performance
    • PySpark
    • Troubleshooting

SQL Server Performance4 prévus

  • PrévuSQL ServerIntermédiaire

    Reading SQL Server Execution Plans Without Guessing

    A systematic way to read execution plans: estimated vs actual rows, the operators that matter, warnings, and how to find the real cost instead of the highest percentage.

    Sujets prévus

    Execution plans are often read top to bottom looking for the biggest percentage, which regularly points at the wrong operator. The article proposes a repeatable reading order based on row estimates and data flow.

    • how a query becomes a plan: parsing, optimization and the plan cache
    • estimated vs actual plans
    • reading data flow right to left
    • cardinality estimate mismatches
    • the operators that usually matter
    • plan warnings: spills, implicit conversions, missing indexes
    • why cost percentages mislead
    • a checklist for every plan
    • SQL Server
    • Execution Plans
    • Performance
  • PrévuSQL ServerIntermédiaire

    When a Missing Index Recommendation Is Actually a Bad Idea

    Why SQL Server's missing index suggestions are a starting point and not an instruction, and how to evaluate them against the whole workload.

    Sujets prévus

    Missing index hints are generated per query, ignore existing indexes and know nothing about write cost. Applying them blindly leads to overlapping indexes and slower writes.

    • how missing index suggestions are produced
    • known limitations of the DMVs
    • overlap with existing indexes
    • column order and included columns
    • write and storage cost
    • consolidating suggestions across a workload
    • checking index usage and size: unused, overlapping and oversized indexes
    • SQL Server
    • Indexing
    • Execution Plans
  • PrévuSQL ServerIntermédiaire

    SQL Server Statistics: Why Good Indexes Can Still Produce Bad Plans

    How statistics drive cardinality estimates, why they go stale on large tables, and how sampling and update thresholds affect plan quality.

    Sujets prévus

    A correct index is not enough if the optimiser misjudges how many rows it will read. Statistics are usually where that misjudgement starts.

    • what statistics contain: histogram and density
    • auto-update thresholds on large tables
    • sampling rate and skewed data
    • ascending keys
    • cardinality estimator versions
    • checking statistics: last update, modification counter and histogram
    • maintenance strategies
    • SQL Server
    • Statistics
    • Execution Plans
    • Performance
  • PrévuSQL ServerAvancé

    Parameter Sniffing in SQL Server: Diagnose It Before You Fix It

    How to confirm that a regression really is parameter sensitivity before reaching for OPTION(RECOMPILE) or plan forcing, and how to choose between the available fixes.

    Sujets prévus

    Parameter sniffing gets blamed for many slow queries and is often fixed with a blanket recompile hint. The article focuses on proving it first and then picking the least invasive fix.

    • how plans are compiled for parameter values
    • symptoms that look like sniffing but are not
    • confirming with Query Store and plan comparison
    • skewed data distributions
    • fix options and their costs
    • Parameter Sensitive Plan optimisation
    • SQL Server
    • Execution Plans
    • Query Store
    • Performance

SQL Server Troubleshooting & Diagnostics3 prévus

  • PrévuSQL ServerIntermédiaire

    How to Find the Most Expensive Queries in SQL Server: CPU, Reads and Duration

    Rank the SQL Server workload recorded in the plan cache by CPU, logical reads and duration, in total and per execution, and know when the cache stops being trustworthy evidence and Query Store takes over.

    Sujets prévus

    The query that hurts a server most depends on the resource under pressure: CPU, reads or elapsed time. Totals identify cumulative consumers, averages highlight individually expensive executions, and neither is durable history when the evidence comes from the plan cache.

    • Cached aggregate statistics vs currently running requests: what `sys.dm_exec_query_stats` and `sys.dm_exec_procedure_stats` record
    • Total worker time, execution count, total CPU and average CPU
    • Ranking by logical reads and by elapsed time, and when each one matters
    • Separate total and per-execution rankings
    • Cache eviction, recompilation, restart, failover and cache-reset caveats
    • When to switch to Query Store for history
    • SQL Server
    • Performance
    • Troubleshooting
    • DMV
  • PrévuSQL ServerIntermédiaire

    Why SQL Server Blocking Happens: Locks, Transactions, Isolation and Application Design

    Why blocking happens in SQL Server, what makes it persist, and how transaction, query and application design reduce it.

    Sujets prévus

    Blocking is normal in a lock-based database, so the useful question is not how to stop it but why some of it persists. The answer usually sits in transaction scope, query shape and how the application handles errors. This article is the explanation and prevention piece; live incidents belong to the SQL Server Blocking Runbook.

    • why blocking is normal, and when it becomes a problem
    • locks and transaction scope
    • isolation levels and row versioning, including read committed snapshot isolation (RCSI)
    • open and sleeping transactions
    • application transaction handling: timeouts, cancels, error paths and connection pooling
    • long-running transactions
    • lock escalation
    • large DML and batching
    • indexing and query shape
    • client behavior, including result sets that are not fully fetched
    • blocking versus deadlocks
    • blocking prevention patterns
    • design tradeoffs
    • SQL Server
    • Blocking
    • Concurrency
    • Performance
  • PrévuSQL ServerIntermédiaire

    SQL Server Wait Types: What Is the Query Waiting For?

    Use SQL Server request and aggregate waits as evidence while avoiding simplistic wait-to-fix mappings.

    Sujets prévus

    A wait type identifies where execution currently cannot proceed; it does not independently explain why. The query, plan, resource, workload and time window supply the diagnosis.

    • Current request waits and aggregate observation windows
    • `LCK_M_*`, `PAGEIOLATCH_*`, `WRITELOG` and `RESOURCE_SEMAPHORE`
    • `CXPACKET`, `CXCONSUMER`, `ASYNC_NETWORK_IO` and `SOS_SCHEDULER_YIELD`
    • Evidence-driven follow-up rather than “wait X means fix Y”
    • SQL Server
    • Performance
    • Troubleshooting
    • DMV

SQL Server at Scale3 prévus

  • PrévuSQL ServerIntermédiaire

    Batching Large UPDATE and DELETE Operations in SQL Server

    Patterns for changing or deleting millions of rows without long blocking, transaction log growth or lock escalation.

    Sujets prévus

    A single statement that touches millions of rows can fill the transaction log, escalate locks and block the application. Batching fixes that, but only if the batches are designed carefully.

    • why single large statements hurt
    • choosing a batch size
    • key-range batching vs TOP
    • transaction log and recovery model
    • lock escalation
    • progress tracking and restartability
    • diagnosing a large UPDATE that is already running: blocking, log use and rollback cost
    • SQL Server
    • Large Scale Data
    • Performance
    • Concurrency
  • PrévuSQL ServerAvancé

    Moving Hundreds of Millions of Rows Without Destroying SQL Server Performance

    Options for copying or migrating very large tables, from batched inserts to partition switching and bulk load, and how to keep the source system usable meanwhile.

    Sujets prévus

    Large data moves compete with the production workload for I/O, log and locks. The approach needs to be chosen based on downtime tolerance and how much the source can absorb.

    • choosing between batching, bulk load and switching
    • minimal logging requirements
    • throttling and scheduling
    • indexes during the load
    • validating row counts and checksums
    • restarting after failure
    • SQL Server
    • Large Scale Data
    • Data Ingestion
    • Performance
  • PrévuSQL ServerIntermédiaire

    SQL Server MERGE vs INSERT/UPDATE: What I Prefer in Production

    A comparison of MERGE with separate INSERT and UPDATE statements: readability, concurrency behaviour, known issues and the pattern I prefer in production.

    Sujets prévus

    MERGE looks like the natural upsert statement, but its concurrency behaviour and history of issues lead many teams to avoid it. The article explains the trade-offs and states a preference with reasons.

    • what MERGE does in one statement
    • concurrency and race conditions
    • known issues and caveats
    • separate UPDATE then INSERT, with the locking needed to make it safe
    • performance comparison method
    • my production default and why
    • SQL Server
    • MERGE
    • Upsert

Fabric in Production1 prévus

  • PrévuMicrosoft FabricIntermédiaire

    How I Structure Fabric Workspaces for Dev, Test and Production

    A workspace layout for separating environments in Fabric, with naming, permissions and how items reference each other across workspaces.

    Sujets prévus

    Environment separation in Fabric starts with workspace design. Getting it wrong makes deployments and permissions painful later.

    • workspace per environment and per domain
    • naming conventions
    • permissions and roles
    • cross-workspace references and shortcuts
    • capacity assignment per environment
    • what Git and deployment actually move, and what they do not: data, connections and environment-specific IDs
    • promoting changes between environments and rolling back
    • what not to share between environments
    • Microsoft Fabric
    • DevOps
    • CI/CD

Production Data Engineering1 prévus

  • PrévuMicrosoft FabricAvancé

    How to Design Fabric Pipelines for Hundreds of Daily Data Files

    Ingestion design for many small files a day: discovery, batching, parallelism, tracking what was processed and avoiding small-file problems downstream.

    Sujets prévus

    Processing files one at a time does not scale, and processing them all at once is hard to recover. The design has to handle both throughput and traceability.

    • file discovery and landing zones
    • batching files per run
    • parallelism limits
    • tracking processed files
    • handling bad or duplicate files
    • compacting output
    • Microsoft Fabric
    • Data Pipelines
    • Data Ingestion
    • File Sizing

Delta Lake at Scale2 prévus

  • PrévuDelta LakeAvancé

    Handling Concurrent Append Conflicts in Delta Tables

    Why ConcurrentAppendException happens, how to read it, and the design and retry patterns that prevent it.

    Sujets prévus

    Concurrent append conflicts often appear only under production load. The fix is usually in how writers' predicates overlap.

    • optimistic concurrency in Delta and the conflict types
    • what the exception means
    • overlapping read predicates
    • making predicates explicit
    • partitioning to separate writers
    • safe retry logic
    • serialising when necessary
    • Delta Lake
    • Concurrency
    • Debugging
  • PrévuDelta LakeIntermédiaire

    Delta Schema Evolution: Safe Patterns for Production Systems

    Using mergeSchema, column mapping and explicit migrations to evolve Delta tables without breaking writers or readers.

    Sujets prévus

    Automatic schema evolution is convenient and also risky: an unexpected column can spread silently. The article covers when to allow it and when to require a migration.

    • schema enforcement
    • mergeSchema and autoMerge
    • adding, renaming and dropping columns
    • column mapping
    • type changes
    • explicit migrations
    • detecting upstream schema changes in Fabric pipelines before they reach curated tables
    • how Parquet stores schema per file, and why the Delta log matters
    • Delta Lake
    • Schema Evolution

Understanding Parquet1 prévus

  • PrévuParquetDébutant

    Parquet Files Explained for Database Engineers

    The Parquet format explained in database terms: columnar layout, row groups, pages, encodings, compression and metadata.

    Sujets prévus

    Parquet is the storage format under Delta Lake and Fabric. Understanding it explains much of their performance behaviour.

    • row vs columnar storage
    • file structure: row groups, column chunks, pages
    • encodings
    • compression codecs
    • footer metadata and statistics
    • why it is fast for analytics: column pruning, statistics and encodings
    • what Parquet does not do
    • Parquet
    • Performance

Fabric SQL Engineering2 prévus

  • PrévuFabric SQLAvancé

    SQL Analytics Endpoint Performance in Microsoft Fabric

    Why queries on a Lakehouse SQL analytics endpoint are slow or return stale data, and how to diagnose them with query insights, plans, metadata sync checks and table health.

    Sujets prévus

    The SQL analytics endpoint looks like SQL Server but has no indexes to add: its performance comes from the query, the Delta files underneath and a metadata sync that can lag behind Spark. A slow or stale result needs a repeatable way to tell which of these is the cause.

    • why the same query can be slower than on SQL Server, and which SQL Server habits do not apply
    • reproducing the problem: cold vs warm runs
    • metadata sync: recognizing "not visible yet", checking sync state and refreshing manually
    • query insights views
    • reading the plan
    • checking table and file health: file counts, V-Order, statistics and data types
    • ruling out capacity contention before tuning (the Capacity guide covers the rest)
    • fixes by cause
    • SQL Endpoint
    • Performance
    • Debugging
  • PrévuMicrosoft FabricIntermédiaire

    SQL Server Mirroring into Microsoft Fabric

    How to mirror an on-premises SQL Server database into Microsoft Fabric: gateway and network prerequisites, supported versions, setup, limitations and what it takes to run mirroring in production.

    Sujets prévus

    Mirroring an on-premises SQL Server is a different job from mirroring Azure SQL Database: the source sits behind a gateway, version and edition rules apply, and the change feed runs on a server you operate. This article covers what the Azure SQL mirroring guide does not.

    • on-premises SQL Server as a mirroring source, and how it differs from Azure SQL Database
    • gateway and network prerequisites
    • SQL Server version and edition requirements
    • setting up mirroring step by step
    • limitations: unsupported tables, data types and DDL changes
    • operational considerations: monitoring replication, source impact, restarts and re-seeding
    • Microsoft Fabric
    • SQL Server
    • Data Ingestion
    • Incremental Loads

Observability & Debugging1 prévus

  • PrévuSQL ServerAvancé

    SQL Server Deadlocks: A Practical Investigation Workflow

    How to capture and read deadlock graphs, identify the access pattern that causes the cycle, and choose a fix that removes it.

    Sujets prévus

    Deadlocks are usually handled with retries, which hides the cause. Reading the deadlock graph properly usually points to a specific access order or missing index.

    • capturing deadlocks with Extended Events
    • reading a deadlock graph
    • common deadlock patterns
    • access order and index design
    • retry logic and where it belongs
    • verifying the fix
    • SQL Server
    • Concurrency
    • Debugging

Python DataFrames for Data Engineering1 prévus

  • PrévuPythonIntermédiaire

    pandas vs PySpark DataFrames

    Compare pandas and PySpark DataFrames through execution model, scale, APIs, null semantics and practical migration choices.

    Sujets prévus

    Similar-looking APIs hide very different local and distributed execution models. Dataset size alone does not decide when Spark is justified.

    • Eager versus lazy execution and local versus distributed memory
    • Equivalent filters, groups and joins
    • Schema, null and ordering differences
    • Operational cost, debugging, scale and migration boundaries
    • Python
    • pandas
    • PySpark
    • DataFrames
    • Large Scale Data