Articles prévus pour Javi on Data. Aucun n’est encore publié ; les sujets et l’ordre peuvent évoluer pendant la recherche et la rédaction.
Les articles prévus sont d’abord rédigés en anglais.
Create a development workspace with sensible capacity, access and ownership decisions from the start.
Create a development workspace with sensible capacity, access and ownership decisions from the start.
Connect a Fabric workspace to a repository and understand branches, synchronization, versioned artifacts and current limitations.
Connect a Fabric workspace to a repository and understand branches, synchronization, versioned artifacts and current limitations.
Understand how OneLake, workspaces, Lakehouses, Warehouses, notebooks, pipelines, SQL endpoints and semantic models fit together.
Understand how OneLake, workspaces, Lakehouses, Warehouses, notebooks, pipelines, SQL endpoints and semantic models fit together.
Use PySpark to load, inspect and summarize a dataset in your first Fabric notebook.
Use PySpark to load, inspect and summarize a dataset in your first Fabric notebook.
Practice the DataFrame operations used to explore and shape data in Fabric notebooks.
Practice the DataFrame operations used to explore and shape data in Fabric notebooks.
Query notebook data with Spark SQL and decide when SQL or PySpark is the clearer tool.
Query notebook data with Spark SQL and decide when SQL or PySpark is the clearer tool.
Build a Lakehouse and understand its Files, Tables and SQL analytics endpoint surfaces.
Build a Lakehouse and understand its Files, Tables and SQL analytics endpoint surfaces.
Persist a DataFrame as a managed Delta table and query it from Spark and SQL.
Persist a DataFrame as a managed Delta table and query it from Spark and SQL.
How to design the Gold layer that reports, semantic models and APIs consume: modelling choices, stable contracts, serving engines and how to change it without breaking consumers.
The curated layer is the contract with consumers. Its design determines how easy it is to change the platform underneath.
A critical look at the Medallion pattern: when two layers or a single curated model are enough, and the cost that each unnecessary layer adds.
The Medallion pattern is often applied by default, producing extra copies of data without a clear purpose. This article looks at concrete cases where fewer layers are the better design, and at the signals that justify adding one.
How to design a Bronze layer that preserves source fidelity: landing formats, ingestion metadata, schema drift, retention and replay.
Bronze is only useful if it can answer what a source actually sent and let a batch be replayed. That depends on decisions made at ingestion time that are hard to add later.
How to turn source-shaped Bronze data into clean, validated, conformed Silver entities that several consumers can share.
Silver is where most data quality work happens and where business keys are established. A weak Silver layer pushes cleanup into every Gold model.
How changes propagate from Bronze to Silver to Gold without full reloads: change detection between layers, affected keys and recomputing only impacted aggregates.
Loading new data from a source is only the first step. Each later layer also needs to process only what changed, or the platform still pays for a full rebuild downstream.
Which write pattern belongs in which layer: append in Bronze, MERGE for Silver entities and history, and targeted overwrite or MERGE for Gold.
MERGE is powerful but expensive, and it is often used in layers where an append or a partition overwrite would be simpler and cheaper.
Why the right partitioning often differs by layer, from load-date partitions in Bronze to unpartitioned Gold tables, and how to decide.
Copying one partitioning scheme into every layer is a common source of small files and missed pruning. Each layer is written and read differently.
Where small files come from in each layer and how to plan OPTIMIZE, V-Order and VACUUM per layer instead of applying one schedule everywhere.
Frequent Bronze appends, Silver MERGE operations and Gold rebuilds create small files for different reasons, so the maintenance plan should differ too.
A complete, production-minded Medallion pipeline in Fabric: incremental ingestion, quality gates, MERGE, maintenance, logging and deployment. It builds on the first end-to-end pipeline tutorial.
The first end-to-end pipeline tutorial shows the flow; this capstone adds what a production pipeline needs, bringing the rest of the series together in one working design.
The criteria I use to choose between a Fabric Lakehouse and a Fabric Warehouse: team skills, write patterns, T-SQL needs and governance.
Both store Delta tables in OneLake, which makes the choice confusing. The difference is mostly in how data is written and who works with it.
Move CSV or Parquet through a notebook-driven Bronze, Silver and Gold flow into a queryable serving layer.
Move CSV or Parquet through a notebook-driven Bronze, Silver and Gold flow into a queryable serving layer.
Pass dates and business identifiers from a pipeline into one reusable Fabric notebook.
Pass dates and business identifiers from a pipeline into one reusable Fabric notebook.
A practical deployment approach for Fabric items using Git integration, deployment pipelines and automation, including what still needs manual steps.
Fabric deployment options are changing quickly and each has gaps. The article describes a strategy that works today and where it needs workarounds.
How to promote Fabric items between environments: configuration per environment, data separation, approvals and rollback.
Promoting changes through environments is where most deployment problems appear: connection strings, IDs and data that differ per environment.
Use Fabric Git integration deliberately across branches, workspace artifacts and team workflows.
Use Fabric Git integration deliberately across branches, workspace artifacts and team workflows.
Separate deployable workspace artifacts from data, connections and environment-specific configuration.
Separate deployable workspace artifacts from data, connections and environment-specific configuration.
Patterns for loading only new and changed data into Fabric, with watermarks, change tracking and Delta MERGE, and how to handle late or corrected data.
Full reloads stop being viable as data grows. Incremental loads are more efficient but bring their own correctness problems.
Design an execution log that makes notebook and pipeline runs traceable in production.
Design an execution log that makes notebook and pipeline runs traceable in production.
Use a repeatable workflow to diagnose Spark, schema, parameter, dependency and pipeline failures.
Use a repeatable workflow to diagnose Spark, schema, parameter, dependency and pipeline failures.
A step-by-step workflow for slow Fabric workloads: deciding whether the problem is capacity, Spark, SQL, storage layout or the query itself.
Fabric has several engines and a shared capacity, so 'it is slow' can have many causes. A workflow narrows it down before anything is changed.
How Delta MERGE works internally and the patterns that keep it fast: narrowing the target, partition and file pruning, and source deduplication.
MERGE is the main upsert tool for Delta tables and also one of the most common causes of slow jobs. Most of the cost is in finding and rewriting files.
Criteria for choosing a partition column: cardinality, query filters, write patterns and data volume per partition.
A good partition column helps reads, writes and maintenance at the same time. The article turns that into concrete criteria.
Where small files come from in Delta tables, how they slow down reads, writes and the transaction log, and how to measure the problem.
Small files accumulate quietly through frequent writes and over-partitioning. By the time queries are slow, there may be millions of them.
How file size affects read parallelism, metadata overhead and write cost, and how to pick a target size for your engines.
Files that are too small add overhead and files that are too large limit parallelism. The right size depends on the engines reading them.
Build a practical baseline for file layout, Spark execution and Delta writes at multi-million-row scale.
Build a practical baseline for file layout, Spark execution and Delta writes at multi-million-row scale.
Plan storage, partitioning, incremental processing and capacity for billion-row Fabric workloads.
Plan storage, partitioning, incremental processing and capacity for billion-row Fabric workloads.
Choose between shared and isolated Fabric notebook sessions based on compatibility, concurrency, state, reliability and cost.
Session reuse can reduce acquisition overhead and improve utilization, while isolation can protect incompatible or resource-heavy workloads.
Diagnose repeated actions, driver collection, shuffle-heavy logic, skew, inappropriate caching and output anti-patterns in Fabric notebooks.
Convenient interactive patterns can trigger repeated jobs, driver pressure, shuffles, skew and small-file output when moved unchanged into production.
Reduce Fabric notebook I/O through projection, early filtering, incremental boundaries, partition pruning and Delta data skipping.
The cheapest byte to process is one the notebook never reads, but source syntax must be confirmed against the executed plan and table design.
Design reliable Fabric notebook Delta writes around mode, partition counts, file size, compaction, V-Order and rerun behavior.
Write performance and future read performance are connected through distribution, file layout, table properties and idempotent pipeline design.
Separate session acquisition, Spark execution, data skew, shuffle, I/O and writes when a Fabric notebook runs slowly.
Notebook elapsed time spans multiple layers, so changing code or capacity without locating the slow layer often moves the symptom rather than fixing it.
A walk through what SQL Server does between receiving a query and returning rows: parsing, binding, optimisation, plan caching and execution.
Most performance problems are easier to reason about once you know which stage of query processing they come from. This article builds that mental model before the rest of the series goes deeper into plans, statistics and indexing.
A systematic way to read execution plans: estimated vs actual rows, the operators that matter, warnings, and how to find the real cost instead of the highest percentage.
Execution plans are often read top to bottom looking for the biggest percentage, which regularly points at the wrong operator. The article proposes a repeatable reading order based on row estimates and data flow.
Why SQL Server's missing index suggestions are a starting point and not an instruction, and how to evaluate them against the whole workload.
Missing index hints are generated per query, ignore existing indexes and know nothing about write cost. Applying them blindly leads to overlapping indexes and slower writes.
How statistics drive cardinality estimates, why they go stale on large tables, and how sampling and update thresholds affect plan quality.
A correct index is not enough if the optimiser misjudges how many rows it will read. Statistics are usually where that misjudgement starts.
How to confirm that a regression really is parameter sensitivity before reaching for OPTION(RECOMPILE) or plan forcing, and how to choose between the available fixes.
Parameter sniffing gets blamed for many slow queries and is often fixed with a blanket recompile hint. The article focuses on proving it first and then picking the least invasive fix.
Using Query Store as the first stop when production performance changes: configuration, finding regressions, comparing plans and forcing plans safely.
When someone reports that the system got slower, the first question is what changed. Query Store keeps the history needed to answer it, provided it is configured sensibly.
Find active requests above a configurable duration and interpret elapsed time alongside CPU, reads, writes, status and waits.
A long duration can describe legitimate ETL, reporting, index maintenance, a large update, or a request that is merely blocked. The useful diagnosis compares elapsed time with work and waits instead of declaring every old request unhealthy.
Rank current and cached SQL Server workload by total CPU and average CPU per execution, with the right cache-lifetime caveats.
Total CPU identifies cumulative consumers, while average CPU highlights individually expensive executions. Both are needed, and neither is durable history when its evidence comes from the plan cache.
Rank cached SQL Server queries by total and average logical reads and connect the result to plans, scans, estimates and indexing.
Logical reads measure pages processed and often expose expensive access patterns more reliably than duration, which changes with blocking and server load. High reads still need workload and plan context.
Inspect index definitions, size, row count, seeks, scans, lookups, updates and observation-window limitations.
Indexes trade read access paths for storage and write maintenance. Zero seeks in a DMV observation window does not prove that an index is safe to remove.
Read SQL Server missing-index DMV recommendations and evaluate benefit, overlap, width and write cost before changing production.
Missing-index DMVs produce optimization hints, not complete index designs. Recommendations can overlap, become extremely wide, disappear after restart, and omit the cost imposed on writes.
Inspect statistics metadata, update dates, samples and modification counters, then use histograms only when the evidence calls for them.
Statistics influence cardinality estimates and plan choices, but age alone does not prove they are harmful. The investigation should connect sampling and modifications to estimate errors in a specific workload.
Combine cached procedure execution statistics, Query Store and application evidence before deciding that a stored procedure is unused.
Zero executions in `sys.dm_exec_procedure_stats` only means the procedure is absent from the current cache evidence. It does not prove that a monthly job, rare recovery path, or external integration never calls it.
Compare Query Store time windows, resource consumers and plan changes to investigate SQL Server regressions safely.
Query Store connects present symptoms to historical runtime statistics and plans. Comparing equivalent time windows can reveal regression after a deployment, upgrade, or compatibility-level change.
Group active SQL Server work by login, host, application and database to explain API, report, batch and integration spikes.
An instance-wide spike becomes actionable when it can be attributed to a workload owner. Login, host and program values are useful signals, though client-supplied identity must be interpreted carefully.
Investigate hundred-million and billion-row tables through workload, storage, distribution, maintenance and retention evidence.
Row count alone does not select a physical design. Very large tables demand a workload-led review of access, writes, retention, distribution and operational constraints.
Diagnose a large SQL Server update through log use, locking, row count, predicates, plans, statistics, indexes and rollback risk.
A large update can be correct yet operationally disruptive through transaction-log growth, blocking, index maintenance and rollback exposure. Batching changes those tradeoffs but is not automatically safer or faster.
Design an evidence-based SQL Server comparison for upgrades, compatibility levels, indexes, deployments and infrastructure changes.
Averages can improve while the slowest user requests regress. A useful comparison controls workload and time windows and reports distribution, work and frequency together.
Use SQL Server request and aggregate waits as evidence while avoiding simplistic wait-to-fix mappings.
A wait type identifies where execution currently cannot proceed; it does not independently explain why. The query, plan, resource, workload and time window supply the diagnosis.
Find long-running and sleeping-session transactions, their owners, database and potential blocking and transaction-log impact.
An old transaction can retain locks and delay log truncation even when its session is sleeping. Joining transaction, session and request state makes that hidden ownership visible.
A concise companion index for the SQL Server diagnostic queries in this series and the performance-tuning checklist resource.
During an incident, readers need a trustworthy route to the right diagnostic rather than one enormous script with no interpretation. This reference will index the focused articles and resource checklist without duplicating their reasoning.
Index design when tables are too large to rebuild casually: clustered key choice, covering indexes, columnstore, maintenance windows and the write cost of every index.
On very large tables an index decision is expensive to reverse: building, rebuilding or dropping it takes hours and affects every write. The article covers how to choose indexes deliberately at that scale.
Patterns for changing or deleting millions of rows without long blocking, transaction log growth or lock escalation.
A single statement that touches millions of rows can fill the transaction log, escalate locks and block the application. Batching fixes that, but only if the batches are designed carefully.
Options for copying or migrating very large tables, from batched inserts to partition switching and bulk load, and how to keep the source system usable meanwhile.
Large data moves compete with the production workload for I/O, log and locks. The approach needs to be chosen based on downtime tolerance and how much the source can absorb.
What table partitioning really gives you, mainly data management, and why it is not an automatic query performance feature.
Partitioning is often introduced to make queries faster and ends up making some of them slower. Its real strengths are loading, archiving and maintenance.
How to choose the partition column and boundaries for very large tables, and how that choice interacts with the clustered index and query patterns.
The partition key is hard to change later and affects loading, querying and maintenance. Choosing it means balancing all three.
Using staging tables and partition switching to load or replace large slices of data as a near-instant metadata operation.
Partition switching turns a long, blocking load into a metadata operation, but it has strict requirements that are easy to miss.
A comparison of MERGE with separate INSERT and UPDATE statements: readability, concurrency behaviour, known issues and the pattern I prefer in production.
MERGE looks like the natural upsert statement, but its concurrency behaviour and history of issues lead many teams to avoid it. The article explains the trade-offs and states a preference with reasons.
Upsert patterns that behave correctly under concurrency, for single rows and for set-based batches, without relying on MERGE.
Upserts that work in testing can produce duplicates or primary key violations under concurrent load. The article shows patterns that hold up.
What limits insert throughput, from logging and indexes to page splits and client batching, and how to remove each bottleneck.
Slow inserts are rarely caused by the INSERT statement itself. The bottleneck is usually in logging, indexes, constraints or the client.
Architecture decisions for a Fabric platform that has to handle large volumes: storage layout, compute choices, workspace boundaries and the limits to plan for.
Fabric makes it easy to start but the early layout decisions matter once volumes grow. The article lays out the decisions worth making deliberately up front.
Isolation options for serving many tenants from Fabric, from shared tables with row-level security to workspace per tenant, and their operational cost.
Tenant isolation affects security, cost, deployment and performance at the same time. The right model depends on how many tenants there are and how different they are.
How Fabric capacity units are consumed by different workloads, how smoothing and throttling work, and how to attribute usage to items and jobs.
Capacity is shared by every workload in it, so one heavy job can affect everything else. Understanding how usage is measured is the basis for sizing and troubleshooting.
A workspace layout for separating environments in Fabric, with naming, permissions and how items reference each other across workspaces.
Environment separation in Fabric starts with workspace design. Getting it wrong makes deployments and permissions painful later.
Patterns for organising tables, files, schemas and shortcuts in a Fabric Lakehouse that has to scale to many sources and teams.
A Lakehouse with a few tables is easy. One with hundreds needs conventions for layout, ownership and naming.
When to use Data Factory pipelines, when to use notebooks, and how to combine them so orchestration and transformation stay clearly separated.
Pipelines and notebooks overlap enough that teams end up mixing logic between them. Clear roles make platforms easier to debug and deploy.
How to design pipelines that can be rerun safely after a failure without duplicating or losing data.
Pipelines fail, and someone will rerun them. If a rerun creates duplicates or skips data, every failure becomes a data quality incident.
How to absorb source schema changes in Fabric without breaking pipelines or consumers: detection, evolution options and contracts.
Source systems change columns without warning. A platform needs to decide which changes it absorbs automatically and which should stop the pipeline.
Ingestion design for many small files a day: discovery, batching, parallelism, tracking what was processed and avoiding small-file problems downstream.
Processing files one at a time does not scale, and processing them all at once is hard to recover. The design has to handle both throughput and traceability.
Delta Lake concepts mapped to what SQL Server engineers already know: transactions, the log, updates, deletes and what has no equivalent.
Engineers coming from SQL Server bring useful intuition and some wrong assumptions. Mapping concepts side by side speeds up the transition.
A look inside a Delta table folder: Parquet data files, the _delta_log, checkpoints and how a read reconstructs the current version.
Most Delta performance issues make sense once you look at the files on disk. This article opens the folder and explains each part.
A diagnostic approach for slow MERGE jobs: reading the operation metrics, finding the expensive phase and matching it to a fix.
A MERGE that took minutes last month can take hours today without any code change. The table history and job metrics usually explain why.
When appends are enough and when MERGE is required, and how append-then-deduplicate patterns compare on cost and correctness.
MERGE is often used where a plain append would do, at a much higher cost. Choosing correctly depends on the data's change pattern.
Design choices that matter once a Delta table holds hundreds of millions of rows: partitioning, clustering, file size and write patterns.
At this size the table is still manageable, but early design choices start to show. It is the right time to set conventions before the table grows further.
Practical considerations for partitioning, file size, ingestion, MERGE operations and query performance when Delta tables reach very large scale.
At billions of rows, mistakes in table design turn into long-running jobs and expensive maintenance. The article covers the decisions that matter most at that scale.
Common Delta partitioning mistakes, such as high-cardinality keys and tiny partitions, what they cause, and how to recover from them.
Over-partitioning is one of the most common causes of slow Delta tables. It is also expensive to undo once data is written.
Compaction options for Delta tables, including OPTIMIZE, optimised writes, auto compaction and V-Order, and how to schedule them.
Compaction fixes small files but costs compute and can conflict with writers. The strategy should match how the table is written and read.
How Delta's optimistic concurrency control works, which operations conflict, and how to design jobs that write to the same table safely.
Two jobs writing to the same table can both succeed, or one can fail with a conflict. Knowing which is which avoids surprises in production.
Why ConcurrentAppendException happens, how to read it, and the design and retry patterns that prevent it.
Concurrent append conflicts often appear only under production load. The fix is usually in how writers' predicates overlap.
Using mergeSchema, column mapping and explicit migrations to evolve Delta tables without breaking writers or readers.
Automatic schema evolution is convenient and also risky: an unexpected column can spread silently. The article covers when to allow it and when to require a migration.
Planning routine maintenance for Delta tables: what OPTIMIZE and VACUUM do, retention and time travel trade-offs, and how often to run them.
Without maintenance, Delta tables slow down and storage grows. With careless maintenance, time travel and concurrent readers can break.
The Parquet format explained in database terms: columnar layout, row groups, pages, encodings, compression and metadata.
Parquet is the storage format under Delta Lake and Fabric. Understanding it explains much of their performance behaviour.
The specific mechanisms that make Parquet fast for analytical queries: column pruning, predicate pushdown, encoding and compression.
Parquet is described as fast, but the speed comes from specific mechanisms, and each one depends on how the file was written.
Why data lakes accumulate small files, what that costs in listing, metadata and query time, and the ingestion patterns that avoid it.
Small files are a general data lake problem, not only a Delta one. Most of them are created by ingestion patterns that can be changed.
Hive-style partitioning of Parquet datasets: how directory partitions help pruning, and the point at which they create more files than benefit.
Directory partitioning is easy to add and easy to overdo. The cost shows up as many small files and slow listing.
Partitioning and sorting strategies for very large Parquet datasets, combining directory partitions with in-file ordering for effective pruning.
At very large scale, directory partitioning alone is too coarse. Sorting data within files is what makes statistics-based pruning effective.
What row groups are, how their size and statistics affect pruning and memory use, and how writers decide row group boundaries.
Row groups are the unit of pruning and parallelism inside a Parquet file. Their size is often left at defaults that do not suit the workload.
What happens when Parquet files with different schemas are read together, how engines reconcile them, and the changes that are safe.
Parquet files carry their own schema, so a dataset can contain several versions. Reading them together works only within certain limits.
Techniques for reading very large Parquet datasets: column selection, predicate pushdown, partition pruning, batching and choosing an engine.
Reading billions of rows is mostly about not reading most of them. The article covers how to make sure the engine skips what it can.
What the Lakehouse SQL analytics endpoint is, how it differs from a SQL Server database, and which familiar features behave differently.
The SQL endpoint looks like a SQL Server database but works very differently underneath. Knowing the differences avoids wrong expectations.
A side-by-side comparison of the Lakehouse SQL endpoint and the Fabric Warehouse for querying and serving data.
Both expose T-SQL over Delta tables, but they differ in writes, transactions and table management. The choice affects how the platform is built.
The usual reasons SQL endpoint queries are slower than expected: table layout, file health, statistics, data types and query patterns.
Teams moving from SQL Server often expect similar performance from the same query. The reasons for differences are usually in the underlying Delta tables.
A workflow for diagnosing slow SQL endpoint queries using query insights, plans and table health checks.
Once a slow query is reported, there needs to be a repeatable way to find out whether the problem is the query, the table or the capacity.
Design considerations for serving data through an API backed by a Fabric SQL endpoint: authentication, query shape, caching, latency and limits.
Serving an API from an analytical endpoint is possible but has different latency and concurrency characteristics from an OLTP database.
How to identify the queries with the biggest real impact, by total resource use and frequency, instead of chasing the single slowest query.
The slowest query is rarely the most expensive one overall. A fast query that runs a million times a day can matter far more.
A practical workflow for finding the head blocker, understanding what it is waiting on, and deciding whether to wait, kill or fix.
Blocking incidents are stressful because the pressure is to kill something quickly. A clear workflow makes it possible to act fast without losing the evidence needed to prevent the next one.
How to capture and read deadlock graphs, identify the access pattern that causes the cycle, and choose a fix that removes it.
Deadlocks are usually handled with retries, which hides the cause. Reading the deadlock graph properly usually points to a specific access order or missing index.
A workflow for finding which item and operation caused a capacity spike, and for preventing the next one.
When capacity is throttled, every user notices. Finding the cause quickly depends on knowing where to look and what the metrics mean.
The signals worth tracking on a production Fabric platform: pipeline outcomes, data freshness, capacity, table health and cost.
Without deliberate monitoring, problems are found by report users. The article lists what to watch and how to alert on it without noise.
Designing a run log for data pipelines: what to record per run and step, where to store it, and how to make it useful for debugging and reporting.
Built-in run history is often not enough to answer what happened to a specific batch. A deliberate execution log makes that question easy.
A step-by-step approach to failed pipeline runs: triage, finding the failing step, reproducing safely, fixing and rerunning without side effects.
Pipeline failures need a calm, repeatable process: understand the impact, find the cause, fix it and rerun safely.
A planned practical guide to finding which client is generating the most load in production data platforms.
A planned practical guide to monitoring query duration, cpu, reads and writes in production data platforms.
A planned practical guide to using p50, p95 and p99 for data platform performance in production data platforms.
A planned practical guide to designing correlation ids across apis, reports and data pipelines in production data platforms.
A planned practical guide to detecting abnormal request spikes in production data platforms.
A planned practical guide to monitoring cache hit rate in a report engine in production data platforms.
A planned practical guide to building operational dashboards for data platforms in production data platforms.
A planned practical guide to alerting without creating alert fatigue in production data platforms.
A planned practical guide to tracking pipeline failures, retries and recovery in production data platforms.
A planned practical guide to monitoring data freshness and sla compliance in production data platforms.
A planned practical guide to monitoring microsoft fabric capacity and workload pressure in production data platforms.
A planned practical guide to monitoring sql blocking and long-running transactions in production data platforms.
A planned practical guide to designing a multi-tenant monitoring model in production data platforms.
A planned practical guide to building an audit trail for data requests in production data platforms.
A planned practical guide from operational logs to root cause through a production troubleshooting workflow.
A planned production guide to querying fabric from a report engine for secure analytical data access.
A planned production guide to returning csv, json and other report formats for secure analytical data access.
A planned production guide to caching report results in a lakehouse for secure analytical data access.
A planned production guide to designing report cache keys for secure analytical data access.
A planned production guide to when should a report be cached? for secure analytical data access.
A planned production guide to building asynchronous reports for large requests for secure analytical data access.
A planned production guide to report engine security and tenant isolation for secure analytical data access.
A planned production guide to rate limiting and throttling report requests for secure analytical data access.
A planned production guide to designing report request parameters for secure analytical data access.
A planned production guide to validating report requests before querying data for secure analytical data access.
A planned production guide to ui data access vs report downloads for secure analytical data access.
A planned production guide to pagination for interactive reports for secure analytical data access.
A planned production guide to streaming large report results for secure analytical data access.
A planned production guide to monitoring report engine performance for secure analytical data access.
A planned production guide to designing custom reports safely for secure analytical data access.
A planned production guide to report engine metadata and configuration for secure analytical data access.
A planned production guide to versioning report definitions for secure analytical data access.
A planned production guide to auditing report access for secure analytical data access.
A planned production guide to handling duplicate report requests for secure analytical data access.
A planned production guide to retry and failure patterns for secure analytical data access.
A planned production guide to building a report engine with .net or python for secure analytical data access.
A planned production guide to app service vs azure functions for a report engine for secure analytical data access.
A planned production guide to api management in front of a report engine for secure analytical data access.
A planned production guide to using fabric warehouse and lakehouse together for secure analytical data access.
A planned production guide to multi-tenant report engine architecture for secure analytical data access.
A planned production guide to end-to-end report engine reference architecture for secure analytical data access.
Architecture and implementation notes for a report engine built with Python and Azure Functions: triggers, data access, rendering, scaling and limits.
Report generation is a common job that does not need a full application server. Azure Functions fits well if its limits are respected.
Select, rename and organize DataFrame columns as an explicit transformation contract rather than carrying an uncontrolled schema forward.
Column selection defines the output contract, reduces unnecessary memory, and prevents accidental propagation of sensitive or unstable fields.
Sort pandas data predictably by one or more columns while handling ties, missing values and stable ordering.
Sorting affects ranking, deduplication, window-style logic and reproducible output, so tie and null behavior must be intentional.
Detect and handle missing pandas values according to business meaning instead of applying one blanket fill or drop rule.
Missing can mean unknown, not applicable, anonymous, delayed, or invalid. Each meaning requires different validation and transformation behavior.
Create typed, testable pandas columns for revenue, margins, flags and date features without fragile chained assignment.
Calculated columns encode business rules. Clear names, typed inputs, null decisions and reconciliation make those rules reviewable.
Parse, validate and transform dates and timestamps in pandas with explicit timezone and invalid-value behavior.
Date logic fails quietly when strings, timezones, boundaries and invalid values are left implicit.
Distinguish exact duplicates from duplicate business keys and choose deterministic pandas survivorship rules.
Removing duplicates without defining grain and survivorship can discard valid events or keep an arbitrary conflicting record.
Reshape pandas data between long and wide forms while controlling aggregation, missing combinations and duplicate keys.
Reshaping changes the grain and often aggregates implicitly, so duplicate key combinations and missing cells need explicit treatment.
Reduce pandas CSV memory and processing cost through column pruning, explicit dtypes, chunking and better storage formats.
CSV parsing is CPU- and memory-intensive, and a file that fits on disk can expand substantially in a DataFrame.
Write pandas outputs with deliberate schemas, indexes, partitions and validation for downstream consumers.
Output format decisions affect types, file size, interoperability and whether a downstream reader can reproduce the intended schema.
Compare pandas and PySpark DataFrames through execution model, scale, APIs, null semantics and practical migration choices.
Similar-looking APIs hide very different local and distributed execution models. Dataset size alone does not decide when Spark is justified.
Diagnose pandas schema drift, dtype surprises, nulls, duplicate keys, bad joins, memory pressure and misleading aggregates.
Most DataFrame failures become easier when the pipeline is reduced to grain, schema, counts, key quality and one reproducible transformation.
Using infrastructure as code tools and cloud SDKs together to provision and manage data platform resources repeatably.
Data platforms have resources that IaC tools cover well and others that need SDK calls. A clear split keeps automation maintainable.