Migrate ADF to Databricks.

Why ADF to Databricks is a rebuild, not a lift

Azure Data Factory and Databricks solve overlapping problems with completely different machinery, and that mismatch is exactly why a naive copy of a factory onto the lakehouse fails. ADF is an orchestration-and-movement service: its unit of work is a pipeline of activities, its transforms run as Mapping Data Flows on a managed Spark cluster you never touch, and its compute lives in Integration Runtimes. Databricks is a lakehouse: Delta tables on object storage, governed by Unity Catalog, with Apache Spark as the engine you do control, orchestrated by Lakeflow Jobs and built either as Lakeflow Spark Declarative Pipelines or as your own framework. There is no button that turns one into the other. What moves cleanly is the intent of each pipeline; what has to be rebuilt is almost everything underneath it. The agent fleet treats the migration as exactly that: read the logic, re-express it on the target, and prove the result against your data before anyone signs off. Autonomous in the loop, accountable at the gates.

What breaks in a naive port

The first thing that breaks is the Copy activity. In ADF, “copy data” is a single declarative step that hides connector logic, type handling and incremental bookkeeping. On Databricks there is no monolithic copy, there is a deliberate ingest choice. A file landing zone becomes Auto Loader, whose cloudFiles source incrementally discovers new files and records their metadata in a checkpoint for exactly-once processing; a high-throughput SQL load becomes COPY INTO; a managed application or database source becomes a Lakeflow Connect connector. Pick the wrong one and you either reprocess files or miss them. Worse, ADF’s Copy is forgiving about schema, while Auto Loader infers types, so a column the legacy data flow quietly cast to decimal(18,2) can land as a string and silently corrupt every downstream aggregate. This is the single most common parity failure the fleet catches on this pair, which is why the iteration log above opens with it.

The second thing that breaks is the Mapping Data Flow. It looks portable because it already runs on Spark, but its visual components (Derived Column, Conditional/Alter Row, Window, Aggregate, and the Slowly Changing Dimension pattern) are an ADF abstraction, not Spark code you can lift. The fleet re-expresses the data flow’s semantics set-based. An Alter Row upsert becomes a Delta MERGE INTO; an aggregate becomes a GROUP BY; a Type 2 SCD becomes either AUTO CDC (create_auto_cdc_flow) inside a Lakeflow Spark Declarative Pipeline or a dbt snapshot with the timestamp strategy, which maintains dbt_valid_from, dbt_valid_to and dbt_scd_id over a mutable source. dbt’s merge incremental strategy (the Delta default) covers the SCD Type 1 “latest value wins” case; snapshots cover the Type 2 “keep the history” case. Choosing the wrong one loses history, and the loop’s row-count parity test is what surfaces it.

Rebuilding the control flow and the triggers

ADF’s dependsOn graph is the third thing that does not survive a copy. Each dependency carries one or more conditions (Succeeded, Failed, Skipped, Completed) and those edges encode the real error-handling logic of the estate, not just the happy path. The fleet rebuilds that graph two ways depending on flavor. In the your-framework flavor it becomes Lakeflow Jobs tasks with dependencies and conditional control flow, where a Failed edge turns into a run-if rule on a recovery task. In the dbt flavor the data dependencies become the dbt DAG via ref(), and the conditional and error branches move up into the orchestrator, because dbt is declarative SQL and runtime branching belongs in Lakeflow Jobs (or a Python model), never inside a model. Metadata-driven ForEach loops over a control table become a Lakeflow Jobs for_each task or a PySpark loop; the fleet resolves the parameter chain to describe what runs but never invents control-table values, flagging them for the design gate instead.

Triggers are the fourth. Schedule and storage-event triggers map onto Lakeflow Jobs scheduled and file-arrival triggers with little friction. The tumbling-window trigger is the awkward one: it runs fixed-size, contiguous, non-overlapping intervals, retains state, and supports backfill through windowStartTime / windowEndTime. The fleet reproduces that by parameterising the job over a window boundary and expressing backfill as bounded runs, or, in dbt, as a microbatch incremental keyed off an event-time column. The cadence and replay semantics are preserved even though the mechanism is rebuilt.

Connectivity is the dependency that does not move

Finally, the Integration Runtime. The Azure IR is just managed cloud compute and disappears into Databricks Spark without ceremony. The self-hosted IR is the one that matters: any Copy or data flow reaching an on-premises or network-restricted store through a SHIR is bound to that network path, its custom drivers and its firewall rules, and none of that migrates on its own. The fleet logs every SHIR-bound activity as an explicit risk item and re-plans the path (typically a Lakeflow Connect connector or a connector with the right network access landing the data as Delta) so modeling can start from a known bronze table. The Azure-SSIS IR is logged separately: native SSIS execution on Databricks is not a generally available lift-and-shift, so any lifted packages are rebuild candidates, never a runtime we claim on the target.

How the fleet proves it

Every claim above is settled the same way: the build-test-run loop. The builders generate the ingest, the transforms and the orchestration; the test agent runs schema checks, the dbt tests (unique, not_null, relationships) and row-count parity queries against the ADF baseline; and any failure goes back to the builders, not onto a list. The loop iterates until green (type drift pinned, SCD2 windows non-overlapping, parity delta zero) and only then does the work reach the promotion gate. If you are weighing the other destination, Migrate ADF to Microsoft Fabric runs the same fleet against Microsoft Fabric instead.