Azure Data Factory, the ADF UX and Git

If you're using Azure Data Factory (ADF), you're probably using Git and almost certainly using the ADF User Experience (ADF UX) – ADF's online integrated development environment (IDE). These three components are so closely interlinked that sometimes it's hard to think about them separately – in this article I try to do exactly that.

A data factory – an instance of ADF – is an Azure resource. You can find it in the Azure portal, without even opening the ADF UX.

I find it useful to think of it in two parts:

• a set of resource definitions (pipelines, datasets, linked services etc) that have been published to the factory
• something I'm describing loosely as factory machinery – this interprets those published resource definitions and executes pipelines. This description is neither precise nor offical 😀.

The factory machinery uses metadata stored in linked service and integration runtime definitions to interact with other services, elsewhere in Azure and beyond.

I described the ADF UX as an IDE, but it combines two different roles:

• an online integrated development environment, used to develop and debug data factory resources (sometimes referred to as “authoring”)
• a management tool, used to monitor the behaviour of published factory resources, verify pipeline success, diagnose faults etc.

An ADF UX session is always connected to a data factory instance. The dashed horizontal arrows between the UX and the connected instance in the diagram indicate monitoring tasks; the others indicate authoring tasks.

Using Git allows you to save your factory resource definitions without having to publish them, and even better, now they're under version control 😅. To use Git, a factory instance must be linked to a Git repository in Azure Repos or GitHub.

Now when you open the ADF UX, the factory resources loaded into your session are the versions stored in Git, instead of the resources published in the factory. When connected to a Git-enabled factory, the UX gains a “Save” button – clicking “Save” stores changes made in the UX to your Git server, without publishing them to the factory.

You can still publish resources to the factory by clicking “Publish”, but you can only do this from a nominated collaboration branch in Git (usually main or equivalent). Publishing also updates a special publish branch in your repository (usually adf_publish), writing factory resources into it in the form of ARM templates. (For the purposes of this post I'm side-stepping the whole area of feature branch workflow, but this is highly recommended).

Now you have at least two (and potentially lots of) different collections of factory resources to consider:

• the set of published resources
• the set of resource definitions stored in your Git collaboration branch
• another set in every active Git feature branch.

You can switch between these views using the dropdown list(s) in the top left.

A common development workflow is to use multiple data factories to provide separate environments for production, testing and development.

In the development ADF instance, the ADF UX now acts more-or-less exclusively as an authoring tool:

• you load working definitions from Git
• you make changes on the pipeline authoring canvas
• you debug them using the dev instance's factory machinery
• you save your changes back to Git.

Even if you use the “Publish” option to save ARM templates to Git, you probably don't care about the resources published to the factory any more. You won't run them there¹⁾ so there's nothing to monitor, and you don't need to be able to inspect them. I've greyed them out in the diagram above.

The resources you build in the development environment and save to Git are deployed to other factory instances for testing or production use. You can either:

• deploy ARM templates, written to the Git repo's adf_publish branch by publishing from dev, or
• deploy JSON resource definitions, written to the Git repo when saving from dev. These can be deployed using tools like PowerShell or a DevOps pipeline task.

Either way, you're deploying definitions stored in Git, using some external tooling to do so.

The non-development ADF instances are not Git-enabled, by design – the whole point about the Git repo attached to your development environment is that it's the single authoritative repository of ADF resource definitions. This means that when you connect to one of the non-development instances using the ADF UX, you automatically see published pipeline definitions.

In the same way that the ADF UX is purely an IDE in development, in other factory instances it's now purely a management tool to monitor behaviour and performance. Because those factories aren't Git-enabled, you can edit and publish updated pipeline definitions directly, but you should avoid doing so to make sure that your Git repository remains the authoritative source.²⁾

The ADF UX is a combined IDE and management tool which is always connected to an ADF instance. The combination of roles can be confusing, but in a multi-instance development workflow using Git they naturally become separated, depending on the type of instance you're connected to:

• when connected to a development ADF instance, the ADF UX is pure IDE, providing a graphical authoring experience, debugging tools and integrated version control
• when connected to other instances – production, test etc – the ADF UX is a service management tool, supporting activities for outcome verification, fault diagnosis and performance monitoring.