Description
Course Outline*
*This is a new course and the outline is subject to change.
Module 1: Introduction to Team Development in ADF (30 min)
- Understanding ADF as a collaborative tool
- Best practices for multi-developer environments
- Git integration in ADF
- Branching strategies (Git Flow, feature branching, hotfixes, etc.)
- 🛠 Hands-on: Setting up ADF source control in Azure DevOps or GitHub
Module 2: ADF Version Control and CI/CD Pipelines (1 hour)
- Differences between ADF Live Mode vs. Git Mode
- Managing deployments using ARM templates
- Setting up a CI/CD pipeline in Azure DevOps
- YAML Pipeline for ADF
- 🛠 Hands-on: Building and deploying an end-to-end CI/CD pipeline for ADF
Module 3: Advanced Triggers in ADF (45 min)
- Understanding ADF triggers: Scheduled, Tumbling Window, and Event-Based Triggers
- Tumbling Window Triggers: Concepts, Use Cases, and Late Arrivals
- Event-Based Triggers: Monitoring file arrival in ADLS
- 🛠 Hands-on: Setting up a Tumbling Window Trigger
- 🛠 Hands-on: Setting up a Event-Based Trigger
Module 4: Notebook Execution in ADF (Fabric, Databricks, and Synapse) (45 min)
- When to use Fabric, Databricks, or Synapse Notebooks
- Executing Databricks Notebooks from ADF (Managed vs. External)
- Notebook execution in Microsoft Fabric Pipelines
- Connecting to Synapse Spark Notebooks
- Passing parameters to notebooks from ADF
- 🛠 Hands-on: Triggering a Databricks/Fabric Notebook from ADF
Module 5: Debugging, ADF Expressions, and Monitoring (45 min)
- Common ADF errors and troubleshooting techniques
- ADF Expressions & Functions:
- String functions (
concat,substring,replace) - Date functions (
formatDateTime,addDays,utcNow) - Collection functions (
filter,union,intersection)
- String functions (
- Debugging pipeline failures and JSON configuration issues
- Using Azure Monitor, Log Analytics, and Application Insights
- 🛠 Hands-on: Writing advanced ADF expressions and debugging a failing pipeline
Module 6: Incremental Load Patterns in ADF (45 min)
- Incremental vs. Full Load in ADF
- Using Watermark Columns for Incremental Loads
- Change Data Capture (CDC) in ADF
- Implementing Delta Lake Merge in Fabric/Synapse
- 🛠 Hands-on: Building an Incremental Load Pipeline with CDC
Module 7: Advanced Orchestration Patterns in ADF (45 min)
- Metadata-driven pipeline design
- Pipeline chaining and dependency management
- Looping & dynamic pipeline execution (
ForEach,Untilloops) - Dynamic parameterization for reusability
- Hybrid Orchestration (Combining ADF with Logic Apps or Functions)
- 🛠 Hands-on: Implementing a metadata-driven dynamic pipeline
- 🛠 Hands-on: Start / stop an SSIS run-time via azure automation
Q&A and Wrap-Up (30 min)
- Review key takeaways
- Open discussion on real-world ADF challenges
- Additional resources for further learning
Prerequisites
- Experience with Azure Data Factory pipeline development
- Basic understanding of JSON-based configurations in ADF
- Familiarity with Azure DevOps, Databricks, and Synapse
To attend this course, please install Microsoft Teams.
