Description
𝐅𝐢𝐫𝐬𝐭 20 𝐫𝐞𝐠𝐢𝐬𝐭𝐫𝐚𝐧𝐭𝐬 𝐠𝐞𝐭 𝐭𝐡𝐞𝐬𝐞 𝐄𝐗𝐂𝐋𝐔𝐒𝐈𝐕𝐄 𝐛𝐨𝐧𝐮𝐬𝐞𝐬:
📓 𝘍𝘰𝘭𝘭𝘰𝘸-𝘢𝘭𝘰𝘯𝘨 𝘸𝘰𝘳𝘬𝘣𝘰𝘰𝘬 𝘵𝘰 𝘶𝘴𝘦 𝘥𝘶𝘳𝘪𝘯𝘨 𝘢𝘯𝘥 𝘢𝘧𝘵𝘦𝘳 𝘵𝘩𝘦 𝘴𝘦𝘴𝘴𝘪𝘰𝘯
🎙️ 30-𝘮𝘪𝘯𝘶𝘵𝘦 𝘱𝘰𝘴𝘵-𝘤𝘭𝘢𝘴𝘴 𝘴𝘦𝘴𝘴𝘪𝘰𝘯 𝘸𝘪𝘵𝘩 𝘵𝘩𝘦 𝘦𝘹𝘱𝘦𝘳𝘵 𝘵𝘳𝘢𝘪𝘯𝘦𝘳 𝘵𝘰 𝘨𝘦𝘵 𝘩𝘦𝘭𝘱 𝘪𝘮𝘱𝘭𝘦𝘮𝘦𝘯𝘵𝘪𝘯𝘨 𝘸𝘩𝘢𝘵 𝘺𝘰𝘶’𝘷𝘦 𝘭𝘦𝘢𝘳𝘯𝘦𝘥
📼 1-𝘺𝘦𝘢𝘳 𝘢𝘤𝘤𝘦𝘴𝘴 𝘵𝘰 𝘵𝘩𝘦 𝘧𝘶𝘭𝘭 𝘴𝘦𝘴𝘴𝘪𝘰𝘯 𝘳𝘦𝘤𝘰𝘳𝘥𝘪𝘯𝘨 – 𝘳𝘦𝘷𝘪𝘦𝘸 𝘢𝘯𝘺𝘵𝘪𝘮𝘦
Course Outline*
*This is a new course and the outline is subject to change.
Module 1: Introduction to Team Development in ADF (30 min)
- Understanding ADF as a collaborative tool
- Best practices for multi-developer environments
- Git integration in ADF
- Branching strategies (Git Flow, feature branching, hotfixes, etc.)
- 🛠 Hands-on: Setting up ADF source control in Azure DevOps or GitHub
Module 2: ADF Version Control and CI/CD Pipelines (1 hour)
- Differences between ADF Live Mode vs. Git Mode
- Managing deployments using ARM templates
- Setting up a CI/CD pipeline in Azure DevOps
- YAML Pipeline for ADF
- 🛠 Hands-on: Building and deploying an end-to-end CI/CD pipeline for ADF
Module 3: Advanced Triggers in ADF (45 min)
- Understanding ADF triggers: Scheduled, Tumbling Window, and Event-Based Triggers
- Tumbling Window Triggers: Concepts, Use Cases, and Late Arrivals
- Event-Based Triggers: Monitoring file arrival in ADLS
- 🛠 Hands-on: Setting up a Tumbling Window Trigger
- 🛠 Hands-on: Setting up a Event-Based Trigger
Module 4: Notebook Execution in ADF (Fabric, Databricks, and Synapse) (45 min)
- When to use Fabric, Databricks, or Synapse Notebooks
- Executing Databricks Notebooks from ADF (Managed vs. External)
- Notebook execution in Microsoft Fabric Pipelines
- Connecting to Synapse Spark Notebooks
- Passing parameters to notebooks from ADF
- 🛠 Hands-on: Triggering a Databricks/Fabric Notebook from ADF
Module 5: Debugging, ADF Expressions, and Monitoring (45 min)
- Common ADF errors and troubleshooting techniques
- ADF Expressions & Functions:
- String functions (
concat,substring,replace) - Date functions (
formatDateTime,addDays,utcNow) - Collection functions (
filter,union,intersection)
- String functions (
- Debugging pipeline failures and JSON configuration issues
- Using Azure Monitor, Log Analytics, and Application Insights
- 🛠 Hands-on: Writing advanced ADF expressions and debugging a failing pipeline
Module 6: Incremental Load Patterns in ADF (45 min)
- Incremental vs. Full Load in ADF
- Using Watermark Columns for Incremental Loads
- Change Data Capture (CDC) in ADF
- Implementing Delta Lake Merge in Fabric/Synapse
- 🛠 Hands-on: Building an Incremental Load Pipeline with CDC
Module 7: Advanced Orchestration Patterns in ADF (45 min)
- Metadata-driven pipeline design
- Pipeline chaining and dependency management
- Looping & dynamic pipeline execution (
ForEach,Untilloops) - Dynamic parameterization for reusability
- Hybrid Orchestration (Combining ADF with Logic Apps or Functions)
- 🛠 Hands-on: Implementing a metadata-driven dynamic pipeline
- 🛠 Hands-on: Start / stop an SSIS run-time via azure automation
Q&A and Wrap-Up (30 min)
- Review key takeaways
- Open discussion on real-world ADF challenges
- Additional resources for further learning
Prerequisites
- Experience with Azure Data Factory pipeline development
- Basic understanding of JSON-based configurations in ADF
- Familiarity with Azure DevOps, Databricks, and Synapse
To attend this course, please install Microsoft Teams.
