Description
๐ ๐ข๐ซ๐ฌ๐ญ 20 ๐ซ๐๐ ๐ข๐ฌ๐ญ๐ซ๐๐ง๐ญ๐ฌ ๐ ๐๐ญ ๐ญ๐ก๐๐ฌ๐ ๐๐๐๐๐๐๐๐๐ ๐๐จ๐ง๐ฎ๐ฌ๐๐ฌ:
๐ย ๐๐ฐ๐ญ๐ญ๐ฐ๐ธ-๐ข๐ญ๐ฐ๐ฏ๐จ ๐ธ๐ฐ๐ณ๐ฌ๐ฃ๐ฐ๐ฐ๐ฌ ๐ต๐ฐ ๐ถ๐ด๐ฆ ๐ฅ๐ถ๐ณ๐ช๐ฏ๐จ ๐ข๐ฏ๐ฅ ๐ข๐ง๐ต๐ฆ๐ณ ๐ต๐ฉ๐ฆ ๐ด๐ฆ๐ด๐ด๐ช๐ฐ๐ฏ
๐๏ธย 30-๐ฎ๐ช๐ฏ๐ถ๐ต๐ฆ ๐ฑ๐ฐ๐ด๐ต-๐ค๐ญ๐ข๐ด๐ด ๐ด๐ฆ๐ด๐ด๐ช๐ฐ๐ฏ ๐ธ๐ช๐ต๐ฉ ๐ต๐ฉ๐ฆ ๐ฆ๐น๐ฑ๐ฆ๐ณ๐ต ๐ต๐ณ๐ข๐ช๐ฏ๐ฆ๐ณ ๐ต๐ฐ ๐จ๐ฆ๐ต ๐ฉ๐ฆ๐ญ๐ฑ ๐ช๐ฎ๐ฑ๐ญ๐ฆ๐ฎ๐ฆ๐ฏ๐ต๐ช๐ฏ๐จ ๐ธ๐ฉ๐ข๐ต ๐บ๐ฐ๐ถโ๐ท๐ฆ ๐ญ๐ฆ๐ข๐ณ๐ฏ๐ฆ๐ฅ
๐ผย 1-๐บ๐ฆ๐ข๐ณ ๐ข๐ค๐ค๐ฆ๐ด๐ด ๐ต๐ฐ ๐ต๐ฉ๐ฆ ๐ง๐ถ๐ญ๐ญ ๐ด๐ฆ๐ด๐ด๐ช๐ฐ๐ฏ ๐ณ๐ฆ๐ค๐ฐ๐ณ๐ฅ๐ช๐ฏ๐จ – ๐ณ๐ฆ๐ท๐ช๐ฆ๐ธ ๐ข๐ฏ๐บ๐ต๐ช๐ฎ๐ฆ
*This is a new course and the outline is subject to change.
Module 1: Introduction to Team Development in ADFย (30 min)
- Understanding ADF as a collaborative tool
- Best practices for multi-developer environments
- Git integrationย in ADF
- Branching strategies (Git Flow, feature branching, hotfixes, etc.)
- ๐ ย Hands-on:ย Setting up ADF source control in Azure DevOps or GitHub
Module 2: ADF Version Control and CI/CD Pipelinesย (1 hour)
- Differences betweenย ADF Live Mode vs. Git Mode
- Managing deployments usingย ARM templates
- Setting up aย CI/CD pipelineย in Azure DevOps
- YAML Pipeline for ADF
- ๐ ย Hands-on:ย Building and deploying anย end-to-end CI/CD pipelineย for ADF
Module 3: Advanced Triggers in ADFย (45 min)
- Understanding ADF triggers:ย Scheduled, Tumbling Window, and Event-Based Triggers
- Tumbling Window Triggers: Concepts, Use Cases, and Late Arrivals
- Event-Based Triggers: Monitoring file arrival in ADLS
- ๐ ย Hands-on:ย Setting up aย Tumbling Window Trigger
- ๐ ย Hands-on:ย Setting up aย Event-Based Trigger
Module 4: Notebook Execution in ADF (Fabric, Databricks, and Synapse)ย (45 min)
- When to useย Fabric, Databricks, or Synapse Notebooks
- Executingย Databricks Notebooksย from ADF (Managed vs. External)
- Notebook execution in Microsoft Fabric Pipelines
- Connecting toย Synapse Spark Notebooks
- Passingย parametersย to notebooks from ADF
- ๐ ย Hands-on:ย Triggering a Databricks/Fabric Notebook from ADF
Module 5: Debugging, ADF Expressions, and Monitoringย (45 min)
- Common ADF errors and troubleshooting techniques
- ADF Expressions & Functions:
- String functionsย (
concat,ยsubstring,ยreplace) - Date functionsย (
formatDateTime,ยaddDays,ยutcNow) - Collection functionsย (
filter,ยunion,ยintersection)
- String functionsย (
- Debuggingย pipeline failures and JSON configuration issues
- Usingย Azure Monitor, Log Analytics, and Application Insights
- ๐ ย Hands-on:ย Writing advanced ADF expressions and debugging a failing pipeline
Module 6: Incremental Load Patterns in ADFย (45 min)
- Incremental vs. Full Loadย in ADF
- Usingย Watermark Columns for Incremental Loads
- Change Data Capture (CDC)ย in ADF
- Implementingย Delta Lake Merge in Fabric/Synapse
- ๐ ย Hands-on:ย Building an Incremental Load Pipeline with CDC
Module 7: Advanced Orchestration Patterns in ADFย (45 min)
- Metadata-driven pipeline design
- Pipeline chaining and dependency management
- Looping & dynamic pipeline executionย (
ForEach,ยUntilย loops) - Dynamic parameterization for reusability
- Hybrid Orchestration (Combining ADF with Logic Apps or Functions)
- ๐ ย Hands-on:ย Implementing a metadata-driven dynamic pipeline
- ๐ ย Hands-on:ย Start / stop an SSIS run-time via azure automation
Q&A and Wrap-Upย (30 min)
- Review key takeaways
- Open discussion on real-world ADF challenges
- Additional resources for further learning
Prerequisites
- Experience withย Azure Data Factory pipeline development
- Basic understanding ofย JSON-based configurations in ADF
- Familiarity withย Azure DevOps, Databricks, and Synapse
To attend this course, please install Microsoft Teams.
