Listen to this Post
Azure Data Factory (ADF) Part 2: Types of Activities and Triggers
URL: Azure Data Factory Notes – Part 1
You Should Know:
Azure Data Factory (ADF) is a cloud-based data integration service that allows you to create data-driven workflows for orchestrating and automating data movement and transformation. In this article, we will explore the types of activities and triggers in ADF, along with practical commands and steps to implement them.
Types of Activities in ADF:
1. Data Movement Activities:
- Copy Data Activity: Moves data from a source to a sink.
{ "name": "CopyDataActivity", "type": "Copy", "inputs": [ { "referenceName": "SourceDataset", "type": "DatasetReference" } ], "outputs": [ { "referenceName": "SinkDataset", "type": "DatasetReference" } ], "typeProperties": { "source": { "type": "BlobSource" }, "sink": { "type": "BlobSink" } } } - Command: Use the `az datafactory pipeline create` command to create a pipeline with a Copy Data activity.
2. Data Transformation Activities:
- Mapping Data Flow: Transforms data using a visual interface.
{ "name": "TransformDataFlow", "type": "MappingDataFlow", "typeProperties": { "sources": [ { "name": "Source1", "dataset": { "referenceName": "SourceDataset", "type": "DatasetReference" } } ], "sinks": [ { "name": "Sink1", "dataset": { "referenceName": "SinkDataset", "type": "DatasetReference" } } ] } } - Command: Use the `az datafactory data-flow create` command to create a data flow.
3. Control Activities:
- Execute Pipeline Activity: Executes another pipeline.
{ "name": "ExecutePipelineActivity", "type": "ExecutePipeline", "typeProperties": { "pipeline": { "referenceName": "ChildPipeline", "type": "PipelineReference" } } } - Command: Use the `az datafactory pipeline create` command to create a pipeline with an Execute Pipeline activity.
Types of Triggers in ADF:
1. Schedule Trigger:
- Command: Use the `az datafactory trigger create` command to create a schedule trigger.
{ "name": "ScheduleTrigger", "type": "ScheduleTrigger", "typeProperties": { "recurrence": { "frequency": "Day", "interval": 1, "startTime": "2023-10-01T00:00:00Z", "endTime": "2023-10-31T23:59:59Z" } } }
2. Event Trigger:
- Command: Use the `az datafactory trigger create` command to create an event trigger.
{ "name": "EventTrigger", "type": "BlobEventsTrigger", "typeProperties": { "blobPathBeginsWith": "/mycontainer/", "blobPathEndsWith": ".csv" } }
3. Tumbling Window Trigger:
- Command: Use the `az datafactory trigger create` command to create a tumbling window trigger.
{ "name": "TumblingWindowTrigger", "type": "TumblingWindowTrigger", "typeProperties": { "frequency": "Hour", "interval": 1, "startTime": "2023-10-01T00:00:00Z", "endTime": "2023-10-31T23:59:59Z" } }
What Undercode Say:
Azure Data Factory (ADF) is a powerful tool for orchestrating and automating data workflows. Understanding the types of activities and triggers is crucial for building efficient data pipelines. The provided commands and JSON snippets should help you get started with implementing these concepts in your own projects. For more detailed information, refer to the Azure Data Factory documentation.
Related Commands:
- Linux Command: Use `curl` to interact with Azure REST APIs.
curl -X GET -H "Authorization: Bearer <ACCESS_TOKEN>" https://management.azure.com/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.DataFactory/factories/<FACTORY_NAME>/pipelines?api-version=2018-06-01
- Windows Command: Use `PowerShell` to manage ADF resources.
Get-AzDataFactoryV2Pipeline -ResourceGroupName "<RESOURCE_GROUP>" -DataFactoryName "<FACTORY_NAME>"
By mastering these activities and triggers, you can significantly enhance your data integration and transformation processes in Azure Data Factory.
References:
Reported By: Abhishek Agrawal – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
