We are looking for a Microsoft Cloud Data Engineer. The engineer will be in charge of:
- Creating generic and parametrized pipelines in Azure DataFactory that will act as a template for ELT processes to orchestrate activities like:
- Executing a Databricks notebook
- Get and manage files from different sources.
- Execute queries against On Premise and Cloud services (MSSQL Server, Azure SQL, Azure Synapse)
- Creating Azure Functions with the Azure Durable Framework that will be integrated with the Azure Data Factory pipelines to define the different needed services. Those pipelines will need to be triggered on a Schedule Basis (TimeTrigger), Rest Web call (HttpTrigger) or MessageQueue (QueueMessageTrigger)