We are looking for a Microsoft Cloud Data Engineer. The engineer will be in charge of:
Creating generic and parametrized pipelines in Azure DataFactory that will act as a template for ELT processes to orchestrate activities like:
Executing a Databricks notebook
Get and manage files from different sources.
Execute queries against On Premise and Cloud services (MSSQL Server, Azure SQL, Azure Synapse)
Creating Azure Functions with the Azure Durable Framework that will be integrated with the Azure Data Factory pipelines to define the different needed services. Those pipelines will need to be triggered on a Schedule Basis (TimeTrigger), Rest Web call (HttpTrigger) or MessageQueue (QueueMessageTrigger)
Specific skills/experience needed:
Cloud Services experience (AWS, GCP, Azure). Azure specific experience is desirable.
Creating and Maintaining Azure resources
DevOps experience is desirable ( Cloud Infrastructure Deployment and Automation (Azure ARM, Pulumi, Terraform...) )
CI/CD experience. Azure DevOps experience is desirable
Data Engineering experience. (Cloud Data Services is desirable). Some specific experience desired is:
Azure Data Factory
Azure Event Hubs (Queue Message Service)
Azure Data Lake
Azure SQL Database.
Microservices and Orchestration Background. Azure Durable Framework (Azure Function Apps Orchestration Interface) is desirable
Python programming experience is desirable or C# (Azure Durable framework supported)
Experience Creating and maintaining Microservices. Experience with Azure Functions is desirable.
Web Rest Services, JSON, semistructured files managing experience
Message Queue protocols experience
English and Spanish speaking (French will appreciate)