LEAD DATA ENGINEER – Azure
Years of Experience – 7 years and above
Location – Chennai, Coimbatore & Bangalore
Description
As an Azure Lead Data Engineer, your main responsibility is to design, develop, and oversee data systems on the Microsoft Azure cloud platform. You will lead a team of data engineers and collaborate with other teams to ensure the reliability and scalability of our data systems.
If you thrive on the challenges posed by big data, then this role is perfect for you. We are seeking Data Engineers who can handle a high volume of events daily, manage petabyte-scale data on Azure SQL, Azure Synapse Analytics and Azure Data Lake, and create data pipelines using technologies such as ADF, Synapse Pipeline, Synapse notebook (PySpark/SparkSQL) and SQL
We are looking for talented, enthusiastic, and detail-oriented individuals with a background in Data Engineering, Data Science, Business Intelligence, or Software Development. You should be adept at tackling big data challenges in an agile manner. Your responsibilities will include designing and analysing big data, data modelling, and developing, deploying, and managing big data pipelines. Additionally, you will assist in recruiting, mentoring, and developing colleagues in the Data Analytics team, including Data Scientists, Data Engineers, and Software Engineers. Your role will involve collaborating with Coda to construct data pipelines and datasets, as well as expanding self-service data knowledge.
This position requires you to operate at the intersection of data and engineering. You should have a deep understanding of data, analytical techniques, and how to apply insights to drive business outcomes. Practical experience in maintaining the highest operational standards for ETL and big data pipelines is essential.
We primarily work with Azure technologies such as Azure SQL, Azure Data Factory, Azure Synapse, Azure Dedicated SQL Pool, Azure Data Lake, Azure Databricks, PySpark, SparkSQL, Azure Analysis Services, SSIS, PowerBI, and Data Modelling. In this role, you will build our data portfolio and collaborate with Product, Marketing, BI, and ML teams to create new behavioural events, pipelines, datasets, models, and reports to support their initiatives. You will also enhance our offline analytics capabilities in Tableau and expand our real-time dashboarding capabilities
Basic Qualifications
- Designing and implementing large-scale data processing systems on Azure, including data ingestion, storage, processing, and visualization.
- Developing data integration and migration strategies to move data from legacy systems to Azure.
- Building data pipelines and ETL workflows using Azure Data Factory, Azure Databricks, or other relevant tools.
- Designing and implementing data models, data warehouses, and data lakes using Azure Synapse Analytics, Azure SQL Data Warehouse, or other relevant tools.
- Working with data scientists and analysts to understand their data requirements and ensure that the data infrastructure supports their needs.
- Monitoring and troubleshooting the performance and availability of the data systems and implementing strategies to improve their reliability and scalability.
- Managing security and access controls for the data systems, including managing roles and permissions, implementing encryption, and complying with regulatory requirements.
- Mentoring and guiding junior data engineers and providing technical leadership to the team.
