Design, develop, and manage data pipelines and ETL (Extract, Transform, Load) processes using AWS and Azure services to efficiently extract data from various sourcesand load it into data storage systems.
Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions that leverage both AWS and Azure cloud capabilities.
Build and maintain data warehouses, data lakes, and other data storage solutions on AWS and Azure, ensuring data organization and accessibility.
Implement data validation, cleansing, and transformation processes to ensure high data quality and integrity.
Optimize data processing systems for performance, scalability, and reliability on both AWS and Azure cloud platforms.
Monitor, troubleshoot, and resolve issues in data pipelines and systems to ensure uninterrupted data flow.
Implement security and compliance measures to safeguard sensitive data throughout the data lifecycle on both cloud platforms.
Stay up-to-date with the latest developments in data engineering technologies on AWS and Azure and recommend relevant tools and solutions.
Document data infrastructure, processes, and workflows to facilitate knowledge sharing and future reference.
Collaborate with DevOps and IT teams to deploy and operate data pipelines and systems seamlessly on both cloud platforms.
Evaluate and select data tools, technologies, and frameworks that align with both AWS and Azure environments.
Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
Proven experience as a Data Engineer or in a similar role, with a strong understanding of data engineering concepts and best practices.
Proficiency in programming languages such as Python, Java, Scala, or similar languages.
Extensive hands-on experience with AWS services like S3, Redshift, Glue, EMR, Lambda, and relevant Azure services.
Familiarity with data warehousing solutions on both platforms, such as Amazon Redshift,Google BigQuery, and Snowflake.
Experience with data processing frameworks like Apache Spark, Apache Flink, or similar technologies on both AWS and Azure.
Strong knowledge of SQL and NoSQL database systems, data formats (JSON, Parquet, Avro), and data serialization.
Ability to troubleshoot complex data issues and implement effective solutions on both AWS and Azure cloud environments.
Excellent communication skills and the ability to collaborate effectively within crossfunctional
Experience with DevOps practices, CI/CD pipelines, and version control systems (e.g.,Git) is advantageous.
Relevant certifications in AWS (e.g., AWS Certified Data Analytics - Specialty) and Azure (e.g., Azure Data Engineer Associate) are a plus.