Job description

Responsibilities

  • Design, develop, and manage data pipelines and ETL (Extract, Transform, Load) processes using AWS and Azure services to efficiently extract data from various sourcesand load it into data storage systems.
  • Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions that leverage both AWS and Azure cloud capabilities.
  • Build and maintain data warehouses, data lakes, and other data storage solutions on AWS and Azure, ensuring data organization and accessibility.
  • Implement data validation, cleansing, and transformation processes to ensure high data quality and integrity.
  • Optimize data processing systems for performance, scalability, and reliability on both AWS and Azure cloud platforms.
  • Monitor, troubleshoot, and resolve issues in data pipelines and systems to ensure uninterrupted data flow.
  • Implement security and compliance measures to safeguard sensitive data throughout the data lifecycle on both cloud platforms.
  • Stay up-to-date with the latest developments in data engineering technologies on AWS and Azure and recommend relevant tools and solutions.
  • Document data infrastructure, processes, and workflows to facilitate knowledge sharing and future reference.
  • Collaborate with DevOps and IT teams to deploy and operate data pipelines and systems seamlessly on both cloud platforms.
  • Evaluate and select data tools, technologies, and frameworks that align with both AWS and Azure environments.

Requirements

  • Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
  • Proven experience as a Data Engineer or in a similar role, with a strong understanding of data engineering concepts and best practices.
  • Proficiency in programming languages such as Python, Java, Scala, or similar languages.
  • Extensive hands-on experience with AWS services like S3, Redshift, Glue, EMR, Lambda, and relevant Azure services.
  • Familiarity with data warehousing solutions on both platforms, such as Amazon Redshift,Google BigQuery, and Snowflake.
  • Experience with data processing frameworks like Apache Spark, Apache Flink, or similar technologies on both AWS and Azure.
  • Strong knowledge of SQL and NoSQL database systems, data formats (JSON, Parquet, Avro), and data serialization.
  • Ability to troubleshoot complex data issues and implement effective solutions on both AWS and Azure cloud environments.
  • Excellent communication skills and the ability to collaborate effectively within crossfunctional
  • teams.
  • Experience with DevOps practices, CI/CD pipelines, and version control systems (e.g.,Git) is advantageous.
  • Relevant certifications in AWS (e.g., AWS Certified Data Analytics - Specialty) and Azure (e.g., Azure Data Engineer Associate) are a plus.

Please let the company know that you found this position on this Job Board as a way to support us, so we can keep posting cool jobs.