Full Stack Data Engineer

Summary

Job Description

We are looking for an experienced Full Stack Data Engineer to join our team, building a next-generation data platform built on Data Mesh architecture/principles.

Responsibilities

You will be responsible for designing, developing, and maintaining web applications and data pipelines, as well as implementing CI/CD best practices.

• Design, build, and maintain scalable and efficient data platforms using data engineering technologies such as Glue, EMR, Athena, Redshift, Lake Formation, Apache Spark, Hive, HDFS, and Trino.
• Build/manage data pipelines, and common data-related cross-cutting concerns like data catalog, data lineage, data quality, data profiling, data discovery, metadata management
• Develop and maintain web applications using AngularJS and Python.
• Build/manage BI/Analytical dashboard reducing time to insight for the business stakeholders.
• Implement CI/CD pipelines using Terraform, Jenkins, Github actions, and Gitflow.
• Develop and maintain APIs using Python and ensure API security and best practices are implemented. • Implement SSO integration with Microsoft Azure AD using oAuth, OIDC, and SAML.
• Implement integration with AWS Cognito for user authentication and authorization.
  
  

Qualifications

The ideal candidate should have extensive hands-on experience in building a big data platform, Big Data Technologies, Data Pipelines, backend development (using Python), BI/Analytics tools as well as experience with DevOps, AWS, and UI Development in Angular JS.

• 8+ years of experience in similar positions;
• Hands-on Experience with data engineering technologies such as AWS Glue, EMR, Athena, Redshift, Lake Formation, Apache Spark, Apache Hive, Apache Airflow, S3FS, Apache Hudi, and Trino.
• Extensive experience in building data pipelines using orchestration tools like Apache Airflow. Hands-on experience in building cross-cutting concerns like data catalog, data lineage, data quality, data profiling, data discovery, metadata management
• Proven experience as a Full Stack Developer with AngularJS and Python.
• Strong understanding of web development technologies including HTML, CSS, and JavaScript.
• Experience working with RESTful APIs and JSON. Familiarity with microservices architecture.
• Experience with core AWS technologies such as EC2, ELB, Auto Scaling, S3, EFS, Lambda, API Gateway, Step Functions, Cloudwatch, VPC, Route 53, ACM
• Hands on experience with SQL and NoSQL databases.
• Hands experience with BI tools like Tableau, AWS QuickSight
• Experience with Git or other version control systems.
• Understanding of agile development methodologies.
• Experience with cloud platforms such as AWS.
• Bachelor's degree in Computer Science, Engineering, or related field