We are looking for an experienced Full Stack Data Engineer to join our team, building a next-generation data platform built on Data Mesh architecture/principles.Responsibilities
You will be responsible for designing, developing, and maintaining web applications and data pipelines, as well as implementing CI/CD best practices.
- Design, build, and maintain scalable and efficient data platforms using data engineering technologies such as Glue, EMR, Athena, Redshift, Lake Formation, Apache Spark, Hive, HDFS, and Trino.
- Build/manage data pipelines, and common data-related cross-cutting concerns like data catalog, data lineage, data quality, data profiling, data discovery, metadata management
- Develop and maintain web applications using AngularJS and Python.
- Build/manage BI/Analytical dashboard reducing time to insight for the business stakeholders.
- Implement CI/CD pipelines using Terraform, Jenkins, Github actions, and Gitflow.
- Develop and maintain APIs using Python and ensure API security and best practices are implemented. Implement SSO integration with Microsoft Azure AD using oAuth, OIDC, and SAML.
- Implement integration with AWS Cognito for user authentication and authorization.
The ideal candidate should have extensive hands-on experience in building a big data platform, Big Data Technologies, Data Pipelines, backend development (using Python), BI/Analytics tools as well as experience with DevOps, AWS, and UI Development in Angular JS.
- 8+ years of experience in similar positions;
- Hands-on Experience with data engineering technologies such as AWS Glue, EMR, Athena, Redshift, Lake Formation, Apache Spark, Apache Hive, Apache Airflow, S3FS, Apache Hudi, and Trino.
- Extensive experience in building data pipelines using orchestration tools like Apache Airflow. Hands-on experience in building cross-cutting concerns like data catalog, data lineage, data quality, data profiling, data discovery, metadata management
- Proven experience as a Full Stack Developer with AngularJS and Python.
- Experience working with RESTful APIs and JSON. Familiarity with microservices architecture.
- Experience with core AWS technologies such as EC2, ELB, Auto Scaling, S3, EFS, Lambda, API Gateway, Step Functions, Cloudwatch, VPC, Route 53, ACM
- Hands on experience with SQL and NoSQL databases.
- Hands experience with BI tools like Tableau, AWS QuickSight
- Experience with Git or other version control systems.
- Understanding of agile development methodologies.
- Experience with cloud platforms such as AWS.
- Bachelor's degree in Computer Science, Engineering, or related field