Senior Data Engineer

We are looking for a a Data Engineer for one of our renowned customer and you would be working alongside with Data Scientist / AI Architect in the team to develop scalable and production ready Advanced Analytics and AI software and products. Additionally, to develop different technical tools/services to enable large scale machine learning solutions.

You should believe in a non-hierarchical culture of collaboration, transparency, safety, and trust. Working with a focus on value creation, growth and serving customers with full ownership and accountability. Delivering exceptional customer and business results.

Responsibilities

• Design, develop and build real-time data pipelines from a variety of sources (streaming data, APIs, data warehouse, messages etc.)
• Leverage the understanding of software architecture and software design patterns to write scalable, maintainable well-designed and future-proof software
• Manage existing pipelines and create new pipelines from a variety of sources (relational, XML, etc.)
• Actively apply best practices within CI/CD
• Propose and implement solutions for data pipeline stabilization and data quality checks
• Coordination with other teams to design optimal patterns for data ingest and egress, as well as lead and coordinate data quality initiatives and troubleshooting
• Design and build solutions to track data quality, stabilize data pipeline, etc. to ensure reliable operations
• Ensure best practices are followed across architecture, codebase and configuration
• Eliminate waste
• Deliver on time

Competences

• Ability to establish with clear goals and responsibilities to achieve a high level of performance.
• Ability to evaluate different options proactively and ability to solve problems in an innovative way. Develop new solutions or combine existing methods to create new approaches.
• Comfortable in working with external product teams to establish the optimal data integration patterns/solutions

Functional Knowledge

• PySpark
• Python
• SQL
• Hadoop
• Jenkins
• Docker
• Kubernetes
• Git
• Azure Data Factory
• PowerShell
• Bash
• DevOps
• CI/CD
• Azure
• GCP
• Architecture Principles Design
• Agile Architecture Delivery
• React