This role is to improve our data and data pipeline architecture, as well as improving the data flow and collection for cross-functional teams. The ideal candidate will also have a chance to figure out things from the ground up for new component integrations.
Role will also include:
- Monitoring & managing data pipelines, ensuring accuracy and stability
- Develop and support BAU tasks
- POC and adopting new technologies to improve data platform management in large-scale high throughput.
- Identify, design and implement improvements, e.g. automating manual processes, optimising data delivery etc
- Overall accomplishment is to maintain a robust data platform that can support BI to AI activities.
At LoveBonito, we continuously look forward to improving things so as a Data Engineer you will enable better measurements and ensure measurement accuracy so that we know what we do and where we want to improve.
- Candidate will need to understand both business demands and available upstream system assets
- Understand structured and unstructured datasets
- Good Knowledge about big data systems like (Hadoop, HDFS, Hive, Apache Spark, Apache Flink, Kafka, etc.) with experience in handling batch/real time data streaming.
- Skilled in Python, Scala programming languages
- Strong Knowledge in SQL and good understanding of DBMS is required
- Proposing new technologies, middle-wares, tools etc. to improve architecture of systems
- Create and automate the data workflows such as extraction, transformation, load (ETL)
- Identify gaps or opportunities in existing systems and contribute to their improvement
- Stakeholder management
- Bring good knowledge / experience to the team
Qualifications & Experience
- 2 to 5 years experience in design and development of Datawarehouse Systems.
- About 1 to 2 years of experience in developing Big data solutions
- BS/MS in engineering or any other technical discipline
- Proven experience delivering production-ready data engineering solutions, including requirements definition, architecture selection, prototype development, debugging, unit-testing, deployment, support, and maintenance
Experience in the following will be considered advantageous:
- Airflow Orchestration
- Redshift/Big Query
- AWS/GCP cloud
- Deploying machine learning models and frameworks
- Building APIs