Internship Project :
Extract, Transform, Load in Foresight Engine
Project Description and Objectives:
The objective of this project is to improve the ETL process in our foresight engine. We collect millions of unstructured data from various sources including social media, recipe databases and online ecommerce platforms.
Processing multiple data streams on the cloud infrastructure is important to generate actionable insights in time at scale. As a data engineer, you will be working on stream data processing using cloud tools, writing custom scripts (python) to clean and process data for downstream tasks, use monitoring and visualisation tools to manage the data ingestion and analytics.
Roles and Responsibilities:
1. Research on state of the art stream processing frameworks
2. Automate ETL pipeline with the help of DevOps team
3. Write custom analytical queries
4. Use visualization tools to manage the ETL pipeline
5. Work with insights team to generate custom analytics
SQL + NoSQL
Good to have:
Kasun Perera : https://www.linkedin.com/in/kasunsp/
Former Mentee Feedbacks: