We are seeking a skilled and motivated Intermediate Data Engineer to join our dynamic team. The successful candidate will play a crucial role in designing, developing, and maintaining our data infrastructure, ensuring efficient processing and analysis of large datasets. As an Intermediate Data Engineer, you will collaborate with cross-functional teams to implement data solutions that align with our business objectives.
Responsibilities
Data Pipeline Development: Design, implement, and maintain scalable data pipelines for the extraction, transformation, and loading of large datasets.
Data Modelling: Develop and optimize data models to support analytical and reporting requirements, ensuring data accuracy and reliability.
Data Integration: Integrate diverse data sources and formats to enable seamless data flow across systems.
Data Quality Assurance: Implement and monitor data quality checks to ensure the integrity of the data throughout the entire data lifecycle.
Collaboration: Work closely with business analysts, colleagues and other stakeholders to understand data requirements and deliver effective solutions.
Performance Optimization: Identify and address performance bottlenecks in data processing and storage systems.
Documentation: Document data engineering processes, data models, and configurations for knowledge sharing and future reference.
Adherence to Best Practices: Stay current with industry trends and best practices in data engineering within company and ensure the adoption of these practices within the team.
Requirements
Bachelor's degree in Computer Science, Information Technology, or a related field.
Minimum of 5 years of proven experience as a Data Engineer or similar role.
Strong proficiency in programming languages such as Spark SQL and Python in a HIVE environment.
Familiarity with cloud platforms such as AWS and Big Data infrastructure, Jira, Bitbucket, etc.
Experience in CI/CD pipeline and working in an Agile environment.
Excellent problem-solving skills and attention to detail.
Strong communication and collaboration skills.
Knowledge of data warehousing concepts and technologies.
Candidates with experience in the Banking Industry will have an added advantage.
Ability to communicate effectively with multiple levels of management and summarize complex technical requests succinctly.