The Senior Data Engineer will design, build and operate high performance data centric solutions utilizing the comprehensive big data capabilities for the company's data platform environment. In this role, you will act as an authority for data access pathways and techniques working with analysts within the functional data analytics team. You will design data structures and pipelines to collect data and design and implement data transformations, combinations or aggregations.
Provide necessary technical support through all phases of testing and incident handling after deployment.
Develop technical solutions using Hadoop and Big data technologies.
Configure and develop data pipelines to move and optimize data assets using SQL scripting, Spark using Scala or Python, Hive and Oozie.
Contribute to designing solutions for Cargill's big data and advanced analytics solutions aligned with requirements.
Independently solve moderately complex issues with minimal supervision, while escalating more complex issues to appropriate staff.
Other duties as assigned
6 or more years of experience in SQL scripting, Spark, Scala and Python
4 or more years of experience working with Hadoop or other big data platform and building solutions that deliver value through leveraging data as an asset.
Bachelor's degree in a related field or equivalent experience
Minimum of four years of related work experience
Other minimum qualifications may apply
Experience working with various types of data sources.
Applied experience and skills with application programming interface development
Applied experience in reporting tools like Power BI, Tableau etc.
Minimum of 6-10 years of information technology and business/industry work experience developing data or software applications including: analysis, design, coding, testing, deploying and supporting of applications