Senior Big Data Engineer

Job description

Job Description

As a Senior Big Data Engineer you are responsible for the data content and applications within the IFRS17 big data platform.
The IFRS17 chain consists of several (modern) technologies, i.e. Big Data technology that ingests and exposes the data from over 15 source systems, Azure Cloud, Power BI, SAS and Oracle Financials.
The current Big Data Platform is on Hadoop (to be migrated to Cloudera Data Platform and Cloud technology in 2023) with Informatica Big Data for the ETL (to be migrated to PySQL and PySpark in 2023).
The majority of the data streams is fully automated in the IFSR17 Chain.
You do not only understand data, and big data technologies but also have hands-on programming experience with languages like Python, Spark and SQL.
You are part of a team that works agile in close cooperation with business and our development partners.


  • Perform and develop end-2-end additional ingestions of new source data in the data platform.
  • Implement data mappings for the IFRS17 contract data
  • Maintain the interfaces with the Big Data platform
  • Develop test scripts (unit, integration and system) and integrate automation of those tests in the CI/CD pipeline
  • Conduct GDPR analysis on data on new ingestions and new use cases, and advise the Security Architect on outcomes
  • Monitor the data loads and the jobs running within the big data platform
  • Keep the documentation up-to-date on the data catalogue, the Data architecture, Data flow diagrams and Data Lineage amongst others
  • Support, propose and execute Proof of Concepts for prospective tools and improvements to the data platform
  • Participate and provide input in future roadmap discussions
  • Provide guidance and be a sparring partner for continuous improvement throughout the IFSR17 Chain.

Skills and Experience:

  • Experience as big data engineer in middle to large companies >5 years of work experience
  • Proficient understanding of distributed computing principles
  • In-depth knowledge of Hadoop, Cloudera Data Platform, Spark, and Databricks
  • Good knowledge of Big Data querying tools, such as Hive.
  • Good knowledge of Azure, Azure data engineering, Azure data services and modern data warehouses within Azure.
  • Experience with data modelling
  • Knowledge of various ETL techniques and frameworks
  • Experience with NoSQL and RDBMS databases
  • In-depth knowledge of scripting languages Python, Spark, SQL and R
  • Knowledge of automated testing and CI/CD, like Jenkins, GitHub and Azure pipelines
  • Experience with Agile development and working in a DevOps environment.
  • Experience with Jira and Confluence
  • Strong analytical and conceptual skills used to evaluate data gathered from multiple sources, while reconciling discrepancies
  • Thorough, analytical, and precise, but also big picture/helicopter view
  • Strong in structuring and organizing data.
  • Hands -on, proactive, and can work very focused at a fast pace.
  • Excellent communication skills (written and verbal) and stakeholder management
  • Effective IT data documentation skills.
  • Curious and thinking out-of-the-box
  • Ability to remove unneeded complexity without oversimplification
  • Strong customer focus and demonstrated ability to work in geographically dispersed teams.
  • Familiarity of orchestration/integration/API’s
  • Experience in working in an DevOps mode in an international team and setting
  • Ability to work in a team environment and efficiently under pressure
  • Fluent English, spoken and written
  • Willingness to travel

What do we offer?

  • A dynamic, international and challenging work environment
  • Training and support to reach your full potential including the opportunity for continuous professional development
  • Attractive terms and conditions, including competitive salary, pension package and a range of flexible benefits and rewards
  • Challenging tasks with individual development and training opportunities

Do you have any questions about our offer?
Aaron PURMANAN, Human Resources, is available by e-mail at

I am Atradius! - Do you want to know who we are?
Get to know Atradius colleagues in this video:

Atradius is a global provider of credit insurance, bond and surety, collections and information services, with a strategic presence in over 50 countries. The products offered by Atradius protect companies around the world against the default risks associated with selling goods and services on credit. Atradius is a member of Grupo Catalana Occidente (GCO.MC), one of the largest insurers in Spain and one of the largest credit insurers in the world.

Senior Big Data Engineer
ITSER-IT Services
31/08/2022, 16:59
Employment type:
Full time
Position type:
Job experience level:
Intermediate Level, Senior Level
Spain, Madrid

Please let the company know that you found this position on this Job Board as a way to support us, so we can keep posting cool jobs.