At IBM, work is more than a job - it's a calling: To build. To design. To code. To consult. To think along with clients and sell. To make markets. To invent. To collaborate. Not just to do something better, but to attempt things you've never thought possible. Are you ready to lead in this new era of technology and solve some of the world's most challenging problems? If so, lets talk.
Your Role and Responsibilities
The Data Engineer is responsible for building and supporting the big data platform and data pipelines. As a Data Engineer - Big Data, your responsibilities include –
- Design and implement complex Data Lake and Data mesh architectures utilising data fabric capabilities like streaming, event-driven and batch patterns.
- Write high-quality code, perform code reviews, and take ownership of software delivery.
- Responsible for building software development frameworks, best practices and guidelines for adoption.
- Identify and resolve defects during test phases, handle deployments, and coordinate with multiple teams to provide ongoing improvements to the code, processes, and overall design.
- Evangelise and democratise data both to internal as well as external stakeholders.
- Knowledge of test-driven development and test automation frameworks:
- Optionally you would have an Application Development background along with knowledge of front end development, Analytics libraries, open-source Natural Language Processing, statistical and big data computing libraries.
- Strong python developer having a great understanding of Python features in version 3.7+
- Experience in using jinja2 template in python
- Java or Scala
- Experience in designing and building data pipelines on Spark utilising PySpark
- Experience in building REST services using python. fast API experience is highly desirable
- Experience in Postgres database programming with python ORM framework like peewee
- Experience in static type checking tools with python and CI/CD
- Experience in AWS technology: Lambda, S3, SQS, SNS and event-sourced mapping
- Experience in python performance optimization, multi-threaded programming is highly desirable
- Front-end development experience with TS/Angular is desirable but not essential
Good to have skills
- Having a great understanding of Python features in version 3.7+ with a background working in data migration, data wrangling initiatives.
- Experience in using jinja2 template in python or other similar templates.
- Experience in building REST services using python. Fastapi experience is highly desirable or similar.
- Experience in SQL database programming using python ORM framework like peewee.
- Experience in static type checking tools with Python and experience in DevOps.
- Experience in AWS technology: Lambda, S3, SQS, SNS and event sourced mapping
- Design and development knowledge on Kafka and spark structured streaming (Databricks etc)
- Good understanding of data modelling patterns
- Working knowledge of Unix/Linux systems
- Working knowledge of multiple database paradigms like RDBMS and NoSQL.
- Strong knowledge of data security management policies and procedures
- Experience with Agile Development methodologies
- Strong troubleshooting and performance tuning skills
- Experience in Python performance optimization, multi-threaded programming
- Front-end development experience with TS/Angular
- 2+ years experience in the Financial Services domain
Required Technical and Professional Expertise
Preferred Technical and Professional Expertise
About Business UnitIBM Services is a team of business, strategy and technology consultants that design, build, and run foundational systems and services that is the backbone of the world's economy. IBM Services partners with the world's leading companies in over 170 countries to build smarter businesses by reimagining and reinventing through technology, with its outcome-focused methodologies, industry-leading portfolio and world class research and operations expertise leading to results-driven innovation and enduring excellence.
Your Life @ IBMAre you craving to learn more? Prepared to solve some of the world's most unique challenges? And ready to shape the future for millions of people? If so, then it's time to join us, express your individuality, unleash your curiosity and discover new possibilities.
Every IBMer, and potential ones like yourself, has a voice, carves their own path, and uses their expertise to help co-create and add to our story. Together, we have the power to make meaningful change – to alter the fabric of our clients, of society and IBM itself, to create a truly positive impact and make the world work better for everyone.
It's time to define your career.
IBM is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, colour, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.
About IBMIBM’s greatest invention is the IBMer. We believe that through the application of intelligence, reason and science, we can improve business, society and the human condition, bringing the power of an open hybrid cloud and AI strategy to life for our clients and partners around the world.Restlessly reinventing since 1911, we are not only one of the largest corporate organizations in the world, we’re also one of the biggest technology and consulting employers, with many of the Fortune 50 companies relying on the IBM Cloud to run their business. At IBM, we pride ourselves on being an early adopter of artificial intelligence, quantum computing and blockchain. Now it’s time for you to join us on our journey to being a responsible technology innovator and a force for good in the world.
Location StatementFor additional information about location requirements, please discuss with the recruiter following submission of your application.
Being You @ IBMIBM is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, pregnancy, disability, age, veteran status, or other characteristics. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.