Senior Data Engineer

The Magic team is seeking a Senior Data Engineer to join our dynamic team and lead the charge in revolutionizing the data engineering space. If you are passionate about data, possess a creative mindset, and have a track record of pushing the envelope in data engineering, we want to hear from you.

As a Data Engineer, you will play a crucial role in the organization's data strategy by developing and maintaining data pipelines, ETL (Extract, Transform, Load) processes, and data architecture. You will work closely with data scientists, analysts, and other stakeholders to ensure data is accessible, accurate, and available for various business needs.

Responsibilities

Data Pipeline Development: Design, develop, and maintain data pipelines to ingest, transform, and load data from various sources into data warehouses or other storage solutions.

ETL Processes: Create and optimize ETL processes to ensure data quality, consistency, and availability for analytics and reporting.

Data Architecture: Collaborate with data architects to define and implement data architecture, ensuring scalability and performance.

Data Integration: Integrate and consolidate data from disparate sources, both structured and unstructured, for analysis and reporting purposes.

Data Modeling: Develop and maintain data models and schemas for efficient data storage and retrieval.

Data Governance: Implement and enforce data governance policies to maintain data quality, security, and compliance with regulations.

Performance Optimization: Continuously monitor and optimize data pipelines and systems for improved performance and efficiency.

Data Documentation: Maintain clear and comprehensive documentation for data pipelines, processes, and systems to facilitate collaboration and knowledge sharing.

Collaboration: Work closely with data scientists, analysts, software engineers, and business stakeholders to understand data requirements and provide data solutions.

Quality Assurance: Conduct data validation, testing, and quality checks to ensure the accuracy and integrity of data.

Security: Implement data security measures to protect sensitive data and ensure compliance with data privacy regulations.

Monitoring and Troubleshooting: Set up monitoring systems and respond to data pipeline failures, resolving issues promptly.

Qualifications

Bachelor's or higher degree in computer science, information technology, or a related field.

Proficiency in programming languages such as Python, Java, or Scala.

Experience with data processing and ETL tools like Apache Spark

Strong knowledge of SQL and database management systems (e.g., SQL Server).

Familiarity with data warehousing solutions.

Knowledge of data modeling, data architecture, and database design.

Experience with version control systems (e.g., Git) and collaboration tools.

Strong problem-solving skills and attention to detail.

Excellent communication and collaboration skills.

Microsoft is an equal opportunity employer. Consistent with applicable law, all qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.