Lead Data Engineer

BigCommerce is disrupting the e-commerce industry as the SaaS leader for fast- growing, mid-market businesses. We enable our customers to build intuitive and engaging stores to support every stage of their growth.
We are looking for a Lead Data Engineer to join our Data Engineering team. Leveraging your big data skills, you will help design and build a Realtime, centralized data distribution service for our core business systems to achieve faster and bigger results. This involves deciphering value from several systems with a variety of access patterns. You will report to Sr. Director of IT and work with cross-functional teams in a highly visible role. The Senior Data Engineer will mentor junior engineers and communicate with senior leadership.

What you'll do

Build scalable near real time data pipelines on AWS and GCP that collects, transforms, loads and curates data from various internal and external data sources

Build a scalable distributed data store that will be central source of truth

Create scalable and low latency solutions

Implement machine learning pipelines and algorithms

Evaluate new technologies and build prototypes for continuous improvements in Data Engineering

Contribute to setting standards and development principles for the team and influence across the larger organization

Quality Skills

Write automated unit, integration and acceptance tests to support our continuous integration pipelines

Build performance and load tests written from scalability and resiliency standpoint

Participate in peer code reviews and advocate for the best coding practices and principles

Partners with Infrastructure and Engineering teams to ensure instrumentation, logging and monitoring is in place

Team / Leadership Skills

Actively participate with our development team in all phases of the software development lifecycle, including requirements gathering, functional and technical design, development, testing and roll-out, and support

Estimate work items and help maintain a scrum backlog

Lead collaboration across different teams

Distill and present research findings to both technical and non-technical leaders

Plays a key role in defining data engineering strategy and communicating priorities to others

Coach and mentor others, models the company values

Partners with engineering management to determine hiring needs and helps with recruiting activities

Who you are

BS or MS in Computer Science or equivalent experience or field

8+ years developing data and software solutions

4+ years of experience in Java, Scala and/or Python

4+ years of experience in building data pipelines in the cloud with tools like S3, Hadoop, Hive; or their equivalents

4+ years of experience in streaming technology such as Kafka

Strong experience with AWS and/or GCP

Strong experience with SQL, ETL, Data Warehousing and Data Orchestration

Experience with scheduling frameworks, preferably Airflow

Experience with monitoring near real time scalable data pipelines

Experience with of all aspects of data systems including database design, ETL, aggregation strategy, performance optimization, and technology trade-offs

Experience with Terraform, Puppet, and Jenkins

Experience with shell scripting

Focused developer with a strong sense of ownership

Ability to independently drive projects consisting of many stories from inception through to completion and production deployment

Capable of working closely across multiple teams to ensure data solutions line up with business initiatives and are of high quality

Excellent written and verbal communication skills

Our Hiring Processes Might Include

We want to see your problem-solving and analytical skills. Be prepared to write good, clean, scalable code. You don't need to know our entire stack, but we're looking for practical experience, someone who can solve production problems in the cloud.

Recruiter Phone Screen

Hiring Manager Screening

Final Team Interview

Note: Visa Work Authorization Sponsorship Supported and Relocation Assistance Provided

#LI-GC1

Diversity, Equity & Inclusion at BigCommerce

We have the opportunity to build not only a great business but a great company, with soul. Our beliefs and commitment to diversity, equity and inclusion are a central part of achieving that.

Our dedication to DEI is grounded in two things: a moral belief in the dignity, value, and potential of every individual, and a practical belief that diverse, inclusive teams will create the best outcomes for our customers, partners, employees, and company. We welcome everyone to be a part of our journey .

Current BigCommerce Employees: Please use the internal job board to apply for openings