Team Lead, Data Engineering

Job description

Fairmarkit is the intelligent sourcing platform that empowers organizations to more efficiently purchase the goods and services they need. By equipping procurement and supply-chain teams with automation and data, Fairmarkit promotes competitive bidding while reducing manual work within existing processes. Fairmarkit has been recognized with awards by organizations such as Gartner and IDC, and is backed by strategic investors like GGV Capital, Insight Partners, 1984.VC, and Newfund.

About the role:

Fairmarkit is looking for an experienced data engineer to lead our growing core data team. This team is responsible for architecting, building, and maintaining the data infrastructure that supports all aspects of the business, including:
  • User-facing analytics products
  • ML model retraining pipelines
  • Ad-hoc analytic queries for internal reporting
  • Search index refresh pipelines
  • Large-scale data science research

In this role, you’ll work alongside our core engineering teams and data scientists as well as key business stakeholders. Your skills in large-scale data engineering will function as a value amplifier across the company.

Primary responsibilities:
  • Develop understanding of key business drivers in order to collect, organize, and standardize data to generate business insights and support key reporting needs
  • Write reliable, resilient, maintainable ETL pipelines for ingesting data from disparate sources, from third-party SaaS platforms to internal relational databases
  • Warehouse and expose data through easy-to-use interfaces, maintaining robust role-based access controls and logging to ensure data security
  • Architect and maintain distributed cluster-based big data processing infrastructure

Essential requirements:
  • 5+ years total backend development experience with at least 2+ years focused primarily on data engineering
  • Deep understanding of PostgreSQL as a systems level (the words “replication slot” should mean something to you)
  • Advanced Python expertise (we use 3.9)
  • Experience with modern data engineering stack: 
    • Data processing (e.g. Spark, Hadoop, EMR, Databricks)
    • Data cataloging (e.g. Glue, Data Factory)
    • Data formats (e.g. Avro, Parquet)
    • Stream processing (e.g. Kafka, Flink)
    • Workflow orchestration (e.g. Argo, Airflow)
  • Developing in/on containerized environments should be second nature (docker, docker-compose, k8s)
  • Knowledge of and preference for managing infrastructure as code (terraform, helm, ansible)

Bonus points for:
  • Knowledge of DevOps best practices, including version control, CI/CD, and monitoring technologies
  • Experience working with full-text search engines (e.g. Elasticsearch/OpenSearch, Lucene, Solr)
  • General-purpose Java development expertise; double points if you’ve submitted a custom jar to a Hadoop cluster
  • Operational experience with machine learning and analytics models and services, including retraining pipelines, versioning, and release management (MLOps)

Headquartered in Boston, and backed by a $30M Series B co-led by GGV Capital and Insight Partners, we are looking for exceptional candidates who want to help grow our company into a global enterprise and make their mark on the B2B tech industry. Come soar to new heights with us!

Fairmarkit is an equal opportunity employer, and selects individuals best matched for the job based upon job-related qualifications regardless of race, religion, color, creed, sex, sexual orientation, age, ancestry, national origin, gender identity, genetic information, disability, pregnancy, veteran or military status or any other status or characteristic protected by law.

Please let the company know that you found this position on this Job Board as a way to support us, so we can keep posting cool jobs.