Staff Data Engineer - Ingest

Obsidian makes SaaS security simple with an innovative, data driven approach to protecting the SaaS space. The company was founded by veterans in endpoint security and is backed by Greylock, Wing, and Google Ventures. The leadership team includes innovators from AWS, Carbon Black, Cylance, and Shape Security, where they have delivered enterprise-grade products to thousands of customers.

At Obsidian, machine learning and analysis are at the core of our business. As a Data Engineer, you will be part of a highly visible, agile team within the Data Science Group working on critical problems that directly affect the company’s success. The Data Engineer role provides a unique opportunity to work across many functions of our growing business including data science, engineering, and product.

Obsidian is seeking an experienced data engineer to build and extend our code engine/ data pipeline framework, enabling our integration partners to build a customizable computational graph. This framework allows our partners to transform semi-structured input logs from Saas applications into Obsidian common schema to answer security questions. This expert engineer is expected to have deep technical knowledge in data architecture; data semantics and software library design.

Who You Are:

• You’re an engineer who is experienced in solving data problems with rigorous data semantics.
• You enjoy building and architecting frameworks that are extensible and scalable.
• You pride yourself in building highly performant data pipelines.

What You'll Do:

• Architect and build the next-generation data engine that would unlock partnership with external developers to extend and scale our existing data engine.
• Contribute heavily to the various python microservices that involve transformation and enrichment of various data streams.
• Work with data scientists/ architects to build robust data processing primitives to help solve cybersecurity questions.
• Develop and maintain an analytics pipeline for the acquisition, storage, and processing of heterogeneous data types feeding real-time machine learning models.

Required skills/experience:

• Expert level knowledge in Python.
• Demonstrated experience in building frameworks/ libraries in Python.
• Demonstrated experience in building concurrent reactive systems.
• Extensive experience with SQL, NoSQL, graph databases.
• Familiarity/ experience with frameworks like Dagster/ Airflow/ Prefect.
• Experience with functional programming principles is highly desirable.
• Familiarity with DevOps.
• 7+ years of experience in Python.

fYnqZlShos