Middle Data Engineer

Описание

We are looking for a Senior Data Engineer with strong Databricks, PySpark, and data modelling experience to join an enterprise data platform team building a Lakehouse that powers a web app, LLM agents, and ML models.

Responsibilities

• Building and maintaining a Databricks Lakehouse on a medallion architecture
• Ingesting data from Oracle systems and APIs into the bronze layer
• Modelling the silver layer with Data Vault and the gold layer with dimensional models
• Optimising Spark performance and Delta table layouts
• Implementing governance, access control, and PII masking in Unity Catalog
• Replacing manual scheduling processes and spreadsheets with a unified data platform
• Collaborating with the application team that builds on top of the gold layer for the web app, LLMs, and ML models

Requirements

• Strong experience with Databricks and Lakehouse medallion architecture (bronze / silver / gold)
• Advanced PySpark skills for building transformations in notebooks
• Hands-on experience with Delta Lake
• Experience with Data Vault modelling (hubs, links, satellites)
• Experience with dimensional modelling (facts and dimensions)
• Experience ingesting data with Fivetran and custom API integrations
• Experience working with Oracle SQL databases as source systems
• Spark performance tuning, including skew handling and Liquid Clustering
• Experience deploying with Databricks Asset Bundles (DABs)
• Experience with Unity Catalog, group-based RBAC, and PII masking

Nice to have

• Experience with Great Expectations or similar data quality frameworks
• Experience handling unstructured files in Blob storage
• Experience supporting LLM / agent and ML use cases with curated data

We offer

• Salary: $600 – $800 (Net) + performance-based bonus
• Format: Remote