Middle Data Engineer

Описание

We are looking for a Senior Data Engineer with strong Databricks, Python, and LLM pipeline experience to join a team turning large volumes of clinical-research documents into reliable, AI-ready data on Azure.

Responsibilities

• Extracting and structuring data from large collections of clinical-research PDFs
• Building bronze, silver, and gold layers in Databricks with Python notebooks and Delta tables
• Integrating LLM endpoints into data pipelines and monitoring token spend
• Building a Neo4j graph alongside Delta tables for agents to query
• Preparing gold-layer data for an agentic layer used as an API tool
• Consolidating scattered and redundant Databricks job pipelines

Requirements

• Strong hands-on experience with Databricks, especially Databricks Jobs
• Advanced Python skills, writing notebooks daily from ingestion to the gold layer
• Solid understanding of medallion architecture and how to define bronze / silver / gold boundaries
• Experience working with Delta tables
• Experience processing large volumes of PDF documents (100K+ files)
• Experience calling LLM endpoints from notebooks and tracking token usage and cost
• Experience with Azure cloud
• Ability to design and consolidate job orchestration pipelines

Nice to have

• Experience with Neo4j or other graph databases
• Experience with healthcare or clinical-research data
• Experience building data layers consumed by AI agents

We offer

• performance-based bonus
• Salary: $600 – $800 (Net)
• Format: Remote