Qualified Health PBC 5 дней назад
Senior Staff Engineer, Data Platform
215 000 – 235 000 $в год
- United States
- Гибрид
- Опыт от 12 лет
PythonAISQLLLMSparkDatabricksTemporalDelta LakeUnity Catalog
Описание
Responsibilities
- Own the lakehouse. Architect and evolve our Databricks lakehouse, including medallion layering (bronze, silver, gold), Delta Lake, Unity Catalog governance, and workspace and compute strategy.
- Build and tune Databricks pipelines. Design, build, and optimize Spark and SQL pipelines on Databricks, including incremental processing, data quality enforcement, performance tuning, and cost management.
- Make data trustworthy. Define data contracts, quality checks, access controls, and PHI-aware governance in partnership with security and compliance.
- Power AI and LLM workloads. Build the data foundations for AI products: LLM-ready datasets, retrieval and vector-store pipelines, evaluation data, and the interfaces that let AI and product teams consume governed data safely.
- Build AI pipelines on Temporal. In addition to Databricks, design durable workflows on Temporal for AI pipelines: scheduled and watermarked pulls from Databricks gold tables, file-ingestion workflows, and long-running LLM extraction and enrichment jobs. Establish patterns for retries, idempotency, versioning, and backfills, and keep the handoff between Databricks and Temporal clean, observable, and well governed.
- Lead agentic engineering. Use AI coding agents daily and define how the team does agentic development: tooling, prompts, guardrails, code review practices, and test strategy for agent-written code.
- Set technical direction. Write and review technical designs, set platform-wide standards, and make build-versus-buy decisions that hold up over multiple years.
- Multiply the team. Mentor engineers across teams, unblock other pods, and communicate trade-offs clearly to technical and non-technical stakeholders.
Requirements
- Bachelors Degree in Computer Science, Electrical Engineering, Mathematics or a related field
- At least 12+ years of software or data engineering experience, with a track record of designing and operating production data platforms at scale and of setting architecture across multiple teams.
- Experience working in a highly regulated domain (Healthcare, Insurance, Finance) is required
- Deep, hands-on expertise with Databricks is highly preferred but expertise in Snowflake or Microsoft Fabric would be aligned as well.
- Experience with the lakehouse stack: Spark, Delta Lake, Unity Catalog, Workflows or equivalent, and performance and cost tuning at scale.
- Experience building pipelines for ML or LLM applications, such as LLM-based extraction, embeddings and vector search, feature pipelines, or evaluation datasets.
- Fluency in Python and SQL, with sound software engineering practices: testing, CI/CD, infrastructure as code, and code review.
- Demonstrated practical use of agentic coding tools (such as Claude Code or similar) in real delivery work, and good judgment about when to trust, verify, and constrain them.
- Experience owning ambiguous, cross-team problems end to end and influencing without direct authority.
- Clear written and verbal communication, including authoring design docs that others build from.
Nice to have
- Experience with healthcare data: claims, EHR, FHIR or HL7, and HIPAA-compliant handling of PHI.
- Databricks certifications or experience with Delta Live Tables, Lakeflow, or Databricks Asset Bundles.
- Experience with other orchestrators (Airflow, Dagster, Prefect) and a view on when each is the right tool.
- Experience with cloud infrastructure (GCP, AWS, or Azure), Terraform, and Kubernetes.
- Contributions to open-source data, workflow, or AI infrastructure projects.
We offer
- Competitive salaries with equity packages
- Robust medical/dental/vision insurance
- Flexible working hours
- Hybrid work options
- An inclusive environment that fosters creativity and innovation.
Hiring process
- To apply, please send your resume through the application below.