Qualified Health PBC 5 дней назад

Senior Staff Engineer, Data Platform

215 000 – 235 000 $в год
  • United States
  • Гибрид
  • Опыт от 12 лет
PythonAISQLLLMSparkDatabricksTemporalDelta LakeUnity Catalog

Описание

Responsibilities

  • Own the lakehouse. Architect and evolve our Databricks lakehouse, including medallion layering (bronze, silver, gold), Delta Lake, Unity Catalog governance, and workspace and compute strategy.
  • Build and tune Databricks pipelines. Design, build, and optimize Spark and SQL pipelines on Databricks, including incremental processing, data quality enforcement, performance tuning, and cost management.
  • Make data trustworthy. Define data contracts, quality checks, access controls, and PHI-aware governance in partnership with security and compliance.
  • Power AI and LLM workloads. Build the data foundations for AI products: LLM-ready datasets, retrieval and vector-store pipelines, evaluation data, and the interfaces that let AI and product teams consume governed data safely.
  • Build AI pipelines on Temporal. In addition to Databricks, design durable workflows on Temporal for AI pipelines: scheduled and watermarked pulls from Databricks gold tables, file-ingestion workflows, and long-running LLM extraction and enrichment jobs. Establish patterns for retries, idempotency, versioning, and backfills, and keep the handoff between Databricks and Temporal clean, observable, and well governed.
  • Lead agentic engineering. Use AI coding agents daily and define how the team does agentic development: tooling, prompts, guardrails, code review practices, and test strategy for agent-written code.
  • Set technical direction. Write and review technical designs, set platform-wide standards, and make build-versus-buy decisions that hold up over multiple years.
  • Multiply the team. Mentor engineers across teams, unblock other pods, and communicate trade-offs clearly to technical and non-technical stakeholders.

Requirements

  • Bachelors Degree in Computer Science, Electrical Engineering, Mathematics or a related field
  • At least 12+ years of software or data engineering experience, with a track record of designing and operating production data platforms at scale and of setting architecture across multiple teams.
  • Experience working in a highly regulated domain (Healthcare, Insurance, Finance) is required
  • Deep, hands-on expertise with Databricks is highly preferred but expertise in Snowflake or Microsoft Fabric would be aligned as well.
  • Experience with the lakehouse stack: Spark, Delta Lake, Unity Catalog, Workflows or equivalent, and performance and cost tuning at scale.
  • Experience building pipelines for ML or LLM applications, such as LLM-based extraction, embeddings and vector search, feature pipelines, or evaluation datasets.
  • Fluency in Python and SQL, with sound software engineering practices: testing, CI/CD, infrastructure as code, and code review.
  • Demonstrated practical use of agentic coding tools (such as Claude Code or similar) in real delivery work, and good judgment about when to trust, verify, and constrain them.
  • Experience owning ambiguous, cross-team problems end to end and influencing without direct authority.
  • Clear written and verbal communication, including authoring design docs that others build from.

Nice to have

  • Experience with healthcare data: claims, EHR, FHIR or HL7, and HIPAA-compliant handling of PHI.
  • Databricks certifications or experience with Delta Live Tables, Lakeflow, or Databricks Asset Bundles.
  • Experience with other orchestrators (Airflow, Dagster, Prefect) and a view on when each is the right tool.
  • Experience with cloud infrastructure (GCP, AWS, or Azure), Terraform, and Kubernetes.
  • Contributions to open-source data, workflow, or AI infrastructure projects.

We offer

  • Competitive salaries with equity packages
  • Robust medical/dental/vision insurance
  • Flexible working hours
  • Hybrid work options
  • An inclusive environment that fosters creativity and innovation.

Hiring process

  • To apply, please send your resume through the application below.