Senior Databricks Data QA Engineer/Data Engineering
60 – 70 $в час
COOLSOFTRemoteУдалённоот 12 летКонтракт
PythonETLData EngineeringApache SparkDatabricksAzure Data FactoryDelta LakeSpark SQLAzure DatabricksData QAUnity CatalogDelta Live Tables
Описание
This role will build automated validation frameworks, establish deployment quality gates, and ensure the accuracy, reliability, performance, and compliance of enterprise data pipelines across healthcare payer data platforms.
Responsibilities
- Design and implement test plans, test cases, and automated validation for ETL/ELT pipelines
- Validate healthcare payer datasets, including claims, membership, provider, and clinical data
- Conduct FHIR-based API testing to support CMS interoperability compliance
- Validate HEDIS measure calculations and quality reporting data
- Build and maintain data quality automation frameworks using Python, PySpark, and Spark SQL
- Develop Azure DevOps test plans and QA gates for Databricks Jobs, Delta Live Tables, and Azure Data Factory deployments
- Collaborate with product owners, data engineers, governance teams, and business stakeholders to identify and resolve data quality issues
- Design automated load and stress tests for large-scale data pipelines
- Ensure data QA practices adhere to HIPAA, CMS, PII protection, and payer industry requirements
- Document QA processes, test results, and audit-ready evidence
- Mentor and guide junior QA engineers
Requirements
- 12+ years of Data Engineering experience
- 5+ years working exclusively on Databricks
- Strong expertise in Azure Databricks, including Workflows, Delta Lake, Delta Live Tables, and Unity Catalog
- Hands-on experience with Azure Data Factory pipelines, monitoring, and CI/CD
- Proficiency in Python, PySpark, and Spark SQL for data quality automation
- Experience validating healthcare payer data and working with FHIR APIs and HEDIS reporting
- Experience with Azure DevOps, automated deployment pipelines, and QA gates
- Knowledge of data governance, PII compliance, and data quality frameworks
- Experience testing the performance and scalability of data pipelines
- Strong communication, problem-solving, and technical leadership skills
- Willingness to convert to full-time employment after the initial contract period
Nice to have
- Experience with Databricks DQX
- Experience with Databricks Asset Bundles (DAB)
We offer
- Three-month contract-to-hire; full-time conversion expected after three months
- Occasional travel required
- Salary: $60 - $70/hr