ShyftLabs 6 дней назад

Senior Data Engineer - Databricks

  • Coimbatore
  • Опыт от 7 лет
PythonSQLETLSparkApache SparkELTDatabricksData GovernanceDelta LakeUnity Catalog

Описание

Responsibilities

  • Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
  • Work on Databricks infrastructure, including workspace configuration, clusters, compute resources, access controls, and performance optimization.
  • Apply Data Governance principles, including data quality, data lineage, metadata management, data cataloging, and access controls.
  • Work with Unity Catalog to manage data permissions, governance policies, and data assets.
  • Develop and manage Delta Lake tables using Medallion Architecture (Bronze, Silver, and Gold).
  • Integrate data from multiple sources, including databases, Amazon S3, files, and REST APIs.
  • Create, schedule, monitor, and optimize Databricks Jobs and Workflows.
  • Implement data validation, error handling, logging, monitoring, and data quality checks.
  • Optimize Spark workloads, SQL queries, and data processing pipelines for performance and scalability.
  • Support data discovery, metadata documentation, and governance standards across data platforms.
  • Collaborate with cross-functional teams to ensure data accuracy, consistency, security, and compliance.
  • Troubleshoot data pipeline and infrastructure issues and recommend improvements to platform reliability.
  • Contribute to enterprise data management and governance best practices.

Requirements

  • 7+ years of overall experience in Data Engineering or a related data technology role.
  • Minimum 5 years of hands-on Databricks experience, including data engineering and/or Databricks infrastructure.
  • Mandatory understanding of Data Governance concepts, including data quality, metadata management, data lineage, data cataloging, data ownership, and access controls.
  • Practical knowledge of data governance principles and their application in data platforms.
  • Understanding of data governance frameworks, policies, standards, and data management best practices.
  • Experience or knowledge of implementing data quality checks, data lineage tracking, metadata documentation, and governed data access.
  • Strong expertise in Python, PySpark, and Advanced SQL.
  • Hands-on experience with the Databricks Lakehouse Platform, Delta Lake, Unity Catalog, Workflows, Jobs, Clusters, and Notebooks.
  • Experience developing ETL/ELT pipelines, data transformations, batch processing, and incremental data loading.
  • Good understanding of data warehousing and data modeling concepts.
  • Knowledge of data security, access management, and governance practices within data platforms.
  • Experience with Spark performance tuning, partitioning, file optimization, and query optimization.
  • Familiarity with Git, CI/CD practices, and production deployment processes.
  • Strong analytical, problem-solving, and communication skills.

We offer

  • A competitive salary alongside a strong insurance package.
  • Extensive learning and development resources.

Hiring process

  • Apply via the provided link.