Cohere Health 6 дней назад

Lead Data Engineer

  • Hyderabad, Telangana, India
  • Опыт от 8 лет
PythonSQLAWSData EngineeringSparkDBTApache SparkApache Iceberg

Описание

Responsibilities

  • Lead delivery of complex data engineering projects across multiple teams
  • Drive technical execution from design through production deployment
  • Establish scalable implementation patterns
  • Design and implement batch and near-real-time pipelines
  • Build reusable ingestion, transformation, validation, and publishing frameworks
  • Support modernization of legacy workloads
  • Lead Apache Iceberg implementation and optimization
  • Define standards for schema evolution, partitioning, compaction, and metadata management
  • Ensure efficient storage and query performance
  • Implement data quality frameworks
  • Drive observability and monitoring practices
  • Improve operational excellence and reliability
  • Review architecture and design proposals
  • Conduct code reviews and engineering reviews
  • Mentor engineers across levels, providing feedback, support, and growth opportunities aligned with career progression frameworks
  • Collaborate with Staff Engineers and Managers to define and evolve platform architecture, technical strategy, and system reliability goals
  • Act as a technical liaison between data platform engineering, analytics, and product teams—translating requirements into scalable, maintainable solutions
  • Good knowledge of Information practices.
  • Assist the manager in all the information security activities implementation and maintenance process.
  • Ensuring the team and imparted with Competence related to Information security
  • Responsible for implementation of security policies and procedures and report any issues to the Information Security Manager.

Requirements

  • 8–12 years of Data Engineering experience.
  • Healthcare claims and authorization datasets.(Insurance)
  • Experience leading enterprise-scale data initiatives
  • AWS S3, Glue, Athena, Lake Formation
  • IAM and security controls
  • Apache Iceberg (preferred)
  • Delta Lake / Hudi
  • dbt
  • Airflow or Step Functions
  • Python
  • SQL
  • PySpark / Spark

Nice to have

  • Kafka / AWS MSK
  • SageMaker
  • Atlan or similar catalogs
  • Multi-tenant data platforms
  • Data contracts and Data Mesh concepts
  • DBT

We offer

  • Ability to commute/relocate: Hyderabad, Telangana: Reliably commute or planning to relocate before starting work (Preferred)

Hiring process

  • Connect with Talent Acquisition
  • Meet with the Hiring Manager
  • Behavioral Interview(s)
  • Case Study
  • Interview with Senior Leadership