Weekday AI 19 часов назад

Lead Data Engineer

4 000 000 – 6 000 000 ₽в год
  • Delhi, NCR, Delhi, India
  • Опыт от 6 лет
PythonSQLJavaDockerKafkaAWSAzureAirflowGCPSparkHadoopScala

Описание

Responsibilities

  • Design and implement robust, end-to-end batch data pipelines using tools such as Apache Spark, Hadoop, and Airflow.
  • Develop real-time streaming pipelines with technologies like Apache Kafka, Flink, or Spark Structured Streaming to deliver low-latency analytics.
  • Optimize pipeline performance through partitioning, indexing, and efficient data format selection (e.g., Parquet, ORC).
  • Establish data governance, quality checks, and automated monitoring to ensure data reliability and compliance.
  • Lead capacity planning and scaling strategies for data infrastructure, including cluster management and cost optimization.
  • Mentor and coach junior data engineers, fostering a collaborative and growth-oriented team culture.
  • Collaborate with Data Scientists to translate analytical requirements into scalable pipeline solutions.
  • Drive integration of data pipelines with downstream applications, dashboards, and reporting tools.
  • Stay current with emerging data engineering technologies and evaluate their applicability to business needs.
  • Participate in incident response and root cause analysis to quickly resolve data pipeline failures.

Requirements

  • 6+ years of experience in data engineering with a proven track record in both batch and real-time processing.
  • Deep knowledge of distributed computing frameworks such as Hadoop, Spark, and Flink.
  • Proficiency in message queuing and stream processing systems like Kafka, Pulsar, or RabbitMQ.
  • Strong scripting skills in Python, Scala, or Java, and experience with SQL on large datasets.
  • Hands-on experience with cloud data services (AWS Glue, Redshift, BigQuery, Snowflake, or Azure Data Factory).
  • Familiarity with containerization (Docker, Kubernetes) and CI/CD pipelines for data workloads.
  • Demonstrated ability to architect and scale data pipelines to handle petabyte-scale volumes.
  • Excellent communication skills, capable of translating technical concepts for non-technical stakeholders.
  • A proactive problem-solver with a passion for data quality, governance, and automation.
  • Leadership aptitude, with experience building, mentoring, and growing data engineering teams.

Nice to have

We offer

  • Salary range: Rs 4000000 - Rs 6000000 (ie INR 40-60 LPA)

Hiring process