Unison Group 7 дней назад

Data Engineer - Data Lake

  • Singapore, Singapore, Singapore
PythonSQLJavaKafkaETLSparkELTHadoopHiveYarnCloudera Data PlatformOozie

Описание

Responsibilities

  • Develop and deliver scalable transformation frameworks across Hadoop platforms by building robust data pipelines, ingestion frameworks, and automation capabilities.
  • Ensure efficient data processing, performance optimization, and reliable integration of enterprise data sources, while supporting data warehouse, enterprise analytics and downstream consumption requirements.
  • This role offers an opportunity to work on large-scale data platforms handling structured, semi-structured, and streaming data across critical banking domains.

Requirements

  • Design, develop, and maintain scalable data pipelines within the Enterprise Data Lake platform using Cloudera CDP, Hadoop, Spark, Hive, Java, Python, and SQL.
  • Develop high-performance batch and near real-time data ingestion frameworks for onboarding enterprise source systems into the Data Lake.
  • Develop complex transformation logic using Spark, HiveQL, SQL, and Java-based processing frameworks.
  • Design and implement data ingestion patterns supporting structured, semi-structured, and unstructured datasets.
  • Build and maintain batch and near real-time data integration solutions.
  • Implement source-to-target mappings and ingestion specifications.
  • Develop and support solutions within the Cloudera Data Platform ecosystem.
  • Build and maintain data pipelines leveraging:
  • Hadoop HDFS
  • Hive
  • Impala
  • Spark
  • Kafka
  • YARN, Oozie
  • Perform deployment, troubleshooting, and performance tuning of applications running on the Hadoop ecosystem.
  • Partner with platform teams to ensure optimal cluster utilization and workload management.
  • Ensure adherence to Data Lake onboarding standards and governance requirements.
  • Investigate and resolve data quality issues and production incidents.
  • Ensure compliance with enterprise data governance standards.
  • Integrate data engineering solutions into CI/CD pipelines using Git, Jenkins, Azure DevOps, or equivalent tools.
  • Automate deployment, testing, monitoring, and operational support activities.
  • Develop reusable frameworks and utilities to improve engineering productivity and platform scalability.
  • Provide L3 support for Data Lake application.
  • Perform root cause analysis for production incidents and implement permanent fixes.
  • Monitor pipeline execution, cluster health, and workload performance.
  • Participate in release implementation, deployment validation, and post-production support activities.
  • Strong hands on experience with Hadoop ecosystem and data pipeline orchestration.
  • Hadoop/HDFS
  • Hive
  • Spark
  • Unix/Linux scripting
  • Shell scripting
  • ETL/ELT development experience.
  • Data modelling fundamentals.
  • Performance tuning and optimization.
  • Experience in Data Warehousing and Data Lake environments.
  • Strong Java / Python competency with experiences.
  • Strong SQL development skills.

Nice to have

We offer

Hiring process