Unison Group 7 дней назад
Data Engineer - Data Lake
- Singapore, Singapore, Singapore
PythonSQLJavaKafkaETLSparkELTHadoopHiveYarnCloudera Data PlatformOozie
Описание
Responsibilities
- Develop and deliver scalable transformation frameworks across Hadoop platforms by building robust data pipelines, ingestion frameworks, and automation capabilities.
- Ensure efficient data processing, performance optimization, and reliable integration of enterprise data sources, while supporting data warehouse, enterprise analytics and downstream consumption requirements.
- This role offers an opportunity to work on large-scale data platforms handling structured, semi-structured, and streaming data across critical banking domains.
Requirements
- Design, develop, and maintain scalable data pipelines within the Enterprise Data Lake platform using Cloudera CDP, Hadoop, Spark, Hive, Java, Python, and SQL.
- Develop high-performance batch and near real-time data ingestion frameworks for onboarding enterprise source systems into the Data Lake.
- Develop complex transformation logic using Spark, HiveQL, SQL, and Java-based processing frameworks.
- Design and implement data ingestion patterns supporting structured, semi-structured, and unstructured datasets.
- Build and maintain batch and near real-time data integration solutions.
- Implement source-to-target mappings and ingestion specifications.
- Develop and support solutions within the Cloudera Data Platform ecosystem.
- Build and maintain data pipelines leveraging:
- Hadoop HDFS
- Hive
- Impala
- Spark
- Kafka
- YARN, Oozie
- Perform deployment, troubleshooting, and performance tuning of applications running on the Hadoop ecosystem.
- Partner with platform teams to ensure optimal cluster utilization and workload management.
- Ensure adherence to Data Lake onboarding standards and governance requirements.
- Investigate and resolve data quality issues and production incidents.
- Ensure compliance with enterprise data governance standards.
- Integrate data engineering solutions into CI/CD pipelines using Git, Jenkins, Azure DevOps, or equivalent tools.
- Automate deployment, testing, monitoring, and operational support activities.
- Develop reusable frameworks and utilities to improve engineering productivity and platform scalability.
- Provide L3 support for Data Lake application.
- Perform root cause analysis for production incidents and implement permanent fixes.
- Monitor pipeline execution, cluster health, and workload performance.
- Participate in release implementation, deployment validation, and post-production support activities.
- Strong hands on experience with Hadoop ecosystem and data pipeline orchestration.
- Hadoop/HDFS
- Hive
- Spark
- Unix/Linux scripting
- Shell scripting
- ETL/ELT development experience.
- Data modelling fundamentals.
- Performance tuning and optimization.
- Experience in Data Warehousing and Data Lake environments.
- Strong Java / Python competency with experiences.
- Strong SQL development skills.