Senior Data Engineer
FawryГибридот 5 лет
PythonSQLData EngineeringETLSparkApache SparkELTApache AirflowData WarehouseData LakeHiveImpala
Описание
Responsibilities
- Design, develop, and optimize scalable ETL/ELT pipelines for batch and near-real-time data processing
- Build and maintain data ingestion solutions and Data Warehouse / Data Lake architectures
- Develop and optimize data processing workloads
- Manage workflow orchestration using Apache Airflow and troubleshoot production pipelines
- Improve data quality, performance, reliability, security, and operational efficiency
- Perform root-cause analysis, resolve production incidents, and ensure pipeline SLAs
- Collaborate with cross-functional teams, conduct code reviews, and mentor junior engineers
Requirements
- SQL
- Python
- Spark
- PySpark
- Hive
- Impala
- Apache Airflow
- Hadoop
- HDFS
- YARN
- Apache NiFi
- Sqoop
- Git
- CI/CD
- Linux
- Unix