MARGO 4 часа назад
Senior Data Engineer
180 – 200 ₽в час
- Удалённо
- Опыт от 6 лет
PythonSQLData EngineeringAzurePower BITableauApache SparkDatabricksApache AirflowData WarehouseAzure Data FactoryUnity Catalog
Описание
Responsibilities
- Hands-on Data Product development
- Implement scalable, reliable and reusable data pipelines using Azure Data Factory and Apache Airflow.
- Develop data transformations in Azure Databricks (PySpark / SQL) to create/extend data products following the Medallion Architecture layers: Bronze (raw ingestion), Silver (cleansed & conformed), Gold (business-ready data products)
- Optimize performance, cost, and reliability of data workloads.
- Raise important questions during business maturation in order to detect risks before development stage.
- Technical Leadership
- Participate in architecture reviews and technical decision-making.
- Promote engineering excellence through code reviews and design reviews.
- Collaborate with Architects, Product Owners in data product technical maturation.
- Support & Operations (Run Team Collaboration)
- Provide support to the Run/Operations team for production pipelines and data products for complex topics.
- Troubleshoot and resolve data incidents, failures, and performance issues
- Participate in incident management, root cause analysis, and post-mortems.
- Ensure proper handover, documentation, and operational readiness of developed solutions.
- Contribute to improving runbooks, monitoring, and operational processes.
Requirements
- 6+ years of experience in data engineering
- Strong technical expertise with: Python (PySpark), SQL (advanced level)
- Hands-on experience with Azure-based Data Platform including: Databricks, Unity Catalog, Apache Airflow, Azure Data Factory
- Reporting and BI tools (Power BI, Tableau or similar)
- Solid understanding of data modelling fundamentals, main architectures and design patterns: Data Warehouse, Data Lake and Data Lakehouse architectures, Medallion architecture, Metadata-driven architecture, ETL/ELT design patterns
- Understanding of: Partitioning, optimization, and performance, Batch and streaming pipelines
We offer
- 180 – 200 PLN per hour