Middle Data Engineer
Sigma SoftwareWarsaw и ещё 1 городот 2 лет
PythonSQLETLSparkApache SparkDatabricksData LakeDelta LakeAWS S3
Описание
Sigma Software is seeking a middle data engineer to build scalable data pipelines and Data Lakehouse solutions for an AdTech client. The role focuses on batch and near real-time data processing using Databricks, PySpark, SQL, Delta Lake, and AWS S3.
Responsibilities
- Develop and maintain data pipelines for processing large volumes of data in batch and near real-time environments
- Work with Databricks, PySpark, and SQL to transform, validate, and optimize data
- Support implementation of Data Lakehouse solutions following engineering standards and best practices
- Build, test, and automate ETL processes and data workflows
- Improve data quality, organization, and reliability across platforms
- Collaborate with engineers, analysts, and stakeholders to deliver scalable solutions
- Troubleshoot, optimize, and monitor data pipelines
- Stay up to date with modern data engineering technologies
- Ensure deliverables meet quality standards and project requirements
Requirements
- 2+ years of experience in Big Data development and database design
- Strong hands-on experience with SQL
- Strong Python development skills
- Hands-on experience with PySpark
- Experience developing and maintaining data pipelines in Databricks
- Experience with Delta Lake
- Experience with Medallion Architecture (Bronze, Silver, Gold layers)
- Strong analytical and problem-solving skills
- Intermediate or higher English level
Nice to have
- Experience with AWS services, including S3