Senior Big Data Infrastructure Engineer

Описание

The role is a senior big data infrastructure engineering position in a Big Data Platform team, responsible for deploying, upgrading, and operating core services such as HDFS, S3-compatible object storage, and Apache Iceberg. It includes leading an infrastructure pilot, introducing quotas, migrating Iceberg catalogs, and rebuilding monitoring and alerting.

Responsibilities

• Own the big data infrastructure of our international contour and be its technical lead
• Deploy, upgrade and operate the core services of the platform: HDFS, S3-compatible object storage (Apache Ozone), Iceberg and others
• Help run an infrastructure pilot that changes how the platform is used: a small number of very large datasets will become thousands of small tables
• Introduce compute and storage quotas for business units
• Migrate us from the default Iceberg catalog to a more capable one (Lakekeeper or Gravitino)
• Rebuild monitoring and alerting so alerts mean something again
• Work closely with an engineer who runs the same stack in a parallel contour
• Share approaches and solutions with them and cover for each other
• Actively replace outdated tools as our infrastructure runs on the latest versions of everything

Requirements

• 5+ years of experience designing, deploying and operating big data platforms
• Hands-on experience with HDFS / Hadoop: deployment, version upgrades, troubleshooting
• Hands-on experience deploying and operating S3-compatible object storage (Apache Ozone is ideal, but MinIO or similar solutions also count)
• Experience with Apache Iceberg
• Experience setting up monitoring and alerting for data infrastructure
• English B2
• Ability to work in the GMT+3 time zone (up to 3 hours difference)

Nice to have

• Experience with Trino, Kafka or Flink
• Experience with Iceberg REST catalogs (Lakekeeper, Gravitino, Polaris or similar)
• Experience setting up resource quotas and cost allocation in a multi-tenant data platform
• Hands-on use of AI tools, such as Copilot or agents, to speed up your work

We offer

• Flexible schedules and opportunity to work remotely
• Ambitious and supportive team who love what they do, appreciate each other, and grow together
• A structured onboarding, a personal development plan, and regular internal lectures and demos from the team
• Internal programs for adaptation and training, development of soft skills, and leadership abilities
• Partial compensation for participating in external training and conferences
• Corporate English school: Group and individual lessons, speaking clubs with colleagues from all over the world
• Corporate prices on hotels and travel services
• MyTime Day Off - an extra non-working day without loss of compensation