Senior Software Engineer - Analytics Data Platform

Описание

The Analytics Data Platform Batch team at Datadog powers batch data processing across internal data infrastructure, executing over 800k Spark jobs daily for internal teams such as billing, cost management, security intelligence, and applied AI.

Responsibilities

• Build AI-driven version migration automation for Spark and JVM
• Build agentic tooling that analyzes compatibility, generates PRs, and orchestrates multi-datacenter testing
• Focus on engine-agnostic batch execution
• Ensure platform reliability and scale
• Implement Spark Pod Autosizing for CPU and memory allocations
• Understand tradeoffs across execution models

Requirements

• You have 6+ years of experience
• You have significant backend programming experience in one or more languages (Go, Java, Rust, Python, Scala)
• You have knowledge of Spark internals/infrastructure including executor lifecycle, memory model, dynamic allocation, and resource tuning
• You have experience building platform or infrastructure that other engineers depend on
• You are a distributed systems engineer with experience building high performance large scale systems
• You have knowledge of Kubernetes fundamentals
• You have excellent communication skills (English)
• You have high proactivity and cross-team collaboration experience
• You are excited about leveraging AI tools to enhance how you code, solve problems, and build – or eager to learn how
• You have demonstrated ability to use AI coding tools in day-to-day workflows and validate, critique, and refine AI-generated output

Nice to have

• AI/LLM integration in engineering workflows
• Observability and cost attribution

We offer

• New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
• Continuous professional development, product training, and career pathing
• Intradepartmental mentor and buddy program for in-house networking
• An inclusive company culture, ability to join our Community Guilds (Datadog employee resource groups)
• Access to Inclusion Talks, our Internal panel discussions
• Free, global mental health benefits for employees and dependents age 6+
• Competitive global benefits