Staff Software Engineer - OpenTelemetry Python instrumentation

Описание

Responsibilities

• You will bring your software engineering expertise to develop improvements to the OpenTelemetry Python SDK and instrumentation libraries, with a strong focus on AI applications and agent workflows.
• Collaborate in open-source communities.
• You will develop and maintain the OpenTelemetry Python SDK and instrumentation libraries, working with maintainers and other vendors on specifications, APIs, and implementation decisions.
• Build Python instrumentation for AI agents.
• Build instrumentation that connects agent runs, model calls, and tool execution, including across asynchronous code.
• Contribute to evolving OpenTelemetry conventions for generative AI, with care for sensitive data and metric cardinality.
• Improve Python SDK performance.
• Use profiling and benchmarks to evaluate implementation approaches with the upstream community.
• Make sound tradeoffs around runtime overhead, compatibility, and maintainability.
• Be visible as an OpenTelemetry expert.
• Teach others, promote best practices around OpenTelemetry, semantic conventions, and various technologies and frameworks.
• Both within Grafana Labs and in the broader OpenTelemetry community.
• Support the technical direction and vision of the team, and contribute to business conversations on the OpenTelemetry strategy at Grafana Labs.
• Own projects from an ambiguous problem through design, implementation, testing, documentation, and release.
• Partner in an open and respectful way.
• Lend your knowledge and skills to our GTM, Product and Support teams.
• Join customer conversations, investigate instrumentation issues, and turn what you learn into fixes and practical guidance.
• As we are remote-first and our engineering organization is largely remote, we provide guidance and meet regularly using video calls, so high agency and good communication skills are a must.

Requirements

• 8+ years of experience in software development, with strong proficiency in Python and experience debugging production software, language internals, libraries, and asynchronous code.
• An interest in AI applications and agent systems, and curiosity about how to instrument and troubleshoot them.
• Experience working in open-source communities.
• You are eager to become an active member of the OpenTelemetry community, collaborate with maintainers and users, and keep learning across languages, frameworks, and the wider observability ecosystem.
• Familiarity with observability tooling such as Grafana, Prometheus, and OpenTelemetry, alongside experience with cloud-based software or systems engineering.
• You understand production environments and can assess operational risks.

Nice to have

• You have experience working on open source observability projects, ideally in OpenTelemetry or Prometheus.
• Experience in Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation
• Experience in a hands-on technical leadership role, setting technical direction and leading project teams
• Experience instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI
• Experience contributing to or maintaining Open Source projects, with evidence of successful pull requests and community collaboration
• Rust experience is a plus for exploring potential Python SDK performance improvements, including Rust-backed implementations.

We offer

• In the UK, the compensation range for this role is GBP 107,000 - GBP 134,000.
• All of our roles include Restricted Stock Units (RSUs), giving every team member ownership in Grafana Labs' success.
• 100% Remote, Global Culture
• Scaling Organization
• Transparent Communication
• Innovation-Driven
• Open Source Roots
• Empowered Teams
• Career Growth Pathways
• Approachable Leadership
• Passionate People
• In-Person onboarding
• Balance is Key - We operate a global annual leave policy of 30 days per annum.

Hiring process

• Grafana Labs may utilize AI tools in its recruitment process to assist in matching information provided in CVs to job postings.
• The recruitment team will continue to review inbound CVs manually to identify alignment with current openings.