Thoughtworks 3 дня назад

Lead Data Engineer

  • Bangalore, India
  • Гибрид
  • Опыт от 10 лет
SQLData EngineeringNoSQLData ModelingApache SparkData QualityBig DataData GovernanceData SecurityData Privacy

Описание

Responsibilities

  • You will lead and manage data engineering projects from inception to completion, including goal-setting, scope definition and ensuring on-time delivery with cross team collaboration.
  • You will collaborate with stakeholders to understand their strategic objectives and identify opportunities to leverage data and data quality.
  • You will design, develop and operate modern data architecture approaches to meet key business objectives and provide end-to-end data solutions.
  • You will be responsible to create, design and develop intricate data processing pipelines, addressing clients' most challenging problems.
  • You will collaborate with data scientists to design scalable implementations of their models.
  • You write clean and iterative code based on TDD and leverage various continuous delivery practices to deploy, support and operate data pipelines.
  • You will lead and advise clients on how to use different distributed storage and computing technologies from the plethora of options available.
  • You will develop data models by selecting from a variety of modeling techniques and implementing the chosen data model using the appropriate technology stack.
  • You will be responsible for data governance, data security and data privacy to support business and compliance requirements.
  • You will define the strategy for and incorporate data quality into your day-to-day work.

Requirements

  • You have 10+ years of experience in leading the system design and implementation of technical solutions.
  • Working with data excites you; You have created Big Data architecture, can build and operate data pipelines, and maintain data storage, all within distributed systems.
  • You have a deep understanding of data modelling and experience with modern data engineering tools and platforms.
  • You have experience in writing clean, high-quality code using Pyspark programming language.
  • You have built and deployed large-scale data pipelines and data-centric applications using any of the distributed storage platforms and distributed processing platforms in a production setting.
  • You have experience with data visualisation techniques and can communicate the insights as per the audience.
  • You have experience with data-driven approaches and can apply data security and privacy strategy to solve business problems.
  • You have experience with different types of databases (i.e.: SQL, NoSQL, data lake, data schemas, etc.).
  • You understand the importance of stakeholder management and can easily liaise between clients and other key stakeholders throughout projects, ensuring buy-in and gaining trust along the way.
  • You are resilient in ambiguous situations and can adapt your role to approach challenges from multiple perspectives.
  • You don’t shy away from risks or conflicts, instead you take them on and skillfully manage them.
  • You coach, mentor and motivate others and you aspire to influence teammates to take positive action and accountability for their work.
  • You enjoy influencing others and always advocate for technical excellence while being open to change when needed.
  • You are a proven leader with a track record of encouraging teammates in their professional development and relationships.
  • Cultivating strong partnerships comes naturally to you; You understand the importance of relationship building and how it can bring new opportunities to our business.

We offer

  • Learning & Development
  • There is no one-size-fits-all career path at Thoughtworks: however you want to develop your career is entirely up to you.
  • But we also balance autonomy with the strength of our cultivation culture.
  • This means your career is supported by interactive tools, numerous development programs and teammates who want to help you grow.
  • We see value in helping each other be our best and that extends to empowering our employees in their career journeys.

Hiring process

  • Responsible Use of AI in Recruitment
  • At Thoughtworks, we use AI tools to support our recruitment team with administrative tasks such as drafting communications, scheduling interviews and writing job descriptions.
  • Crucially, our AI tools do not screen, assess, rank or make hiring decisions.
  • Every application is reviewed by our team and all selection decisions are made exclusively by our interviewers and hiring managers.
  • We are committed to fairness and responsible AI.
  • We actively manage our AI systems by testing, monitoring for biased outcomes and implementing mitigation measures.
  • We hold our third-party vendors to these same high standards through a rigorous governance process.
  • For additional information, please see our full Thoughtworks AI Policy for Recruitment.

Other things to know

  • About Thoughtworks
  • Thoughtworks is a dynamic and inclusive community of bright and supportive colleagues who are revolutionizing tech.
  • As a leading technology consultancy, we’re pushing boundaries through our purposeful and impactful work.
  • For 30+ years, we’ve delivered extraordinary impact together with our clients by helping them solve complex business problems with technology as the differentiator.
  • Bring your brilliant expertise and commitment for continuous learning to Thoughtworks.
  • Together, let's be extraordinary.
  • #LI-Hybrid