Senior Data Engineer (Ab Initio, AWS, PySpark)

Role overview:

As a Senior Data Engineer, you will be technically mentoring Data Engineers and will play a thought leadership role in our team – the team will look to you for advice on data architectures and business issues facing them.

In this role, you will build data processing platforms which power the Data Science and Core Product teams to answer questions with data, using software engineering best practices, data management fundamentals, and recent advances in distributed systems (Big Data Technologies).

You will work with passionate scientists, business intelligence engineers, software development engineers, product managers and business analysts, to deliver a comprehensive set of stable and performant data architectures to build data stacks complementing software that powers the organization.

You are an expert at designing, implementing, and operating stable, scalable, low cost solutions to flow data from production systems into other systems. The role requires someone who loves data, understands enterprise information systems, has a strong business sense, and can mentor a team to put these skills into action.

The successful candidate will be eager to learn; building highly available, scalable applications using hybrid set of tools and technologies like Spark, Airflow, Ab Initio, EKS, Lambda and Snowflake.

What you’ll be doing:

• Hands-on participation in Engineering tasks such as coding, peer reviews, building POCs
• Lead, influence, and be a contributor across our entire technology team while acting as an area specialist for your team
• Partner closely with other tech leads to design and implement org wise solutions
• Implement all the necessary infrastructures which allow self-healing and observable monitoring of data ingestion, and consumption
• Define, develop and maintain artifacts like technical design or user documentation and look for continuous improvement in software and development process within agile
• Proactive enhancement and support of the Data architecture to minimize technical debt and maximize agility
• Resolve problems and roadblocks as they occur with help from peers. Follow through on details and drive issues to closure
• Communicate and work effectively with geographically distributed multi-functional teams.
• Drive problem and conflict resolution for team
• Share technology takeaways and POCs across engineering community in organization
• Take on technical ownership of a work stream, be the subject matter expert and managing risks and issues
• Performance tuning of code and review of fellow developer’s code to confirm adherence to best practices
• Mentoring of more junior members of the team
• Work with Project Managers and Business Analysts to ensure the timely delivery of project deliverables that meet requirements
• Actively communicating and collaborating with stakeholders across geographies.

What we are looking for:

• Bachelor’s degree in computer science or related technical field or equivalent related professional experience
• 5+ years of proven experience in Big Data / distributed compute projects
• Equivalent years' experience in designing and crafting high volume data processing applications
• Extensive Ab Initio development experience
• Extensive data modelling and SQL experience
• Good to have Python development experience
• Good to have PySpark development experience
• Experience building data pipelines using cloud services (AWS preferable)
• Experience in leading and mentoring junior engineers.
• Good understanding of data modelling, schema design patterns and modern data access patterns (including API, stream, data lake) in cloud (AWS preferable)
• Knowledge around Distributed Systems Architecture and orchestration (Airflow preferable)
• Team uses both PySpark and Ab Initio to build high performance ETL solutions hence the ideal candidate should be open to learn and excel in the complete tech stack.
• Able to justify technology choices to technical and non-technical observers making well-defined choices
• Highly proficient in SQL and knowledgeable about data warehousing concepts.