Lead AI QA Engineer

Bengaluru, India

Full-Time

TLDR

Lead design of evaluation harnesses and QA pipelines for production-grade AI systems, implementing gold datasets, LLM-as-judge regimes, and CI-integrated testing.

Join our India Tech Hub – Be among the first hires! Kobie, a 35-year veteran of the loyalty industry, a multi-year Forrester Leader, and USA Top Workplace is expanding its global footprint by establishing a Tech Hub in India. Kobie partners with global brands to build deep connections with their customers through personalized, data-driven loyalty experiences and has a mission of growing enterprise value through loyalty. The Tech Hub will serve as a Global Capabilities Center for a broad range of technology roles, and this is your chance to play a pivotal role in shaping our presence in India. Join us as we continue to lead in loyalty, delivering innovative customer experiences for some of the world’s most recognized brands while working alongside some of the best and brightest in loyalty. How you will make an impact

Design and build evaluation harnesses for agentic systems in Python — golden datasets, LLM-as-judge graders, multi-turn regression suites and trace-based assertions. In addition, develop framework to verify generated AI output.

Author automated test suites for prompts, tools, structured outputs (Pydantic / JSON schema), retrieval pipelines (ETL Experience) and end-to-end agent workflows

Validate guardrails around tool execution: auth scoping, input/output validation, PII and prompt-injection protections, and hallucination mitigation

Wire evaluations into CI using Dataiku Evaluations, GitHub Actions or Jenkins so every change is graded against quality, safety and cost SLOs before it ships

Build observability into testing by instrumenting traces with LangSmith, Langfuse, MLflow or OpenTelemetry and triaging production drift back into the eval harness

Own quality end-to-end — define release criteria, run pre-prod and shadow tests, and partner with engineering to root-cause and fix regressions quickly

Partner with data engineers on Snowflake-backed retrieval testing patterns (Cortex Analyst and Cortex Search Services) and with platform teams on observability, security and cost

Help shape internal QA standards for AI & Data engineering as the stack evolves, contributing to design reviews and sharing knowledge across the India and U.S. teams

Participate in a collaborative DevOps environment, working closely with developers, AI engineers, Data Engineers, DBAs and product partners across environments

In your first 90 days

By the end of your first 90 days, you will have stood up at least one production-grade evaluation harness — golden dataset, LLM-as-judge graders and regression suite — wired into CI for an internal agent. You will have automated trace-based assertions running against staging traffic, a clear quality scorecard for at least one shipped agent, and a clear opinion about what our next testing investment should be.

What you need to be successful

3+ years of professional QA / SDET experience, with production experience automating tests for backend services or data pipelines

1+ years of hands-on experience testing LLM or AI features in production: prompt regression, tool / function-call validation, structured outputs and RAG correctness

Working knowledge of evaluation frameworks such as RAGAS, DeepEval, LangSmith, Langfuse or comparable LLM-as-judge tooling

Strong Python and PyTest skills; solid SQL skills and comfort with at least one cloud platform (AWS, Azure or GCP)

Fluency with Git, Docker, REST APIs and at least one CI tool (GitHub Actions, Jenkins, GitLab CI or CircleCI)

Solid understanding of data security and responsible AI practices, particularly in PCI-compliant or regulated environments

Proven ability to work independently and within a team, managing priorities across concurrent projects and time zones

Strong written and verbal communication skills; able to work effectively with both technical and non-technical stakeholders

A bachelor’s degree is not required — equivalent practical experience (including bootcamps, self-taught work, career changes or non-CS technical degrees) counts

Bonus Skills:

Hands-on experience with Dataiku DSS (Python / SQL recipes, scenarios, code environments, the dataiku and dataikuapi clients) or Dataiku Evaluations

Experience with Dataiku LLM Mesh, Knowledge Banks, Prompt Studio, or Visual / Code Agents

Experience with Snowflake, Snowpark, or Snowflake Cortex (Search, Analyst, Agents)

Experience with red-teaming, prompt-injection testing or adversarial test generation for LLMs

Familiarity with multi-agent patterns: supervisor / router, subagent / handoff, reflection, human-in-the-loop

Experience with performance and load testing tools such as Locust, JMeter or k6

ISTQB, AI Testing or comparable QA certification

Experience in loyalty, martech, adtech or a comparable data-rich B2B domain

About Kobie

Named a Top Workplace in the USA and Top Remote Workplace, Kobie is where the best minds in loyalty come together, driven by passion and innovation. We’re always looking for talented individuals ready to join a collaborative, growth-focused culture.

As a trusted partner to some of the world’s most recognized brands, we are loyalty leaders, helping brands build lasting emotional connections with their consumers.  We do this with a strategy-led technology approach that uncovers the truth behind what drives consumers on an emotional level.As we launch our India Tech Hub, we are excited to bring our award-winning culture to a new region - creating an environment where collaboration, flexibility, and career growth come together to build something truly special.

We are proud to be the only loyalty provider to be externally recognized for their culture. We believe people thrive when they feel valued, supported, and empowered to be their authentic selves. Our commitment to diversity, equity, and inclusion ensures every teammate has a voice and the opportunity to be heard.

Giving back is in our DNA at Kobie,through an annual fundraiser, charitable partnerships, and volunteer opportunities, we encourage our teammates to make a difference in their communities.

To support our teammates beyond just their careers, we offer highly competitive benefits, comprehensive health coverage, and well-being perks that support our teammates and their dependents. We understand the importance of time for life outside of work - recognizing public holidays,offering flexible time off, and prioritizing work-life balance.

As we expand into India, our new teammates will be fully integrated with our U.S. teams, working on global projects and gaining exposure to top industry leaders. With continued growth, we will establish a physical office in Bengaluru, India, giving teammates aspace for collaboration and fostering connection.

Now is the perfect time to join Kobie. Be part of something big and help shape the future of our global capabilities center, the Kobie India Tech Hub.

Benefits

Health Insurance

comprehensive health coverage

Volunteer opportunities

Paid Time Off

prioritizing work-life balance

Wellness Stipend

well-being perks that support our teammates and their dependents

Apply for this job

Kobie Marketing

Kobie Marketing is a loyalty technology provider that partners with global brands to create personalized, data-driven loyalty experiences. By combining strategy-led technology with deep consumer insights, Kobie helps brands forge lasting emotional connections with their customers. With a commitment to innovation and an expanding presence, including a new tech hub in India, Kobie is shaping the future of loyalty solutions.

Founded: Founded 1990
Employees: 201-500 employees
Industry: Professional Services

Enterprise Software Marketing Tech SaaS

View company profile

Quality Assurance (QA) Engineer

Apply for this job