logo[мetahunt]
> DOU
middle

QA Engineer

Windmill Digital
company:Outsource
manual testingtest planningtest automationplaywrightcypressselenium
api testingui testing
experience3+ years
locationKraków, Poland
> full description

Role Overview

We are looking for a QA-origin engineer with a strong background in manual testing and a growing or established focus on AI evaluation, verification, and validation. This role is dedicated to ensuring the correctness, reliability, safety, and consistency of AI-driven and automation-heavy systems. The ideal candidate comes from a traditional QA background, understands system behavior deeply, and is either experienced in or actively transitioning toward modern AI evaluation, QA Vibe, and next-generation automation technologies.

Key Responsibilities

  • Design and execute evaluation strategies for AI-powered features (LLM-based flows, recommendations, decision logic).
  • Validate AI outputs for correctness, consistency, bias, hallucination risk, and edge cases.
  • Define qualitative and quantitative AI evaluation criteria, metrics, and acceptance thresholds.
  • Perform deep manual, exploratory, and scenario-based testing on AI and non-AI features.
  • Build and maintain automated evaluation and regression pipelines for AI-enabled systems.
  • Collaborate with engineers and product teams to improve AI testability and observability.
  • Review and verify fixes, prevent regressions, and ensure enterprise-grade quality before releases.

Required Background

  • Strong QA foundation with hands-on manual testing experience.
  • 3–5+ years in QA / software testing, including complex or enterprise systems.
  • Experience defining test strategies, test plans, and acceptance criteria.
  • Understanding of AI system behavior, non-determinism, and evaluation challenges.
  • Exposure to test automation frameworks (UI and/or API).
  • Strong analytical skills and attention to detail.

Preferred

  • Experience with AI evaluation frameworks, prompt testing, or model validatio Familiarity with Playwright, Cypress, Selenium, or similar tools.
  • Interest in agentic QA, AI-assisted testing, and QA automation evolution.

We offer

  • Flexible and fast-paced environment where the product comes first
  • Competitive salary
  • Challenges and personal growth
  • Flexible working practices
  • Friendly environment
  • Training program allowance
  • Team-building activities
  • Respectful and inclusive environment
Відгукнутись на вакансію