QA Engineer

Our Customer:

A technology company developing an AI-based application for enterprise users. The project focuses on delivering accurate, reliable, secure, and user-friendly AI-powered functionality.

You will join a cross-functional team and work closely with developers, AI engineers, product owners, and business stakeholders to ensure the quality of AI features throughout the product lifecycle.

Your Tasks:

• Analyze project requirements, user stories, and AI-related business scenarios.
• Create and execute test cases and test scenarios for AI features and workflows.
• Perform functional, system, integration, regression, and end-to-end testing.
• Validate AI-generated responses for accuracy, relevance, completeness, clarity, and consistency.
• Test different types of user inputs, including valid, invalid, incomplete, and ambiguous queries.
• Identify hallucinations, incorrect or inconsistent responses, bias, and gaps in AI understanding.
• Prepare prompt-based test scenarios and test different prompt variations.
• Verify that AI follows defined instructions, guardrails, business rules, and expected tone.
• Validate AI responses against approved knowledge sources and expected outcomes.
• Test integrations between AI components, APIs, databases, knowledge bases, and third-party or enterprise systems.
• Verify authentication, authorization, role-based access, and data correctness.
• Perform regression testing after changes to prompts, models, APIs, or business logic.
• Maintain reusable regression test suites.
• Test AI behavior in edge cases and unexpected user scenarios.
• Log defects with clear reproduction steps, test data, expected and actual results, and severity.
• Track defects using Jira, Azure DevOps, or similar tools.
• Work closely with developers, AI engineers, product owners, and business stakeholders to resolve issues.
• Provide testing results and release sign-off input.

Required Experience and Skills:

• 3+ years of commercial experience in software testing / QA.
• Strong understanding of functional, regression, system, integration, and end-to-end testing.
• Experience creating test cases, test scenarios, and test data.
• Experience testing AI-based applications, chatbots, APIs, enterprise applications, or similar solutions.
• Experience validating AI-generated outputs for correctness, relevance, and quality.
• Understanding of AI concepts such as prompts, LLM responses, hallucinations, context, and knowledge retrieval.
• Experience with defect tracking tools such as Jira, Azure DevOps, or similar.
• Good analytical and problem-solving skills.
• Strong attention to detail and ability to identify inconsistencies in AI-generated responses.
• Good communication skills and ability to work effectively with cross-functional teams.

Would be a Plus:

• Hands-on experience testing Generative AI, NLP, ML, chatbot, or conversational AI solutions.
• Experience testing RAG-based or knowledge-based AI applications.
• Basic understanding of prompt engineering and LLM behavior.
• Experience with API testing tools such as Postman or Swagger.
• Experience with automation testing tools such as Selenium, Playwright, or similar.
• Knowledge of Azure OpenAI, Azure AI, Microsoft Copilot Studio, Power Platform, or similar AI platforms.
• Understanding of prompt injection, data leakage, access control, and responsible AI testing.
• Experience testing AI integrations with databases, knowledge bases, or third-party services.

Відгукнутись на вакансію