Role Summary

Own quality across the POD through automation. Both cover functional, API and integration testing.

Key Responsibilities

• Build and maintain automated test suites across the Python and JVM services and React UIs.
• Own API, contract and integration testing for the Orchestrator, Workflow Builder, Connectors and MCP / A2A surfaces.
• Integrate tests into CI/CD gates; report coverage and quality signals.
• AI-eval focus: design and run evaluation harnesses for the Signal / Action Engines and Librarian Agent — accuracy, hallucination, regression and guardrail checks for LLM/agent outputs.
• Performance & data focus: build load and resilience tests for orchestration and event driven flows, and data-quality tests for pipelines.

Must-have Skills & Technologies

• Test automation with Pytest / Playwright / Selenium and a JVM framework (REST Assured, JUnit) for API and UI testing.
• API and contract testing (Postman, Pact) and CI integration.
• One of: LLM/agent evaluation frameworks and techniques (for the AI-eval role) or performance testing (k6, JMeter, Locust) and data-quality validation (for the performance/data role).

Prior Experience

• 3–6 years as an SDET / automation-focused QA engineer.
• Shipped automation running in CI for microservices; comfortable reading application code.

Nice to Have

• Experience testing event-driven / workflow systems (Temporal, Kafka / Event Hub).
• Familiarity with AI/LLM quality practices or data observability.