Senior Automation QA Engineer
About the Role
We are looking for a
Senior Automation QA Engineer
to own end-to-end test automation across API and UI layers for our AI-native platform, including agent workflow validation.
Key Responsibilities
Design and execute
test strategies for API and UI layers
using
Pytest
and
Playwright
(Angular).
Own
CI/CD quality gates
in
Azure DevOps
, ensuring reliable build promotion and release readiness.
Build and maintain coverage for
agent workflow testing
, including non-determinism, agent graph validation, and confidence score assertions.
Develop reusable automation frameworks and data setup utilities backed by
PostgreSQL
.
Contribute to
LLM evaluation and observability
workflows using
Langfuse
or similar tooling.
Required Qualifications
6+ years
of experience in QA automation.
Strong hands-on expertise with
Pytest
and
Playwright
, with proven depth in both
API and UI automation
.
Solid proficiency in
Python
and experience working with
PostgreSQL
for test data and validation.
Hands-on experience with
Azure DevOps
CI/CD pipelines and quality gates.
Awareness of
LLM/agent testing challenges
(non-determinism, agent graph validation, confidence score assertions); exposure to
Langfuse
or LLM evaluation frameworks is a plus and can be built on the job.
