Automation Test Analyst - Senior Manager

About Vialto Labs (VLabs) 

Vialto Labs (VLabs) is responsible for redesigning how work is delivered in the tax and immigration service lines, as well as driving operational efficiency across Vialto’s functional areas using AI. The team builds and deploys novel AI-enabled solutions that directly improve productivity and increase delivery quality for our clients. VLabs is accountable for rapidly turning innovative experiments into production-ready deliverables at scale and embedding them into day-to-day operations. This team focuses on the highest-impact workflows, creating standardized, repeatable capabilities that can be deployed globally. Operating with a mandate for speed and measurable outcomes, VLabs works alongside service line, product, and platform leaders. 

About the Role 

The Automation Test Analyst is a hands-on role responsible for building and scaling automation capabilities within the VLabs delivery lifecycle. This role operates within sprint cycles, embedding automation into development rather than executing post-build validation. The role focuses on ensuring the reliability, stability, and production readiness of both traditional and AI-enabled platform components. Given the presence of AI-generated outputs, this role requires designing resilient validation approaches based on structure, confidence, and range rather than static assertions. 

Working closely with AI Test Engineering, product, and engineering teams, this role contributes to continuous testing, release readiness, and scalable quality assurance aligned to VLabs’ speed and delivery expectations. 

Key Responsibilities 

Test Automation Development 

• Build and maintain Playwright-based UI regression suites covering core platform workflows including case creation, document processing, workflow interaction, and deliverable generation 

• Develop API contract tests using Postman/Newman for document pipelines, classification outputs, extraction results, and system integrations 

• Create Python and SQL-based data validation scripts to validate field mapping, data integrity, and transformation logic 

AI-Aware Validation 

• Design validation approaches for AI-generated outputs using structural checks, field presence, value ranges, and confidence indicators 

• Partner with AI Test Engineering to identify and automate applicable AI output validations 

• Ensure test design accounts for variability in AI outputs while maintaining reliability and meaningful validation 

Sprint-Embedded Automation 

• Build automation incrementally within sprint cycles, prioritizing API validation, followed by data validation and UI regression 

• Ensure regression suites are continuously runnable and provide actionable insights for each release cycle 

• Support release readiness by contributing to automated validation within CI/CD pipelines where applicable 

Data & Test Environment Management 

• Maintain test data pipelines including parameterized datasets, database seeding, and volume test preparation 

• Ensure test environments are stable, repeatable, and aligned with automation requirements 

Integration & Quality Assurance 

• Partner with engineering teams to ensure testability, observability, and reliability of platform components 

• Validate end-to-end workflows including AI-integrated components and multi-step processes 

• Support performance testing efforts through JMeter contributions and result interpretation 

Standardization & Scaling 

• Contribute to reusable automation frameworks and standardized testing patterns across VLabs 

• Design automation with sufficient abstraction to localize failures and reduce maintenance overhead 

• Support continuous improvement of automation coverage and efficiency 

Governance & Reporting 

• Integrate automated test results into ADO Test Plans with traceability to test cases and release milestones 

• Provide outputs that support milestone sign-off and production readiness decisions 

• Ensure adherence to quality, security, and governance standards in testing practices 

Qualifications & Experience 

Professional Experience 

• 4+ years in test automation as a primary discipline 

• Demonstrated experience working with AI-enabled systems, including GenAI, LLMs, or agentic applications 

• Proven ability to build automation frameworks in active, sprint-based development environments 

• Experience designing validation strategies for variable or non-deterministic system outputs 

• Strong practical judgment on where to prioritize automation investment for maximum impact 

Technical Expertise 

Playwright: 

• Development of scalable UI automation frameworks using page object models 

• Handling dynamic, asynchronous, and AI-populated interfaces 

• Headless execution for CI/CD and debugging workflows 

Postman / Newman: 

• API contract testing with parameterized, data-driven scenarios 

• Multi-step workflow validation and CI/CD integration 

• Python: 

• Data validation scripting, JSON/XML parsing, and automation utilities 

• Development of reusable validation and testing components 

SQL:

• Data validation, persistence checks, and mapping verification 

JMeter: 

• Basic performance testing support and result interpretation 

ADO:

• Test case management, automation integration, and defect tracking 

Operating Capabilities 

• Ability to operate independently in fast-paced, iterative environments 

• Strong problem-solving and analytical skills with attention to detail 

• Ability to balance speed and rigor in automation development 

• Proactive mindset in identifying automation opportunities and improving test coverage 

• Strong collaboration with engineering and AI teams 

Education 

• Bachelor’s degree required 

• Advanced degree in Computer Science, Data Science, or related field preferred