Job summary

AI testing Engineer Experience: 8+ years Remote (Canada) Video Testing experience with AWS environment We are seeking an AI Testing Engineer to validate and assure quality of AI/ML systems powering real-time contextual advertising on live video streams. The role involves testing multi-modal AI outputs (video + text), ensuring accuracy of detected moments, and automating validation workflows. This role is critical for native moments validation and system-level QA in a real-time streaming environment. Key Responsibilities Validate AI-generated outputs against labelled datasets and expected out…

Job details

AI testing Engineer Experience: 8+ years Remote (Canada) Video Testing experience with AWS environment We are seeking an AI Testing Engineer to validate and assure quality of AI/ML systems powering real-time contextual advertising on live video streams. The role involves testing multi-modal AI outputs (video + text), ensuring accuracy of detected moments, and automating validation workflows. This role is critical for native moments validation and system-level QA in a real-time streaming environment. Key Responsibilities Validate AI-generated outputs against labelled datasets and expected outcomes Perform content validation for native moments detection across video streams Design and implement automated testing frameworks (Python-based) Build validation harness for: JSON parsing and comparison Structured output validation using defined schemas Conduct multi-modal validation (video, audio, transcript outputs) Execute system integration and end-to-end testing Track accuracy metrics, defect logging, and root cause analysis Automate regression and validation pipelines across datasets Collaborate with Applied Scientists to validate LLM and GenAI outputs Work within AWS environment using services such as: S3 (data storage).Lambda (processing). CloudWatch (monitoring) Required Skills & Experience: Strong Python scripting for automation and testing. Experience in AI/ML model validation and QA workflows. Hands-on with: JSON parsing, schema comparison Data validation pipelines Experience in testing: Multi-modal AI systems (vision + NLP + audio) LLM outputs and generative AI pipelines. Good understanding of: Structured output validation AI accuracy evaluation techniques Familiarity with AWS cloud environment (S3, Lambda, logging/monitoring) Experience with media/video content analysis workflows preferred Ability to handle mixed manual + automated testing approaches We are an Equal Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.

What you’ll do

The AI Testing Engineer will validate AI-generated outputs against labeled datasets and perform content validation for native moments detection across video streams. This role is critical for ensuring the quality of AI/ML systems in a real-time streaming environment.

Requirements

Candidates should have strong Python scripting skills for automation and testing, along with experience in AI/ML model validation and QA workflows. Familiarity with AWS services and multi-modal AI systems is also required.

Listed skills

Évaluation · Preferred
Validation · Preferred
Automated testing · Preferred
analysis · Preferred
Organization · Preferred
Data Validation · Preferred
Accuracy · Preferred
Root Cause Analysis · Preferred
Amazon Web Services · Preferred
Time · Preferred
Integration · Preferred
Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

Python Scripting
AI/ML Model Validation
QA Workflows
JSON Parsing
Schema Comparison
Data Validation Pipelines
Multi-modal AI Systems
LLM Outputs
Generative AI Pipelines
Structured Output Validation
AI Accuracy Evaluation
AWS Cloud Environment
Media/Video Content Analysis
Automated Testing Frameworks
Regression Pipelines
Content Validation