Senior Analyst
KPMG is currently seeking an AI Agent Testing Specialist to join our Legal, Regulatory and Compliance organization.
•Execute structured test plans aligned with AI Agent Building Code and acceptance criteria.
•Validate knowledge grounding to ensure responses use approved, current sources only.
•Ensure adherence to instructions, guardrails, boundaries, and escalation protocols.
•Test output quality for accuracy, completeness, consistency, and hallucination risks.
•Conduct security checks including prompt injection, jailbreak, and data exposure testing.
•Validate agent workflows, including source selection, fallback, and ambiguity handling.
•Maintain comprehensive testing evidence, including results, defects, and sign-offs.
•Support regression testing following changes to prompts, models, tools, or configurations.
•Identify recurring issues and escalate defects to developers and stakeholders.
•Contribute to continuous improvement via monitoring, reusable test assets, and collaboration.
Mandatory technical & functional skills
•Bachelor’s degree in business, technology, data analytics, computer science, engineering, risk management, or related field
•1–3 years of experience in QA, testing, risk, data analytics, business analysis, or AI/automation support
•Experience executing test scripts, documenting results, identifying defects, and tracking remediation
•Strong attention to detail to assess accuracy, completeness, and alignment of AI outputs
•Ability to follow structured testing protocols and identify edge cases or control gaps
•Strong written communication skills for documenting test evidence, defects, and observations
Preferred technical & functional skills
•Familiarity with AI tools, generative AI, LLM workflows, chatbots, or agent-based solutions
•Experience working with global (U.S.-based) stakeholders and managing multiple priorities
•Understanding of risk management concepts, control frameworks, or professional services environments
•Hands-on experience with Microsoft 365, SharePoint, Excel, PowerPoint, Teams, Copilot, or similar tools
•Awareness of data privacy, confidentiality, access control, and responsible AI principles
•Ability to analyze AI outputs critically for grounding, consistency, and reliability