The AI Quality Analyst, AI & Innovation safeguards the quality, reliability, and measurable performance of the team’s AI data extraction processes, which convert commercial risk insurance documents — such as policies, endorsements, submissions, schedules of values, and loss runs — into structured data. Rather than building the extraction systems, this role exists to support the organization’s Enterprise AI Evaluation Strategies — the standards, methods, and processes used to determine whether AI systems are accurate, reliable, and ready for use. The role spans both the foundation and the practice of evaluation: building and curating the ground-truth (“golden”) datasets against which AI performance is measured, and contributing directly to the evaluation processes those datasets power. Reporting to the Director, AI & Innovation, the AI Quality Analyst creates, curates, and maintains ground-truth files, helps design and run evaluations, reviews AI extraction outputs against ground truth, and delivers clear, data-driven feedback that the development team uses to advance the extraction process.

A central part of the role is engaging directly with business users: training them on the ground-truth application, guiding them through the process of producing and reviewing ground-truth data, and collecting the associated source files and validated outputs. The AI Quality Analyst maintains a comprehensive, working understanding of the extraction processes to define what “correct” looks like and to safeguard extraction quality end to end. The role is well suited to someone who brings a working knowledge of commercial risk insurance, is familiar with AI/ML and large language model concepts, and is skilled at quality review, data stewardship, and stakeholder engagement — without requiring hands-on AI/ML development experience.

Key Responsibilities
•    Own, create, and maintain the curated ground-truth (“golden”) datasets and ground-truth labels that developers rely on to measure, benchmark, and improve AI extraction quality — the foundation of the team’s evaluation work.
•    Contribute to the design, execution, and continuous improvement of the organization’s Enterprise AI Evaluation Strategies, making sure ground-truth data is built to serve the evaluations it supports rather than simply collected and stored.
•    Help design evaluation plans and test sets for AI extraction use cases, selecting representative and edge-case samples across commercial risk insurance document types, lines of business, and levels of complexity.
•    Participate in evaluation cycles — baseline, regression, and pre-release testing — whenever models, prompts, or processes change, and help define the acceptance thresholds that determine whether changes are ready for production.
•    Help calibrate automated evaluation methods, such as LLM-as-judge scoring, against human-reviewed ground truth to confirm those methods can be trusted at scale.
•    Contribute to enterprise evaluation standards, templates, and playbooks so evaluation practices are consistent and reusable across AI initiatives beyond data extraction.
•    Define, document, and uphold clear standards and annotation guidelines for what constitutes a correct extraction, so quality can be measured consistently and reproducibly.
•    Apply commercial risk insurance knowledge to define what a correct extraction looks like for key policy and risk data — such as named insureds, lines of coverage, limits, deductibles and retentions, premiums, exclusions, and loss history — and to resolve ambiguous or complex cases.
•    Engage with and train business users on the ground-truth application, guide them through creating and reviewing ground-truth data, and collect the associated source files and validated outputs needed to build and expand the ground-truth library.
•    Review AI extraction outputs against ground truth, identify and categorize discrepancies and error types, and deliver actionable diagnostics and feedback to the team advancing the extraction process.
•    Coordinate human-in-the-loop review and annotation workflows, including inter-annotator agreement, to keep ground-truth data accurate, consistent, and trustworthy.
•    Manage the versioning and lifecycle of ground-truth datasets so quality results remain reproducible and comparable as models, prompts, and processes change.
•    Partner with developers to define and interpret quality metrics and acceptance criteria — accuracy, precision, recall, F1, completeness, groundedness/faithfulness, hallucination rate, and schema conformance — and help translate results into clear findings.
•    Monitor quality trends across extraction runs to surface drift, degradation, and anomalies before they reach downstream products, escalating issues to the team.
•    Maintain a comprehensive, working understanding of the existing AI data extraction processes the team leads, partnering closely with the Manager and team in support of their continuous improvement.
•    Support responsible-AI review of AI outputs, including robustness, bias, safety, and compliance considerations at the output level.
•    Define and maintain quality metrics, dashboards, and reporting that give the team and stakeholders clear visibility into extraction performance and ground-truth coverage.
•    Document ground-truth methodology, metric definitions, annotation standards, and quality practices to establish and mature the team’s AI quality practice.
•    Collaborate with the Analytics Engineering function, which consumes validated AI-extracted data into Gold-layer assets, and the Applied AI Analyst, which evaluates AI from a business and product perspective, to align quality standards with downstream needs.

•    2–5+ years of experience in data quality, quality assurance, data analysis, annotation/labeling, or a related analytical or business-facing role, ideally with exposure to AI/ML, LLM, or data-extraction systems. Hands-on AI/ML or software development experience is not required.
•    Experience contributing to structured testing, validation, or evaluation processes, such as QA test plans, user acceptance testing, or data or model validation.
•    Working knowledge of commercial risk insurance — including policy structure, coverage terms, limits, deductibles, exclusions, endorsements, submissions, and loss runs — gained through underwriting, brokerage, claims, risk management, or insurance operations, sufficient to judge whether extracted data is accurate and complete.
•    General familiarity with large language models and generative AI — what they do well, where they fall short, and why they can produce inaccurate or fabricated outputs — sufficient to understand, review, and reason about AI outputs.
•    Strong analytical rigor and exceptional attention to detail, with the ability to assess extraction accuracy and translate findings into clear, actionable recommendations.
•    Excellent communication and stakeholder-engagement skills, with the ability to train and support business users and to build effective ground-truth workflows with non-technical partners.
•    Comfort working with data and data tools.

Preferred Qualifications
•    Bachelor’s degree in Computer Science, Information Systems, Mathematics, or a related field.
•    Familiarity with technical generative-AI concepts such as prompt engineering, retrieval-augmented generation (RAG), and LLM-as-judge evaluation.
•    Understanding of evaluation methodology and metrics for AI/ML and information extraction, such as precision, recall, F1 score, accuracy, and hallucination detection.
•    Experience building, curating, or maintaining labeled or ground-truth datasets and annotation workflows, including inter-annotator agreement.
•    Insurance industry designations such as CPCU, AINS, or ARM.
•    Basic proficiency in SQL for querying evaluation and production data.
•    Basic scripting or Python familiarity; the role is centered on quality review and data stewardship rather than software development.
•    Familiarity with AI/LLM observability and evaluation tooling such as LangSmith.
•    Familiarity with the Databricks platform and tools such as MLflow and Unity Catalog.

Success Measures
•    Coverage, accuracy, and freshness of the ground-truth datasets, and how well they support the evaluations they are built for.
•    Meaningful contribution to Enterprise AI Evaluation Strategies, including reliable evaluation cycles, clear acceptance criteria, and adoption of evaluation standards across AI initiatives.
•    The degree to which developers can quickly and confidently measure and validate extraction quality using the ground-truth files provided.
•    Adoption and proficiency of business users on the ground-truth application, and the volume and quality of source files and validated outputs collected.
•    Early detection of quality drift, degradation, and anomalies before they impact downstream products.
•    Clarity and usefulness of the quality metrics, dashboards, diagnostics, and feedback provided to the team and stakeholders.

#LI-JM