Director, Psychometric Operations & Data Science

Director, Psychometric Operations & Data Science  

College Board - Global Strategy and Talent, College Board LATAM 

Location: Ideal- San Juan, Puerto Rico. This is a hybrid role and requires the candidate to be on-site at College Board offices every Tuesday and Wednesday and as per business needs. 

The search may be extended to include candidates from Continental USA .  

Role Type: This is a full-time position. 

About the Team 

College Board LATAM is committed to expanding educational opportunity and advancing evidence-based decision-making across the region. As part of the Research & Development division, the Research & Measurement team ensures that our assessments are scientifically rigorous, operationally sound, and aligned with our mission to expand educational opportunities across the region. 

We work at the intersection of psychometrics, data science, and applied research, guiding how our assessments are designed, scored, analyzed, and continuously improved. Our team ensures that research and measurement practices are transparent, defensible, reproducible, and grounded in robust methodological standards. 

R&M collaborates closely with internal teams — including Test Design & Development, Technology, Assessment Operations, Scoring, Reporting, and Market Development — as well as external stakeholders such as institutions, ministries, and research partners to support high-quality assessment programs across Latin America. As we modernize our assessment workflows and strengthen our data capabilities, we are seeking a hands-on director who can lead psychometric operations, improve assessment data workflows, and ensure that scoring and reporting processes are accurate, efficient, reproducible, and scalable. 

About the Opportunity 

The Director, Psychometric Data Operations will lead the operational execution and continuous improvement of psychometric data workflows for College Board LATAM assessments. This role sits at the intersection of assessment operations, psychometrics, data science, and technology. 

In this role, you will manage scoring cycles, improve psychometric data workflows, strengthen quality assurance processes, and help modernize item analysis, calibration, scoring, and reporting data processes. You will be responsible for ensuring that psychometric operations are accurate, reproducible, well-documented, and delivered on schedule. 

This is a highly hands-on role. You will work directly with data, analyses, validation routines, and scoring files while also influencing how work is structured across Research & Measurement, Technology, Scoring, and Reporting. You will partner with psychometricians on technical direction for scaling, equating, calibration, and form processes, while owning the operational execution, documentation, QA, and workflow reliability needed to support those processes. 

You will also help identify opportunities to reduce manual work, improve automation, and responsibly apply AI-enabled tools to strengthen QA, anomaly detection, documentation, and operational efficiency. This role does not own enterprise data governance or technology infrastructure, but it will help define operational requirements and promote strong data practices within R&M and across cross-functional assessment workflows. 

Key Responsibilities 

Psychometric Operations — 55% 

• Manage recurring scoring cycles, ensuring that scoring, analysis, validation, and reporting readiness activities meet defined timelines, accuracy expectations, and QA standards.  

• Execute and oversee psychometric data workflows related to item analysis, item calibration, scoring validation, equating support, reporting file readiness, and post-administration review. 

• Partner with psychometricians to operationalize technical guidance for scaling, equating, calibration, form rotation, and scoring rules.  

• Run, review, and improve statistical and data validation checks using tools such as SQL, R, Python, or similar platforms.  

• Conduct root-cause analysis when scoring irregularities, data discrepancies, or reporting issues arise.  

• Maintain strong documentation, version control, traceability, and reproducibility standards across scoring and psychometric workflows.  

• Translate psychometric requirements into clear operational steps, data specifications, QA checks, and handoffs across teams.  

• Ensure that scoring and psychometric processes are defensible, repeatable, and aligned with relevant professional expectations for educational assessment.  

Data Workflow Modernization & Automation — 30% 

• Identify and reduce manual, file-based, or error-prone processes across scoring, item analysis, calibration, validation, and reporting workflows.  

• Define business and operational requirements for improved psychometric data workflows, automated checks, dashboards, and pipeline enhancements.  

• Partner with Technology to implement data workflow improvements, while ensuring that technical solutions meet the operational needs of R&M, Scoring, and Reporting.  

• Develop or support scripts, queries, validation routines, and repeatable processes that improve speed, consistency, and transparency.  

• Establish practical standards for data inputs, outputs, file formats, naming conventions, documentation, and handoffs.  

Cross-Functional Leadership, Documentation & Data Practices — 15% 

• Represent R&M in cross-functional conversations related to scoring data, reporting data, psychometric workflows, and assessment data operations.  

• Promote consistent documentation, data definitions, process maps, and decision logs across psychometric and assessment data workflows.  

• Apply existing College Board data policies and responsible data practices within R&M workflows.  

• Help define access, retention, documentation, and data-use expectations for psychometric and operational assessment data, in partnership with appropriate governance and Technology partners.  

• Communicate complex operational, statistical, and data quality issues clearly to technical and non-technical audiences.  

• Mentor colleagues in reproducible data practices, workflow documentation, QA routines, and responsible use of automation.  

• Contribute to a culture of accuracy, accountability, continuous improvement, and thoughtful innovation.  

About You 

You are a hands-on assessment data operations leader who is comfortable working at the intersection of education, psychology, psychometrics, and data science. You may not be a career psychometrician, but you understand assessment data, statistical workflows, and the importance of accuracy, reproducibility, and defensibility in high-stakes testing environments. 

You are able to work directly with data and code while also leading cross-functional improvements. You enjoy solving operational problems, improving workflows, reducing manual effort, and making complex processes more reliable. 

You have: 

• Master’s degree in Data Science, Statistics, Information Systems, Educational Measurement, Psychology, Psychometrics, Computer Science, or a related field. 

• 5+ years of hands-on experience in psychometrics, educational measurement, advanced statistical analysis, or similarly rigorous quantitative environments, with strong technical fluency in SQL and at least one programming language such as R or Python.  

• Experience supporting or managing scoring cycles, assessment data workflows, statistical analysis workflows, or reporting data processes.  

• Strong technical fluency in SQL and at least one programming language such as R or Python. 

• Experience running data validation checks, documenting workflows, troubleshooting data discrepancies, and improving repeatability of analytic processes.  

• Working knowledge of educational assessment concepts such as item analysis, classical test theory, item response theory, scaling, equating, calibration, scoring rules, or form development.  

• Ability to partner effectively with psychometricians, researchers, technology teams, scoring teams, and reporting teams.  

• Strong attention to detail and a track record of supporting processes where accuracy, documentation, and quality control are essential.  

• Ability to communicate clearly with both technical and non-technical stakeholders.  

• Bilingual proficiency in Spanish and English.  

• Ability to travel 1–2 times a year to College Board offices or on behalf of College Board business.  

Preferred qualifications: 

• Experience with assessment programs, educational testing, credentialing, certification, admissions, large-scale assessment, or similar testing environments.  

• Experience improving or automating data workflows using R, Python, SQL, or related tools.  

• Familiarity with structured and semi-structured data formats, such as CSV, relational tables, JSON, or similar formats.  

• Familiarity with cloud-based data environments or modern data platforms.  

• Comfort using AI-enabled tools to improve QA, anomaly detection, documentation, workflow automation, or analytic review.  

• Experience working with Technology teams to define requirements for data tools, pipelines, dashboards, or workflow improvements.  

This position primarily involves sedentary work performed at a computer. The role requires the ability to sit or stand for extended periods, communicate effectively, and operate standard office equipment. 

All roles at College Board require: 

• A passion for expanding educational and career opportunities and mission-driven work.  

• Curiosity and enthusiasm for emerging technologies, with a willingness to experiment with and adopt new AI-driven solutions and comfort with learning and applying new digital tools independently and proactively.  

• Clear and concise communication skills, written and verbal.  

• A learner’s mindset and a commitment to growth: welcoming diverse perspectives, giving and receiving timely, respectful feedback, and continuously improving through iterative learning and user input.  

• A drive for impact and excellence: solving complex problems, making data-informed decisions, prioritizing what matters most, and continuously improving through learning, user input, and external benchmarking.  

• A collaborative and empathetic approach: working across differences, fostering trust, and contributing to a culture of shared success.  

• Resident in USA, Authorization to work in the United States for any employer.  

About Our Process 

Application review will begin immediately and will continue until the position is filled. This role is expected to accept applications for a minimum of 5 business days. 

While the hiring process may vary, it generally includes resume and application submission, recruiter phone/video screen, hiring manager interview, a performance exercise such as a data analysis or workflow review exercise, a panel interview, a conversation with leadership, and reference checks. 

What We Offer 

At College Board, we offer more than a paycheck — we provide a meaningful career, a supportive team, and a comprehensive package designed to help you thrive. We’re a self-sustaining nonprofit that believes in fair and competitive compensation grounded in your qualifications, experience, impact, and the market. 

A Thoughtful Approach to Compensation 

• The salary range for a new employee in this role based in Puerto Rico is $64,000 - $80,000  

• The salary range for a new employee in this role based in Continental USA is $80,000–$125,00 

Your exact salary will depend on your location, experience, and how your background compares to others in similar roles at the College Board. 

We aim to make our best offer upfront, rooted in fairness, transparency, and market data. 

We adjust salaries by location to ensure fairness, no matter where you live.