Assessment Data Analyst

Title: Assessment Data Analyst

Department: Health Professions Education Centre, RCSI University of Medicine and Health Sciences

Tenure: Full time Permanent

Reporting to: Director of Psychometrics

Location: St Stephens Green, Dublin, Ireland

Closing date: 18th July 2024

Objective of the post:

The successful candidate will become a member of the Health Professions Education Centre, and will support the Director of Psychometrics in providing leadership, policy development, and implementation in the areas of student assessment and evaluation including: quantitative, qualitative and descriptive evaluation instruments; analysis and reporting of statistical and descriptive data for all internal and external evaluative components of the education program.

The role holder is responsible for contributing to the delivery and continuous development of an efficient assessment psychometrics service for the RCSI Schools of Dentistry, Medicine, Pharmacy & Biomolecular Sciences, and Physiotherapy. As part of the small team, you will provide rigorous evidence-informed analysis and expert support for the evaluation and quality assurance of assessment data in both undergraduate programmes. The primary function of the role is to provide core psychometric support, helping the Team to deliver cutting-edge psychometric/statistical input to the curricula and supporting RCSI’s reputational development in contemporary assessment in clinical education.

You will work with academic and professional colleagues from across RCSI to plan, develop, implement and assess all methods of student assessment and evaluation across the programmes to ensure methods deployed are fit-for-purpose and up to date. You will be thoughtful, pragmatic, and creative in your approach, a source of expertise and advice, ensuring an ethical and robust approach to assessment rooted in RCSI’s values and strategic aims. You will support the operationalisation and implementation of assessment strategies, approaches, and practices across the university.

Given the nature of the role, the successful candidate must have exceptional influencing skills and an outstanding focus on assessment to ensure transparent, reliable and valid assessment protocols are provided. First-class stakeholder management and organisational skills are a prerequisite, as is a can-do, enterprising attitude and the tact, resilience and discretion to engage across the range of opinions.

Educated to degree level, with substantial, in-depth experience of assessment services in the higher education or professional postgraduate training sector, the successful candidate must have proven experience of influencing stakeholders to deliver solutions.

Specifically the duties of the post will include:

Applying knowledge of assessment practices and tools to advise academic faculty on the most effective and sustainable way to evaluate students, ensuring methodological soundness and alignment with institutional initiatives

• Conducting routine statistical/psychometric analyses of assessment data following every summative assessment. These analyses may include standard setting, student performance, assessor behaviour, item quality, score reliability and potential variation between demographic subgroups as appropriate.
• Working closely with the RCSI Head of Examinations and Assessment, develop and deliver routine written reports on the above analyses and present such reports at a range of formal meetings.
• Modelling the effect of potential changes to assessments on student progression including developing and running simulations.
• Working closely with the RCSI Head of Examinations and Assessment develop and deliver ad-hoc reports on the Schools’ assessment procedures and their performance as necessary, and especially in response to requests from outside bodies.
• Designing and delivering materials and methods for presenting information to stakeholders such as visiting educational or quality assurance teams and the student body, including answering questions.
• Contributing to staff development and undergraduate teaching by designing and delivering materials as appropriate.
• Contributing to the delivery and development of assessment psychometric services and scholarship in partnership with colleagues across the professional services teams, schools and the Health Professions education research theme as appropriate.
• Depending on service development requirements the role holder may be required to undertake research as directed by their line manager in order to develop an evidence-based approach to the support provided.
• Become recognised at RCSI as a source of expertise in assessment techniques and strategies
• Engage with the wider higher education community to stay up to date with developments in the sector, representing RCSI at external meetings, conferences and venues as appropriate
• Remain abreast of developments in the field of assessment and advise about their testing and adoption at RCSI
• Ensure compliance with legal responsibilities and adhere to University policies and codes of practice in relation to data protection, health and safety, equality and diversity, risk management etc
• Undertake other duties commensurate with the role and as requested by the Director of Psychometrics.

Key Accountabilities:

• Contributes to the delivery and continuous development of an effective and flexible service
• Reviews and monitors processes, procedures and associated documentation drawing in specialist knowledge, implementing or making recommendations for change as appropriate
• Researches, interprets and manipulates data using a variety of systems, analysing and reporting on key findings
• Co-ordinates and contributes to meetings, visits, conferences, committees and facilitates knowledge exchange
• Provides excellent customer service and detailed specialist advice and guidance, within an unpredictable environment, to enable appropriate solutions
• Provides guidance and delivers training on designated specialist activities and associated policies and procedures
• Guides, collaborates and takes a lead as appropriate with colleagues and peers across the University to plan and achieve joint projects and objectives
• Ensures all activities undertaken are in compliance with the University policies
• Acts in a way that demonstrates the University’s commitment to Equality, Diversity and Inclusion for staff, students and partner

Personal competencies associated with this role:

• Consistently meets and exceed targets
• Demonstrates a “can-do” approach to service innovation and delivery
• Strong emphasis on maintaining an atmosphere of mutual trust and open communications
• Adaptable and flexible; possesses a high tolerance for ambiguity and thrives in a fast-paced environment
• Ability to work independently and with others to overcome challenges
• Ability to manage multiple complex tasks without direct supervision
• Strong influencing skills and develops positive collaborations that deliver productive outcomes
• Actively and regularly reflects on own performance to ensure alignment, recognise success and proactively manage under achievement
• Strong oral and written communication skills including public speaking and facilitation skills
• Adhere to strict confidentiality and other ethical standards

Key Relationships

• Health Professions Education Centre
• Examinations and Assessment Unit
• Quality Enhancement Office
• Other relevant University officers (e.g. Heads of Schools, Programme Directors, Operational Team Leaders, Programme Year Leads), professional services teams and other University staff
• The management teams and staff of departments including I.T., Estates & Facilities and other areas
• Student Leaders including Student Unions and Class Reps

Person Specification - Essential

• Educated to level 8 (Batchelor’s) degree level in a quantitative discipline
• Proven experience with a variety of software packages including one or more statistical / programming packages (e.g. R, Python, SPSS, Stata, etc.).
• Practical data analytic experience including (but not limited to):

1. Importing & processing data from multiple formats (e.g. .csv, .xlsx, .txt, etc.)
2. Visualising, summarising & reporting data

• Exceptional attention to detail
• Ability to express technical information clearly to a wide range of people through both written and verbal communication.

Person Specification - Desirable

• Experience within third-level education or postgraduate training (preferably science / health professions education)
• Familiarity with the current scientific evidence related to assessment & evaluation
• Post graduate qualification in a cognate discipline e.g. assessment, teaching and learning or psychometrics / statistics
• Practical experience with assessment-related tasks including (but not limited to):

1. Item / test construction
2. Item / test analysis
3. Setting performance standards

• An understanding of statistical analysis

We are all too aware that imposter syndrome and the confidence gap can sometimes stop fantastic candidates putting themselves forward, so please do submit an application — we’d love to hear from you.

Application Process

Please apply online with your CV and Covering Letter. Informal enquiries can be directed to Rebecca O’Brien at rebeccaobrien@rcsi.ie