How Exam Pass Marks Are Set

General Information

How Exam Pass Marks Are Set

Recently, the SOA Board of Governors and the CAS Board of Governors each held discussions on pass mark setting for multiple-choice exams. Each Board directed its respective Education and Examination Committees to issue a public statement describing content–based pass marks with the intent of enhancing candidates' understanding of how pass marks are set for multiple–choice exams.

Why Content Based Pass Marks?

The goal of the SOA and CAS Education and Examination system is to pass all candidates who have demonstrated adequate knowledge of the syllabus material and to fail those candidates who have not. The objective of the examinations is to evaluate candidate performance using criteria for demonstrating adequate knowledge that remain constant throughout the lifetime of the exam series. Pre–set pass marks (e.g., a candidate will pass if he/she answers x% of the questions correctly) are counter to this philosophy. The examinations are meant to measure the candidate's level of achievement of the required learning objectives, and their required level of capability of accomplishing specified learning outcomes.

Multiple–Choice Pass Mark Setting

Exam P/1 is administered and scored according to computer–based testing methodologies. For the other multiple–choice examinations, a modified Angoff passing score study is performed. This is a common testing and measurement technique where a panel of experts in the subject material reviews the examination. Each expert is asked to review each question in the examination, and assess the difficulty of that question. More specifically, they are asked to estimate the likelihood that a candidate with minimum adequate knowledge competency would answer the question correctly. The sum of these probabilities, averaged across the panel of experts, gives a preliminary estimate of the pass mark.

The estimated pass mark resulting from the modified Angoff passing score study is compared to and balanced with the actual performance statistics on the examination in finalizing the pass mark. The effects of any particularly difficult questions are also factored into the determination of the final pass mark.

SOA Written–Answer Pass Mark Setting

For all SOA written–answer examinations, the assessment process is different. Before the exam is administered, the exam committee sets a minimum adequate knowledge point value for each question. Minimum adequate knowledge is what the well–prepared candidate should know and be able to demonstrate in order to achieve a passing grade on a question.

After the exam is administered and first grading of all questions has taken place, a tentative pass mark is set where actual performance statistics are balanced against minimum adequate knowledge. Approximately one–third to one–half of the candidates–those with scores fairly near the expected pass mark–will have their written–answer papers re–graded at a central grading session. The papers of the other candidates will not be re–graded, since their scores would not change sufficiently to move from pass to fail or vice versa. When central grading has been completed, the actual performance statistics are recalculated to factor in score changes resulting from second grading.

The examination committee will then determine the final pass mark by again balancing actual performance statistics against minimum adequate knowledge while taking into account other factors such as time pressure situations that may have occurred on some questions. The effects of any particularly difficult questions are also factored in when determining the final pass mark.

Who Decides the Final Pass Marks?

With the use of content–based pass marks, fluctuation in pass rates from session to session is expected. The final decision is reached by consultation between the Chairperson and Vice–Chairpersons of the individual examination committee and Officers of the E&E committee and any co–administrators. In the event that a proposed pass mark would produce a pass rate that deviates significantly from pass rates experienced in prior exam sessions, the deviation must be supported by analysis and an explanation of the basis for the pass mark must be reported to the SOA Board of Governors. For any jointly sponsored examinations, the CAS Executive Committee would also receive such a report.

These procedures are somewhat different, however, for EA–1, EA–2, A and EA–2, B, which are jointly administered by ASPPA, the Joint Board for the Enrollment of Actuaries, and the SOA.