Are you a trainer and/or designer on a THEIA platform and want to find out more about Script Concordance Tests? In this article, we’ll look at what a SCT is, what its specific features are, and how to create them step by step on your platform.
1. What is a Script Concordance Test (SCT)?
Script Concordance Tests, or SCTs, are assessment tools developed as part of healthcare training. The French National Authority for Health gives the following definition: [SCTs are] “standardised tools used to assess the clinical reasoning of the professional carrying them out, and in particular their ability to make relevant decisions in complex cases or in situations of “uncertainty”, by comparing them with the opinions of a panel of experts”. The SCT is therefore designed to assess students’ decision-making when faced with a case on which there is no consensus. The teaching objective is not to check that they choose the one and only right option, but rather to assess their ability to mobilise a network of knowledge to respond to an uncertain situation.
To find out more about the definition and objectives of SCTs as an assessment modality, you can consult the document “Script Concordance Test” from the Haute Autorité de Santé (2017).
SCTs are based on a short vignette presenting a case to be studied. This case is followed by several hypotheses which provide additional information. The students are asked to judge the relevance of each hypothesis in the light of the new information provided, using a Likert scale (the values of which are defined by the designer). The students’ answers are compared with those of “experts”, professionals to whom the case has also been put and who have also chosen an option.
Here is an example:
In this case, the student will have to judge the relevance of the proposed treatment (including a C3G) in the light of the fact that a mixed formula has been found. Their choice will then be compared with that of the experts who answered before them.
2. Framework for the use of SCT
SCTs can be used when we do not wish to assess learners’ pure knowledge, but rather their ability to activate networks of acquired knowledge (called “scripts”) in order to solve a task. The advantage of this is that it provides a more accurate representation of potential professional situations and is therefore able to assess professional skills in a more relevant way.
Although SCTs were born in the context of medical school examinations, they can nevertheless be of just as much interest in other disciplines. They make sense when you want to assess students’ decision-making in the context of a situation they may encounter in their future professional environment. For example, SCTs could be designed to assess managerial situations, marketing decisions or the choice of technical options when developing a product.
3. SCTs in THEIA
In the THEIA platform, SCTs are therefore folders that comply with the following rules:
- They behave in the same way as a FF (Free Folder): all the questions are displayed as soon as you enter the exam, the order in which the answers are given is free and it is possible to modify the answers recorded (as long as the copy is not submitted).
- The introduction to the file presents the case to be studied.
- They can only contain models of specific questions linked to the SCT format (medical investigation, prognosis, diagnosis and therapy questions) – See “6. Adding and configuring SCT questions”.
- The scoring system is specific to this format and is based on a comparison between the answer given by the student and the answers given by experts – See “4. The scoring system”.
4. The scoring system
The specific pedagogical features of SCT require a special assessment system which is only used for this type of dossier. In THEIA, corrections are automatic (without manual correction) and marks are calculated instantly using the following formula:
Score obtained for each question = Number of experts who chose the same option as the student / Maximum number of experts who chose the same option.
Let’s take the example below:
In this example, the student chose the “Most probable” option, considering that the information “Have observed a vagal syndrome” had an impact on the hypothesis of a fracture of the right orbital floor with muscle incarceration. Of the 12 experts who also responded to this case, 10 agreed with him. For this question, there is no answer for which more experts agree. 10 is therefore the reference base for scoring.
As the student answered as they did, he was awarded a mark of 10/10, the maximum possible. As the question is out of 1 point, he therefore gets the full point.
If he had chosen the “Almost certain” option, he would only have obtained 2/10 (because only two experts agreed on this point) and therefore 0.2 points. Any other answer would have earned him 0.
5. Create a SCT folder
To create a SCT in THEIA, go to the question catalogue, under “Evaluation tests”. Here you will find a list of all the types of file available on your platform. We will therefore select “SCT”.
Please note: if SCT folders do not appear in your question catalogue, it is possible that this option is not active on your platform. If this is the case, please contact THEIA support.
The platform asks you to enter a name for your question folder, then you enter the folder settings. Here you will find the classic tabs for setting up content on THEIA: the summary and preview tabs allow you to view the content that already exists in the folder, and the ‘Parameters’ tab allows you to define the folder’s visibility options (for other platform designers, for example). The ‘Content’ tab allows you to create questions.
6. Add and configure SCT questions
As explained above, the SCTs propose types of questions specific to the pedagogical challenge of this type of folder. The THEIA platforms offer five types of question:
- The generic question
- The medical diagnostic question
- The investigative medical question
- The medical therapeutic question
- The medical prognostic question
For each question, you can choose a name and then confirm. This will allow you to enter its parameters and enter its content.
These five types of question are all based on the same model: they present a hypothesis, new information, and a sentence asking the student to judge the impact of this new information on the hypothesis. There is then a customisable Likert scale to help the student (and the experts) position themselves.
The construction of the five types of questions is the same; only the pre-loaded content of the questions changes. This allows you to choose the form in which you want to pose the case to your students and experts. For example, the therapeutic question will focus on treatments and their effects. Its hypothesis takes the form “If you are thinking of prescribing such and such a drug…”, then the new information, and the statement becomes “This new information makes the proposed treatment/treatment…”. Note that this is for guidance only: each question statement can be modified to suit your needs. The generic question also allows you to design questions without specific pre-filled text.
7. Have the experts vote
Once you have created your questions, you need to mobilise the experts to create the reference base for grading the students. There are two ways of doing this.
7.1. Invite them to respond on the platform
If you would like your experts to respond directly on THEIA, you can invite them via the “Expert panel” tab in your SCT folder. Here you will find a list of all the users of your platform who have SCT expert rights. You can select them and add them to your file.
To find out more about assigning rights to users, see the following article: “Rights profiles”.
Once this has been done, the users concerned will see a special interface dedicated to SCTs in their THEIA platform: the “SCT Evaluation” area. Still in your expert management tab, you can send them email notifications (which include a direct link to the “SCT Evaluation” area) to inform them that a response is expected from them.
You can monitor the responses received in the table summarising the experts registered in the file and decide, depending on the information you have (for example if the average difference with the other respondents is too high) not to take responses into account when calculating the scores. To do this, simply uncheck the option in the “Use copy” column.
7.2. Managing fictitious experts
It may happen that the experts you need to mobilise do not have a THEIA account, or a means of accessing the content on their side. In this case, you can indicate their answers on the platform so that they can be taken into account in the final grading of the students. To do this, go to the questions concerned and indicate for each proposal how many experts have chosen it, as ‘fictitious’ experts.
Once you have done this, you need to tell the platform that the evaluation method is no longer based on the real respondents but on the answers recorded by the fictitious experts. To do this, go back to the “Expert panel” tab and tick the “Use manual correction as default correction” option. This will activate your manual answers for scoring the dossier. Note that this can also be changed in post-processing.
Please note: at present, the two voting methods cannot be combined. If you enter answers from fictitious experts, you must activate the calculation mode by fictitious expert to take them into account (and direct answers will no longer be counted).
Once your questions have been added and set up, and your experts have submitted their answers, your file is ready for use. It can now be chosen as exam content, transferred to a local bank or used as the subject of a pre-defined training session.
