FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W2346401102> ?p ?o ?g. }

• W2346401102 abstract "Standard setting of assessment questions has matured as a psychometric practice since the 1980s1 and is now a common practice in Medical Education.2 This is done to ensure that a fair, defensible cutoff score or pass mark is reached for each assessment paper.3 Standard setting is thus a policy decision as standards are not a natural phenomenon waiting to be discovered,4  neither does the process signify scientific precision. Nevertheless, in order to be defensible standard-setting must be procedurally credible.At the College of Medicine and Health Sciences (CMHS), United Arab Emirates (UAE) University, the Modified Angoff5 procedure of standard setting has been adopted as applicable to all high stakes and end course MCQ examinations since 2014. This test-centered, criterion referenced method of standard setting is one of the most studied and widely used procedures for high stakes examinations.4At the CMHS, following a period of question vetting, the procedure was initially conducted in face-to-face meetings of appropriate faculty members with subject expertise. In this exercise judges make independent estimates of the proportion of minimally competent candidates that would be expected to answer each item correctly. Where there is a large variation in estimates, judges then discussed discrepancies and revised estimates. Depending on length of discussion and necessity to refine previously vetted questions, the procedure often took an average of three to six hours to standard set an examination of 100 MCQs. In the case of high stakes examinations, the procedure could take several times that amount of time.Although this process had its benefits in further vetting of questions, useful discussion and feedback between faculty members about their understanding of the task, issues surrounding the minimally competent candidate, etc., it was also critiqued by faculty as arduous and time consuming for busy medical subject matter experts. Availability of new technologies have however facilitated evolution of methods and processes of test administration and setting cut scores to meet challenges posed by real and perceived inadequacies of existing processes.6,7The purpose of this paper is to describe how we implemented an online standard setting procedure which significantly reduces the amount of time needed to standard set examinations.ImplementationWe set out to automate the process of standard setting at CMHS. Relying on an online, secure Assessment Management System (AMS)8 where the questions are stored safely by the Medical Education Department, vetted and standard set by faculty, and delivered securely to students. The system is developed in-house using a secure internal website running ASP and MS SQL Server. The system involves two types of users, judges who are faculty members able to make a judgment of the appropriate cutoff mark for each question, and coordinators who are in charge of courses and associated examinations.The system was used at CMHS as a pilot for several course final examinations and is now used for all less high-stakes examinations with judges making judgments using their office computers while entering comments on questions that warrant discussion. It is also used for high-stakes examinations in face-to-face meetings with judges using handheld devices to access the questions, have any needed discussion, and then enter their judgments in their handheld devices. The resulting cutoff average, standard deviation, and histogram are displayed on the handheld device and an overhead projector and modification of cutoffs can be performed if necessary.The judges can read the question clearly, make a judgment on the percent of minimally competent students who should answer the question correctly, enter that percent easily, and add a comment about the question if they wish. Judges can be assigned, removed or emailed by the coordinator. Completion percent and average cutoff for each judge are clearly shown and comments can be displayed.  Questions that have a cutoff standard deviation among judges of more than 20% are marked so that they may be reviewed in a short meeting along with the questions and comments from judges. Each question that has been standard set display the average cutoff, standard deviation and number of judges along with the histogram of cutoffs and all judges’ comments on that question.  The assessment analysis shows, question by question, percentage of options chosen, point biserials for correct options, and the difference between students' correct option percentage and judges' cutoffs. This provides an indication regarding the accuracy of judges' cutoff estimations.We feel that we have uncovered some novel ideas in Medical Education standard setting which warrant further research. For example, what is the appropriate cutoff standard deviation to identify questions that need review (we use 20% as rule of thumb)? And, what classifies a reasonable judgment in comparison to students’ correct response (we use judges' average within ±20 of students' correct response as another rule of thumb)?Finally, similar to other researchers, we are confident that in time online standard setting methods will proliferate, but believe this will only happen when these new ideas are addressed effectively. Furthermore, as in the case of most standard setting research, we cannot state with confidence that the procedure described here is widely generalizable. We can nevertheless, attest to the overwhelming positive feedback received from standard setters in our institution and recommend that other institutions faced with similar constraints necessitating consideration of an alternative approach, at least pilot online standard setting." @default.
• W2346401102 created "2016-06-24" @default.
• W2346401102 creator A5010925018 @default.
• W2346401102 creator A5064125299 @default.
• W2346401102 creator A5081463512 @default.
• W2346401102 date "2016-05-05" @default.
• W2346401102 modified "2023-10-01" @default.
• W2346401102 title "Online assessment standard setting for multiple choice questions" @default.
• W2346401102 cites W1965933703 @default.
• W2346401102 cites W2100173888 @default.
• W2346401102 cites W2141967493 @default.
• W2346401102 doi "https://doi.org/10.5116/ijme.5715.3481" @default.
• W2346401102 hasPubMedCentralId "https://www.ncbi.nlm.nih.gov/pmc/articles/4860288" @default.
• W2346401102 hasPubMedId "https://pubmed.ncbi.nlm.nih.gov/27149419" @default.
• W2346401102 hasPublicationYear "2016" @default.
• W2346401102 type Work @default.
• W2346401102 sameAs 2346401102 @default.
• W2346401102 citedByCount "1" @default.
• W2346401102 countsByYear W23464011022019 @default.
• W2346401102 crossrefType "journal-article" @default.
• W2346401102 hasAuthorship W2346401102A5010925018 @default.
• W2346401102 hasAuthorship W2346401102A5064125299 @default.
• W2346401102 hasAuthorship W2346401102A5081463512 @default.
• W2346401102 hasBestOaLocation W23464011021 @default.
• W2346401102 hasConcept C105795698 @default.
• W2346401102 hasConcept C151730666 @default.
• W2346401102 hasConcept C15744967 @default.
• W2346401102 hasConcept C176730311 @default.
• W2346401102 hasConcept C177264268 @default.
• W2346401102 hasConcept C17744445 @default.
• W2346401102 hasConcept C199360897 @default.
• W2346401102 hasConcept C199539241 @default.
• W2346401102 hasConcept C2776548248 @default.
• W2346401102 hasConcept C2777230681 @default.
• W2346401102 hasConcept C2777267654 @default.
• W2346401102 hasConcept C3018023364 @default.
• W2346401102 hasConcept C33923547 @default.
• W2346401102 hasConcept C38652104 @default.
• W2346401102 hasConcept C41008148 @default.
• W2346401102 hasConcept C509550671 @default.
• W2346401102 hasConcept C71924100 @default.
• W2346401102 hasConcept C86803240 @default.
• W2346401102 hasConceptScore W2346401102C105795698 @default.
• W2346401102 hasConceptScore W2346401102C151730666 @default.
• W2346401102 hasConceptScore W2346401102C15744967 @default.
• W2346401102 hasConceptScore W2346401102C176730311 @default.
• W2346401102 hasConceptScore W2346401102C177264268 @default.
• W2346401102 hasConceptScore W2346401102C17744445 @default.
• W2346401102 hasConceptScore W2346401102C199360897 @default.
• W2346401102 hasConceptScore W2346401102C199539241 @default.
• W2346401102 hasConceptScore W2346401102C2776548248 @default.
• W2346401102 hasConceptScore W2346401102C2777230681 @default.
• W2346401102 hasConceptScore W2346401102C2777267654 @default.
• W2346401102 hasConceptScore W2346401102C3018023364 @default.
• W2346401102 hasConceptScore W2346401102C33923547 @default.
• W2346401102 hasConceptScore W2346401102C38652104 @default.
• W2346401102 hasConceptScore W2346401102C41008148 @default.
• W2346401102 hasConceptScore W2346401102C509550671 @default.
• W2346401102 hasConceptScore W2346401102C71924100 @default.
• W2346401102 hasConceptScore W2346401102C86803240 @default.
• W2346401102 hasLocation W23464011021 @default.
• W2346401102 hasLocation W23464011022 @default.
• W2346401102 hasLocation W23464011023 @default.
• W2346401102 hasLocation W23464011024 @default.
• W2346401102 hasOpenAccess W2346401102 @default.
• W2346401102 hasPrimaryLocation W23464011021 @default.
• W2346401102 hasRelatedWork W2080338264 @default.
• W2346401102 hasRelatedWork W220088780 @default.
• W2346401102 hasRelatedWork W2412384794 @default.
• W2346401102 hasRelatedWork W2744140594 @default.
• W2346401102 hasRelatedWork W2748952813 @default.
• W2346401102 hasRelatedWork W2899084033 @default.
• W2346401102 hasRelatedWork W2911362709 @default.
• W2346401102 hasRelatedWork W3087422525 @default.
• W2346401102 hasRelatedWork W3112308582 @default.
• W2346401102 hasRelatedWork W6044103 @default.
• W2346401102 isParatext "false" @default.
• W2346401102 isRetracted "false" @default.
• W2346401102 magId "2346401102" @default.
• W2346401102 workType "article" @default.