FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W1977807981> ?p ?o ?g. }

• W1977807981 endingPage "S95" @default.
• W1977807981 startingPage "S93" @default.
• W1977807981 abstract "The objective structured clinical examination (OSCE) has typically been found to be a valid and reliable method for assessing clinical knowledge and skills when evaluating performances of residents. For example, Sloan et al.1 found a 19-problem, 38-station OSCE to be reliable (rxx =.91) and valid in assessing the clinical skills of 56 surgical residents. Often, OSCE performance is summarized in an overall score, which may represent a combination of history, physical examination, interpersonal and communication skills, technical skills, and organization. Interpersonal skills scores are sometimes reported separately because of their importance in overall performance. Warf et al.2 found that when faculty judges evaluated general surgery residents' performances on a neurosurgical station there was no statistically significant difference between the junior and senior residents in performing the neurologic examination. Since general surgery residents do not receive training in neurosurgery beyond their intern year, it was not unexpected that there was no significant difference between levels of training. However, the senior residents were judged to be competent significantly more frequently than were the junior residents. It was also found that interpersonal skills correlated significantly with both competence and level of training. This study suggested that interpersonal skills are a very important facet of clinical competence that differentiates between residents at different skill levels. Colliver et al.3 also found statistically significant correlations (in the.30 to.50 range) between interpersonal skills and clinical competence. Similarly, Sloan et al.4 found that global interpersonal skill judgments were moderately reliable and correlated highly with overall OSCE performance scores. Thus, it is clear that interpersonal skills are highly associated with the judged competency of medical students' or residents' performances. Several studies have raised the question of who should evaluate interpersonal skills, a faculty proctor (FP) or the standardized patient (SP). Given the increasing clinical demands on faculty time, it is important to know whether SPs can assess interpersonal skills as validly and reliably as faculty members. Cooper and Mira5 found that, on average, SPs gave more positive evaluations of communication skills of undergraduate medical students than did faculty members or other professional staff. They found that the communication scores derived from faculty's ratings did not correlate with the scores derived from the SPs' ratings. Finlay et al.6 assessed the communication skills of primary care physicians who had just received training in communication skills. Professional examiners and SPs evaluated the physicians' communication skills by means of a checklist. The two sets of scores correlated between.40 and.50 on the different OSCE problems, indicating that the SPs' evaluations cannot be used interchangeably with the faculty's evaluations. In a test of the validity of eight faculty raters, Kalet et al.7 videotaped the performances of 21 year-two medical students. Faculty evaluated the interviewing skills of those students on two different occasions using a checklist. The correlations of the communication scores among faculty members were low. Furthermore, the correlations between a faculty member's evaluations of the interviewing skills of the same students' performances on two occasions were also low. A related question is whether checklist scores or global ratings provide more reliable and valid measures of performance. Regehr et al.8 compared the psychometric properties of checklists with those of global rating scales on an eight-problem OSCE given to residents at all levels of training. They found better reliability and construct validity for global rating scales than for checklists. On the other hand, Hodges et al.9 also evaluated the comparative reliability and validity of checklists and global ratings of communication skills. They found high correlations between global ratings and checklists. Based on this review of the literature, we conclude that interpersonal skills are an important component of clinical competence. Global ratings are at least as valid and reliable as checklist scores. However, the levels of reliability and validity of interpersonal-skills ratings have not been clearly established. Also, it is not clear whether faculty or SPs, provide the more valid and reliable evaluations. The purpose of this study was to determine the psychometric characteristics of global interpersonal skills ratings of faculty proctors (FPs) and SPs. Method All 56 residents of a general surgery program participated in a 12-problem, 24-station surgery OSCE. Each OSCE problem was divided into two stations: Part A, in which a history and/or physical examination was performed or information was given to the SP, and Part B, in which the resident responded to several short-answer questions about the patient or SP seen in Part A. This study focused on the 12 Part A stations during which the FPs and SPs evaluated the residents' interpersonal skills. Each of the 24 OSCE stations was five minutes in duration. At each station were either actual patients or SPs who had been trained to act in a consistent manner. As part of their training, they had been instructed in evaluating the residents' interpersonal skills. They had practiced making these evaluations during their training, formally evaluating the interpersonal skills of a resident, who was also evaluated by the trainer. Their evaluations were compared and the trainer and the SP discussed any differences in their evaluations. The typical training session lasted about one hour. During each resident-patient encounter, an FP checked off indicated behaviors as they occurred. At the end of each of the Part A stations, the faculty member evaluated the resident's interpersonal skills (along with several other global performance dimensions). (Note that the trainer had reviewed the checklist and global ratings with each FP immediately before the OSCE.) Faculty rated their level of agreement with the statement “Interacted effectively with the patient” (0 = “not at all” to 4 = “very much”). The SPs independently evaluated the residents' interpersonal skills by telling the preceptors their ratings on the same five-point scale. In order to determine the similarity of FPs' and SPs' ratings, the following analyses were done. First, the reliability of each of the 12 sets of paired FP and SP ratings was estimated by coefficient α. It was also estimated for the mean rating of the FP and the mean rating of the SP. Second, the reliability of the faculty's ratings of the residents' interpersonal skills across the 12 stations was estimated by means of coefficient α, and it was also calculated separately for the SPs' ratings. The Spearman-Brown formula was then used to estimate the expected reliabilities for two faculty raters and two SP raters. These Spearman-Brown estimates provide a standard against which to judge the magnitudes of paired FP and SP reliabilities. It is possible to have relatively high reliabilities even though the FPs' and SPs' ratings may not be very closely calibrated. For example, an SP might be a more lenient evaluator than an FP. One indicator of similar calibration is that the mean rating of the FP is not significantly different from the corresponding mean rating of the SP. A two-way analysis of variance (faculty versus standardized patient, a “between-groups” factor; and comparing clinical problems, a “within-groups” factor) and analyses of simple effects were used to determine whether the FPs and the SPs evaluated the residents' interpersonal skills at approximately the same performance level. If the paired FPs' and SPs' ratings are valid (convergent validity), they ought to correlate more highly with each other than with any other interpersonal skill rating. However, it is possible that this might not be the case. For example, faculty evaluations could correlate most highly with other faculty evaluations as SPs could with other SPs. To determine how the different ratings relate to one another, a hierarchical cluster analysis, using 1 - Pearson r as the similarity metric and the complete linkage amalgamation rule, was performed.10 Clustering methods represent a variety of procedures that identify how variables group (cluster) together. Cluster analysis joins variables together based on the magnitude of the inter-correlations among the variables. A cluster is defined by two or more variables that correlate more highly with each other than they do with the other variables. We chose hierarchical cluster analysis over factor analysis because hierarchical cluster analysis better represents the relationships among variables when most of the variables intercorrelate substantially with each other. This analysis indicated whether the interpersonal-skills ratings clustered predominantly by (1) clinical problem (FP and SP couplets) or (2) FPs separately and SPs separately. Finally, if the interpersonal-skills ratings are valid (construct validity), senior residents ought to perform better than junior residents and interns.2 To this end, Pearson's correlations were calculated between interpersonal skills ratings and postgraduate year. These analyses were carried out for the 12 OSCE stations and the across—station averages for both FPs and SPs. Based on our experience with validity studies such as this, we expected the validity coefficients to be around.40 to.50. Fisher's z-test for differences between correlations was used to test whether the validity coefficients for the FPs and the SPs were significantly different from one another. Results The first data column of Table 1 presents the reliability coefficients for each of the paired (FP and SP) ratings for each of the 12 stations, and of the mean ratings of the FPs and SPs. The reliability of the mean FPs' and SPs' ratings in high,.92. The magnitudes of the reliabilities for the various stations vary: eight of these reliabilities were above.70, two were in the.60s, one was in the 50s, and one was.28.TABLE 1: Psychometric Properties of Faculty Proctors' Ratings and Standardized Patients' Ratings of General Surgery Residents' Interpersonal Skills on a 12-Problem OSCEThe reliability of the faculty's interpersonal-skills ratings for the 12 OSCE stations was.77. The reliability was.74 for the SPs' ratings. The Spearman-Brown formula was used to estimate what these reliabilities would be if there were only two raters—to make them comparable to the paired (FP and SP) reliabilities. The estimated reliability of two faculty raters was.36, and it was.33 for two SPs. If the ratings of FPs and SPs provide fairly equivalent information about the residents' interpersonal skills, there should not be significant differences in their mean ratings. The two-way analysis of variance comparing the equality of faculty's and SPs' means and the equality of means across problems indicated that (1) there was not a significant difference between the two groups (p >.05), (2) there were statistically significant differences among the means for the various OSCE problems (p <.001), and (3) there were significant interactions between groups and problems. Since the significant interactions made the interpretation of the main effects equivocal, analyses of simple effects were performed to identify the exact pattern of differences. The second and third data columns of Table 1 summarize these analyses. As can be seen from this table, the differences in the mean FPs' ratings and the mean SPs' ratings (across the 12 stations) are statistically significant (p <.001). The mean difference is −0.16 on a five-point scale, indicating that the FPs tended to evaluate the residents' interpersonal skills at a slightly lower level than did the SPs. There were significant differences between paired ratings (FP and SP) for four of the OSCEs. In those four cases, the faculty evaluated the residents between three and five tenths of a scale point lower than did the SPs. In the other eight cases, the mean differences were small and not statistically significant. If the interpersonal-skills ratings of a given FP and SP on a particular OSCE are valid, they should correlate highest with each other; however, they should also correlate significantly with the other measures if interpersonal skills is generally a valid construct. A hierarchical cluster analysis was performed to determine whether the FP and SP rating pairs for each station clustered closest. This dyadic clustering did take place for nine of the 12 possible OSCE stations. In these nine cases, the pairs correlated highest with each other. On two of the OSCE problems, the dyadic pairs did not correlate most highly with each other for unknown reasons, and on one OSCE problem, the pair did not cluster with each other because of the lack of variability in the SP's ratings. To explore further the similarities in the ratings, the intercorrelations were calculated among the 24 different ratings of the residents' interpersonal skills. The median correlation among all 276 pairs of ratings was.20 (range −.24 to.89). The median correlation among the faculty's ratings was also.20 (range −.20 to.56), while the median was.17 (range −.16 to.50) for the SPs. In the case of the 12 paired correlations, the median correlation was.60 (range.20 to.89). The construct validity of the FPs' and SPs' ratings was determined using the construct of experience; residents with greater experience should interact more effectively with patients than should junior residents. Pearson correlations were calculated between interpersonal skills ratings and level of experience. The fourth and fifth data columns of Table 1 present these correlations for the faculty and the SPs, respectively. None of the OSCE's paired correlations were significantly different from one another (Fisher's z-test for paired correlations). The average FP's rating (across the 12 stations) and the average SP's rating had higher construct validities than any of the individual interpersonal ratings. Construct validities of.68 and.73 are very high. In the experience of the authors, construct validities usually do not exceed.50. Nine of the 12 construct validities for the faculty ratings were statistically significant, while ten were significant for the SPs. Discussion In this study, faculty proctors and standardized patients were asked to evaluate residents' interpersonal skills at the end of each OSCE station. They made their judgments using a simple single-item scale. For the most part, the level of agreement (reliability) between the FPs and the SPs was adequate. On four of the stations, the reliabilities were sufficiently low to minimize their usefulness in making decisions about performance competency. On the other hand, the reliabilities of the average rating of the FPs and the average rating of the SPs were very satisfactory. Thus, it appears that these simple judgments are for the most part “reliable.” On the other hand, the variability in the magnitudes of the reliability coefficients across stations suggests that one probably should not plan to make educational decisions about competency from performances at individual stations. Rather, it appears that one should use average performance measures. An important consideration in estimating the reliability of ratings of interpersonal skills is whether to estimate reliability across problems or within problem pairs. The reliability of within-OSCE ratings is higher than that of between-OSCE ratings. It may be that interpersonal skills, like clinical reasoning skills, are affected by the context of the clinical case. To explore this possibility, the 24 different ratings of the residents' interpersonal skills were intercorrelated. The median correlation among all possible combinations of raters, among the FPs and among the SPs was about.20. On the other hand, the median correlation among the paired ratings (FP and SP) was.60. Further, the hierarchical cluster analysis indicated that the interpersonal-skills ratings primarily clustered by OSCE station and not by rater type (FP or SP). This result has several implications. First, when the SPs and FPs are evaluating the same patient, their ratings tend to be more valid and more reliable than when the ratings are made on different OSCEs. The reliability appears to be more a function of the OSCE's case than the OSCE's evaluator. Standardized patients tend to give slightly higher evaluations than do faculty proctors. Our data do not indicate whether the FP or the SP is to be preferred. In summary, global ratings of interpersonal skills are both reliable and valid. Faculty proctors and standardized patients appear to be interchangeable as evaluators of interpersonal skills. Case content is an important factor that influences residents' performances of interpersonal skills." @default.
• W1977807981 created "2016-06-24" @default.
• W1977807981 creator A5027050574 @default.
• W1977807981 creator A5027167250 @default.
• W1977807981 creator A5034195468 @default.
• W1977807981 creator A5077606846 @default.
• W1977807981 date "2000-10-01" @default.
• W1977807981 modified "2023-10-09" @default.
• W1977807981 title "Assessment of Residentsʼ Interpersonal Skills by Faculty Proctors and Standardized Patients" @default.
• W1977807981 cites W1973708338 @default.
• W1977807981 cites W1988768643 @default.
• W1977807981 cites W2002542292 @default.
• W1977807981 cites W2008689489 @default.
• W1977807981 cites W2053038863 @default.
• W1977807981 cites W2072613910 @default.
• W1977807981 cites W2156835004 @default.
• W1977807981 doi "https://doi.org/10.1097/00001888-200010001-00030" @default.
• W1977807981 hasPubMedId "https://pubmed.ncbi.nlm.nih.gov/11031186" @default.
• W1977807981 hasPublicationYear "2000" @default.
• W1977807981 type Work @default.
• W1977807981 sameAs 1977807981 @default.
• W1977807981 citedByCount "38" @default.
• W1977807981 countsByYear W19778079812012 @default.
• W1977807981 countsByYear W19778079812013 @default.
• W1977807981 countsByYear W19778079812014 @default.
• W1977807981 countsByYear W19778079812015 @default.
• W1977807981 countsByYear W19778079812020 @default.
• W1977807981 crossrefType "journal-article" @default.
• W1977807981 hasAuthorship W1977807981A5027050574 @default.
• W1977807981 hasAuthorship W1977807981A5027167250 @default.
• W1977807981 hasAuthorship W1977807981A5034195468 @default.
• W1977807981 hasAuthorship W1977807981A5077606846 @default.
• W1977807981 hasBestOaLocation W19778079811 @default.
• W1977807981 hasConcept C118552586 @default.
• W1977807981 hasConcept C15744967 @default.
• W1977807981 hasConcept C159110408 @default.
• W1977807981 hasConcept C164850336 @default.
• W1977807981 hasConcept C509550671 @default.
• W1977807981 hasConcept C512399662 @default.
• W1977807981 hasConcept C71924100 @default.
• W1977807981 hasConcept C77805123 @default.
• W1977807981 hasConcept C86772948 @default.
• W1977807981 hasConcept C91034043 @default.
• W1977807981 hasConceptScore W1977807981C118552586 @default.
• W1977807981 hasConceptScore W1977807981C15744967 @default.
• W1977807981 hasConceptScore W1977807981C159110408 @default.
• W1977807981 hasConceptScore W1977807981C164850336 @default.
• W1977807981 hasConceptScore W1977807981C509550671 @default.
• W1977807981 hasConceptScore W1977807981C512399662 @default.
• W1977807981 hasConceptScore W1977807981C71924100 @default.
• W1977807981 hasConceptScore W1977807981C77805123 @default.
• W1977807981 hasConceptScore W1977807981C86772948 @default.
• W1977807981 hasConceptScore W1977807981C91034043 @default.
• W1977807981 hasIssue "Supplement" @default.
• W1977807981 hasLocation W19778079811 @default.
• W1977807981 hasLocation W19778079812 @default.
• W1977807981 hasLocation W19778079813 @default.
• W1977807981 hasOpenAccess W1977807981 @default.
• W1977807981 hasPrimaryLocation W19778079811 @default.
• W1977807981 hasRelatedWork W2067114768 @default.
• W1977807981 hasRelatedWork W2349430399 @default.
• W1977807981 hasRelatedWork W2352062312 @default.
• W1977807981 hasRelatedWork W2361869160 @default.
• W1977807981 hasRelatedWork W2369624109 @default.
• W1977807981 hasRelatedWork W2373901468 @default.
• W1977807981 hasRelatedWork W2381515849 @default.
• W1977807981 hasRelatedWork W2391530046 @default.
• W1977807981 hasRelatedWork W3048968805 @default.
• W1977807981 hasRelatedWork W2563120717 @default.
• W1977807981 hasVolume "75" @default.
• W1977807981 isParatext "false" @default.
• W1977807981 isRetracted "false" @default.
• W1977807981 magId "1977807981" @default.
• W1977807981 workType "article" @default.