FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W2907416181> ?p ?o ?g. }

• W2907416181 abstract "Rater variation in performance assessment can impact test-takers’ scores and compromise assessments’ fairness and validity (Crooks, Kane, & Cohen, 1996). Rater variation can also undermine a test’s validity and fairness; therefore, it is important to investigate raters’ scoring patterns in order to inform rater training. Substantial work has been done analyzing rater cognition in writing assessment (e.g., Cumming, 1990; Eckes, 2008); however, few studies have tried to classify factors that could contribute to rater variation in speaking assessment (e.g., May, 2006). The present study used a mixed methods approach (Tashakkori & Teddlie, 1998; Greene, Carcelli, & Graham, 1989) to investigate the potential differences between native English-speaking and non-native English-speaking raters in how they assess L2 students’ speaking performance. Kane’s (2006) argument-based approach to validity was used as the theoretical framework. The study challenged the plausibility of the assumptions for the evaluation inference, which links the observed performance and the observed score and depends on the assumption that the raters apply the scoring rubric accurately and consistently. The study analyzed raters’ scoring patterns when using a TOEFL iBT speaking rubric analytically. The raters provided scores for each rubric criterion (i.e., Overall, Delivery, Language Use, and Topic Development). Each rater received individual training, practice, and calibration experience. All the raters filled out a background questionnaire asking about their teaching experiences, language learning history, the background of students in their classrooms, and their exposure to and familiarity with the non-native accents used in the study. For the quantitative analysis, the two groups of raters 23 native (North American) and 23 non-native (Russian) raters graded and left comments for speech samples from Arabic (n = 25), Chinese (n = 25), and Russian (n = 25) L1 background. Students’ samples were in response to two independent speaking tasks; the students’ responses varied from low to high proficiency levels. For the qualitative part, 16 raters (7 native and 9 non-native) shared their scoring behavior through think-aloud protocols and interviews. The speech samples graded during the think-aloud included Arabic (n = 4), Chinese (n = 4), and Russian (n = 4) speakers. Raters’ scores were examined using the Multi-Faceted Rasch Measurement using FACETS (Linacre, 2014) software to test group differences between native and non-native raters as well as raters who are familiar and unfamiliar with the accents of students in the study. In addition, raters’ comments were coded and also used to explore rater group differences. The qualitative analyses involved thematical coding of transcribed think-aloud sessions and interview sessions using content analysis (Strauss & Corbin, 1998) to investigate the cognitive processes of raters and their perceptions of their rating processes. The coding included such themes as decision-making and re-listening patterns, perceived severity, criteria importance, and non-rubric criteria (e.g., accent familiarity, L1 match). Afterward, the quantitative and qualitative results were analyzed together to describe the potential sources of rater variability. This analysis was done employing side-by-side comparison of qualitative and quantitative data (Onwuegbuzie & Teddlie, 2003). The results revealed that there were no radical differences between native and non-native raters; however, some different patterns were observed. Non-native raters also showed more lenient grading patterns towards the students with whom their L1 matched. In addition, all raters, regardless of the group, demonstrated several patterns of rating depending on their focus while listening to examinees’ performance and interpretations of the rating criteria during the decision-making process. The findings can motivate professionals who oversee and train raters at testing companies and intensive English programs to study their raters’ scoring behaviors to individualize training to help make exam ratings fair and raters interchangeable." @default.
• W2907416181 created "2019-01-11" @default.
• W2907416181 creator A5028296890 @default.
• W2907416181 date "2018-05-01" @default.
• W2907416181 modified "2023-09-26" @default.
• W2907416181 title "Native and Non-Native Raters of L2 Speaking Performance: Accent Familiarity and Cognitive Processes." @default.
• W2907416181 cites W119360514 @default.
• W2907416181 cites W134005242 @default.
• W2907416181 cites W135624618 @default.
• W2907416181 cites W1509317141 @default.
• W2907416181 cites W1548015960 @default.
• W2907416181 cites W1564229462 @default.
• W2907416181 cites W1609477948 @default.
• W2907416181 cites W1658908529 @default.
• W2907416181 cites W1704942559 @default.
• W2907416181 cites W1914972138 @default.
• W2907416181 cites W1942444330 @default.
• W2907416181 cites W1958015814 @default.
• W2907416181 cites W1964860056 @default.
• W2907416181 cites W1969594068 @default.
• W2907416181 cites W1972315868 @default.
• W2907416181 cites W1972399362 @default.
• W2907416181 cites W1980081737 @default.
• W2907416181 cites W1981835473 @default.
• W2907416181 cites W1983203079 @default.
• W2907416181 cites W1984484958 @default.
• W2907416181 cites W1988344745 @default.
• W2907416181 cites W1990423818 @default.
• W2907416181 cites W1991701363 @default.
• W2907416181 cites W1994119827 @default.
• W2907416181 cites W1995617856 @default.
• W2907416181 cites W1995633917 @default.
• W2907416181 cites W1997210479 @default.
• W2907416181 cites W1998407119 @default.
• W2907416181 cites W2000130928 @default.
• W2907416181 cites W2000641383 @default.
• W2907416181 cites W2004174491 @default.
• W2907416181 cites W2006444775 @default.
• W2907416181 cites W2010821160 @default.
• W2907416181 cites W2011522158 @default.
• W2907416181 cites W2012225036 @default.
• W2907416181 cites W2013215188 @default.
• W2907416181 cites W2034865276 @default.
• W2907416181 cites W2039681496 @default.
• W2907416181 cites W2043544836 @default.
• W2907416181 cites W2043914905 @default.
• W2907416181 cites W2044848129 @default.
• W2907416181 cites W2046498560 @default.
• W2907416181 cites W2046812810 @default.
• W2907416181 cites W2048370658 @default.
• W2907416181 cites W2048740180 @default.
• W2907416181 cites W2052836008 @default.
• W2907416181 cites W2059236258 @default.
• W2907416181 cites W2062175493 @default.
• W2907416181 cites W2064408614 @default.
• W2907416181 cites W2066448258 @default.
• W2907416181 cites W2066516853 @default.
• W2907416181 cites W2068511252 @default.
• W2907416181 cites W2077938266 @default.
• W2907416181 cites W2078309767 @default.
• W2907416181 cites W2082677244 @default.
• W2907416181 cites W2082767250 @default.
• W2907416181 cites W2092161955 @default.
• W2907416181 cites W2094553432 @default.
• W2907416181 cites W2095002948 @default.
• W2907416181 cites W2096160735 @default.
• W2907416181 cites W2097473141 @default.
• W2907416181 cites W2099288386 @default.
• W2907416181 cites W2099970223 @default.
• W2907416181 cites W2100044678 @default.
• W2907416181 cites W2103074295 @default.
• W2907416181 cites W2109685530 @default.
• W2907416181 cites W2110266099 @default.
• W2907416181 cites W2111266316 @default.
• W2907416181 cites W2113523640 @default.
• W2907416181 cites W2116204118 @default.
• W2907416181 cites W2120764744 @default.
• W2907416181 cites W2121567902 @default.
• W2907416181 cites W2122490249 @default.
• W2907416181 cites W2122873537 @default.
• W2907416181 cites W2130631175 @default.
• W2907416181 cites W2134146415 @default.
• W2907416181 cites W2139969072 @default.
• W2907416181 cites W2143149791 @default.
• W2907416181 cites W2145055698 @default.
• W2907416181 cites W2149334204 @default.
• W2907416181 cites W2155314571 @default.
• W2907416181 cites W2157049701 @default.
• W2907416181 cites W2159738567 @default.
• W2907416181 cites W2162140083 @default.
• W2907416181 cites W2167685582 @default.
• W2907416181 cites W2168303754 @default.
• W2907416181 cites W2169958266 @default.
• W2907416181 cites W2170448089 @default.
• W2907416181 cites W2170588127 @default.
• W2907416181 cites W2171039963 @default.
• W2907416181 cites W225466892 @default.
• W2907416181 cites W2294237342 @default.
• W2907416181 cites W2319784989 @default.
• W2907416181 cites W2320386680 @default.

• next