FRINK Linked Data Fragments server

Matches in SemOpenAlex for { <https://semopenalex.org/work/W4387582605> ?p ?o ?g. }

• W4387582605 endingPage "919" @default.
• W4387582605 startingPage "901" @default.
• W4387582605 abstract "Objectives The purpose of this study is to investigate the magnitudes of and relationships among different effect-size indexes for inter-rater agreement between human and machine scoring in writing assessments.&#x0D; Methods The procedure of meta-analyses consists of data gathering, including literature search with criteria for inclusion and exclusion, and data analysis, including data cleaning and coding, after tests of heterogeneity for each index, hierarchical weighted models, and moderator and contrast analyses were conducted. Appropriate analyses were conducted using rma, robust, and robu functions in the metafor and robumeta packages in R software Version 3.3.2.&#x0D; Results Based on the results, the overall random-effects means for correlations, kappa values, and adjacent proportions of agreement between automated and human scoring of essay writing were .75, .48, and .99, respectively. The results from hierarchical weighed models and heterogeneity tests indicate that the rates of agreement for each index were inconsistent across studies. According to moderator and contrast analyses, correlations and kappa values using 6-point scales significantly differed from those using 3-, 4-, and 5-point scales, respectively, at alpha level .05. On the other hand, the adjacent proportions of agreement on either 3- or 4-point scales significantly differed from the adjacent proportions of agreement on the 5- and 6-point scales, respectively, at alpha level .01. Regarding the exact and adjacent proportions of agreement, the average of discrepancies was 0.34 units, and the variance of discrepancies between exact and adjacent proportions of agreement was 0.004. Similarly, the mean of discrepancies between the correlation and kappa was 0.27, and the variance of discrepancies between the correlation and kappa was 0.003.&#x0D; Conclusions According to this finding, machine scoring is similar to human scoring in terms of relative consistency and absolute consensus. Compared to the evaluation criteria suggested by prior studies, the degrees of inter-rater agreement seen in this study were above the thresholds for correlations, moderate agreement for kappa, and in the range of consensus rates for adjacent proportion agreement. The rates of agreement for each index were inconsistent across studies. This means that the all agreement indexes had relatively large between-studies differences compared to the between-effects differences within the studies. It is necessary to investigate if some moderators explain the between-studies differences. The number of score-scale points used for measuring inter-rater agreement between human and machine scoring influenced the agreement rates. The relationships across the four indexes (i.e.,  , , , and ) from the study appear to be reasonably strong and linear. Regardless of the number of points on the score scales, kappa values are 0.27 points lower than correlations. In addition, the mean exact proportions of agreements is 0.34 points lower than the mean adjacent proportions of agreements. Since each inter-rater agreement index has its own disadvantages, such as scale dependency, not showing the degree of identical matching and matching patterns, and so on, it is advised to report several inter-rater agreement indexes." @default.
• W4387582605 created "2023-10-13" @default.
• W4387582605 creator A5011263299 @default.
• W4387582605 date "2023-09-30" @default.
• W4387582605 modified "2023-10-14" @default.
• W4387582605 title "Relationships among Different Effect-Size Indexes for Inter-Rater Agreement between Human and Automated Essay Scoring" @default.
• W4387582605 doi "https://doi.org/10.22251/jlcci.2023.23.18.901" @default.
• W4387582605 hasPublicationYear "2023" @default.
• W4387582605 type Work @default.
• W4387582605 citedByCount "0" @default.
• W4387582605 crossrefType "journal-article" @default.
• W4387582605 hasAuthorship W4387582605A5011263299 @default.
• W4387582605 hasConcept C105795698 @default.
• W4387582605 hasConcept C117220453 @default.
• W4387582605 hasConcept C121955636 @default.
• W4387582605 hasConcept C138885662 @default.
• W4387582605 hasConcept C144133560 @default.
• W4387582605 hasConcept C149782125 @default.
• W4387582605 hasConcept C154945302 @default.
• W4387582605 hasConcept C15744967 @default.
• W4387582605 hasConcept C163864269 @default.
• W4387582605 hasConcept C196083921 @default.
• W4387582605 hasConcept C2524010 @default.
• W4387582605 hasConcept C2776502983 @default.
• W4387582605 hasConcept C2776818064 @default.
• W4387582605 hasConcept C2778724333 @default.
• W4387582605 hasConcept C33923547 @default.
• W4387582605 hasConcept C41008148 @default.
• W4387582605 hasConcept C41895202 @default.
• W4387582605 hasConcept C61863361 @default.
• W4387582605 hasConcept C83849319 @default.
• W4387582605 hasConcept C93225998 @default.
• W4387582605 hasConceptScore W4387582605C105795698 @default.
• W4387582605 hasConceptScore W4387582605C117220453 @default.
• W4387582605 hasConceptScore W4387582605C121955636 @default.
• W4387582605 hasConceptScore W4387582605C138885662 @default.
• W4387582605 hasConceptScore W4387582605C144133560 @default.
• W4387582605 hasConceptScore W4387582605C149782125 @default.
• W4387582605 hasConceptScore W4387582605C154945302 @default.
• W4387582605 hasConceptScore W4387582605C15744967 @default.
• W4387582605 hasConceptScore W4387582605C163864269 @default.
• W4387582605 hasConceptScore W4387582605C196083921 @default.
• W4387582605 hasConceptScore W4387582605C2524010 @default.
• W4387582605 hasConceptScore W4387582605C2776502983 @default.
• W4387582605 hasConceptScore W4387582605C2776818064 @default.
• W4387582605 hasConceptScore W4387582605C2778724333 @default.
• W4387582605 hasConceptScore W4387582605C33923547 @default.
• W4387582605 hasConceptScore W4387582605C41008148 @default.
• W4387582605 hasConceptScore W4387582605C41895202 @default.
• W4387582605 hasConceptScore W4387582605C61863361 @default.
• W4387582605 hasConceptScore W4387582605C83849319 @default.
• W4387582605 hasConceptScore W4387582605C93225998 @default.
• W4387582605 hasIssue "18" @default.
• W4387582605 hasLocation W43875826051 @default.
• W4387582605 hasOpenAccess W4387582605 @default.
• W4387582605 hasPrimaryLocation W43875826051 @default.
• W4387582605 hasRelatedWork W1966976587 @default.
• W4387582605 hasRelatedWork W1979852909 @default.
• W4387582605 hasRelatedWork W1980457457 @default.
• W4387582605 hasRelatedWork W1984207165 @default.
• W4387582605 hasRelatedWork W1996751524 @default.
• W4387582605 hasRelatedWork W2054015915 @default.
• W4387582605 hasRelatedWork W2330902540 @default.
• W4387582605 hasRelatedWork W2581818947 @default.
• W4387582605 hasRelatedWork W4294214983 @default.
• W4387582605 hasRelatedWork W96748260 @default.
• W4387582605 hasVolume "23" @default.
• W4387582605 isParatext "false" @default.
• W4387582605 isRetracted "false" @default.
• W4387582605 workType "article" @default.