Matches in SemOpenAlex for { <https://semopenalex.org/work/W3166650478> ?p ?o ?g. }
Showing items 1 to 77 of
77
with 100 items per page.
- W3166650478 abstract "In many classification tasks, the ground truth is either noisy or subjective. Examples include: which of two alternative paper titles is better? is this comment toxic? what is the political leaning of this news article? We refer to such tasks as survey settings because the ground truth is defined through a survey of one or more human raters. In survey settings, conventional measurements of classifier accuracy such as precision, recall, and cross-entropy confound the quality of the classifier with the level of agreement among human raters. Thus, they have no meaningful interpretation on their own. We describe a procedure that, given a dataset with predictions from a classifier and K ratings per item, rescales any accuracy measure into one that has an intuitive interpretation. The key insight is to score the classifier not against the best proxy for the ground truth, such as a majority vote of the raters, but against a single human rater at a time. That score can be compared to other predictors' scores, in particular predictors created by combining labels from several other human raters. The survey equivalence of any classifier is the minimum number of raters needed to produce the same expected score as that found for the classifier." @default.
- W3166650478 created "2021-06-22" @default.
- W3166650478 creator A5050228339 @default.
- W3166650478 creator A5058603769 @default.
- W3166650478 creator A5084597959 @default.
- W3166650478 creator A5073672615 @default.
- W3166650478 date "2021-06-02" @default.
- W3166650478 modified "2023-09-23" @default.
- W3166650478 title "Survey Equivalence: A Procedure for Measuring Classifier Accuracy Against Human Labels." @default.
- W3166650478 cites W1995875735 @default.
- W3166650478 cites W1996468215 @default.
- W3166650478 cites W2054141820 @default.
- W3166650478 cites W2102348129 @default.
- W3166650478 cites W2217796615 @default.
- W3166650478 cites W2540646130 @default.
- W3166650478 cites W2588172982 @default.
- W3166650478 cites W2798706400 @default.
- W3166650478 cites W2963697299 @default.
- W3166650478 cites W2964215131 @default.
- W3166650478 cites W3001359517 @default.
- W3166650478 cites W3124505561 @default.
- W3166650478 cites W3163078977 @default.
- W3166650478 cites W621249151 @default.
- W3166650478 hasPublicationYear "2021" @default.
- W3166650478 type Work @default.
- W3166650478 sameAs 3166650478 @default.
- W3166650478 citedByCount "0" @default.
- W3166650478 crossrefType "posted-content" @default.
- W3166650478 hasAuthorship W3166650478A5050228339 @default.
- W3166650478 hasAuthorship W3166650478A5058603769 @default.
- W3166650478 hasAuthorship W3166650478A5073672615 @default.
- W3166650478 hasAuthorship W3166650478A5084597959 @default.
- W3166650478 hasConcept C100660578 @default.
- W3166650478 hasConcept C119857082 @default.
- W3166650478 hasConcept C146849305 @default.
- W3166650478 hasConcept C153180895 @default.
- W3166650478 hasConcept C154945302 @default.
- W3166650478 hasConcept C15744967 @default.
- W3166650478 hasConcept C180747234 @default.
- W3166650478 hasConcept C41008148 @default.
- W3166650478 hasConcept C81669768 @default.
- W3166650478 hasConcept C95623464 @default.
- W3166650478 hasConceptScore W3166650478C100660578 @default.
- W3166650478 hasConceptScore W3166650478C119857082 @default.
- W3166650478 hasConceptScore W3166650478C146849305 @default.
- W3166650478 hasConceptScore W3166650478C153180895 @default.
- W3166650478 hasConceptScore W3166650478C154945302 @default.
- W3166650478 hasConceptScore W3166650478C15744967 @default.
- W3166650478 hasConceptScore W3166650478C180747234 @default.
- W3166650478 hasConceptScore W3166650478C41008148 @default.
- W3166650478 hasConceptScore W3166650478C81669768 @default.
- W3166650478 hasConceptScore W3166650478C95623464 @default.
- W3166650478 hasOpenAccess W3166650478 @default.
- W3166650478 hasRelatedWork W1544396233 @default.
- W3166650478 hasRelatedWork W1776786959 @default.
- W3166650478 hasRelatedWork W183268581 @default.
- W3166650478 hasRelatedWork W205133396 @default.
- W3166650478 hasRelatedWork W2093184528 @default.
- W3166650478 hasRelatedWork W2216621827 @default.
- W3166650478 hasRelatedWork W2231509025 @default.
- W3166650478 hasRelatedWork W2294006067 @default.
- W3166650478 hasRelatedWork W2343109543 @default.
- W3166650478 hasRelatedWork W2752279082 @default.
- W3166650478 hasRelatedWork W2753133191 @default.
- W3166650478 hasRelatedWork W2786063148 @default.
- W3166650478 hasRelatedWork W2913942287 @default.
- W3166650478 hasRelatedWork W2990522702 @default.
- W3166650478 hasRelatedWork W3035203924 @default.
- W3166650478 hasRelatedWork W3048307881 @default.
- W3166650478 hasRelatedWork W3092952793 @default.
- W3166650478 hasRelatedWork W3102927640 @default.
- W3166650478 hasRelatedWork W3139216302 @default.
- W3166650478 hasRelatedWork W3198677743 @default.
- W3166650478 isParatext "false" @default.
- W3166650478 isRetracted "false" @default.
- W3166650478 magId "3166650478" @default.
- W3166650478 workType "article" @default.