Matches in SemOpenAlex for { <https://semopenalex.org/work/W2003497265> ?p ?o ?g. }
Showing items 1 to 100 of
100
with 100 items per page.
- W2003497265 abstract "The creation of golden standard datasets is a costly business. Optimally more than one judgment per document is obtained to ensure a high quality on annotations. In this context, we explore how much annotations from experts differ from each other, how different sets of annotations influence the ranking of systems and if these annotations can be obtained with a crowdsourcing approach. This study is applied to annotations of images with multiple concepts. A subset of the images employed in the latest ImageCLEF Photo Annotation competition was manually annotated by expert annotators and non-experts with Mechanical Turk. The inter-annotator agreement is computed at an image-based and concept-based level using majority vote, accuracy and kappa statistics. Further, the Kendall τ and Kolmogorov-Smirnov correlation test is used to compare the ranking of systems regarding different ground-truths and different evaluation measures in a benchmark scenario. Results show that while the agreement between experts and non-experts varies depending on the measure used, its influence on the ranked lists of the systems is rather small. To sum up, the majority vote applied to generate one annotation set out of several opinions, is able to filter noisy judgments of non-experts to some extent. The resulting annotation set is of comparable quality to the annotations of experts." @default.
- W2003497265 created "2016-06-24" @default.
- W2003497265 creator A5005712172 @default.
- W2003497265 creator A5055679306 @default.
- W2003497265 date "2010-03-29" @default.
- W2003497265 modified "2023-09-30" @default.
- W2003497265 title "How reliable are annotations via crowdsourcing" @default.
- W2003497265 cites W1985514943 @default.
- W2003497265 cites W1990892398 @default.
- W2003497265 cites W2108598243 @default.
- W2003497265 cites W2120308175 @default.
- W2003497265 cites W2120665002 @default.
- W2003497265 cites W2125943921 @default.
- W2003497265 cites W2137935418 @default.
- W2003497265 cites W2149489787 @default.
- W2003497265 cites W2149490731 @default.
- W2003497265 cites W2155803963 @default.
- W2003497265 cites W2164777277 @default.
- W2003497265 cites W2170931744 @default.
- W2003497265 doi "https://doi.org/10.1145/1743384.1743478" @default.
- W2003497265 hasPublicationYear "2010" @default.
- W2003497265 type Work @default.
- W2003497265 sameAs 2003497265 @default.
- W2003497265 citedByCount "257" @default.
- W2003497265 countsByYear W20034972652012 @default.
- W2003497265 countsByYear W20034972652013 @default.
- W2003497265 countsByYear W20034972652014 @default.
- W2003497265 countsByYear W20034972652015 @default.
- W2003497265 countsByYear W20034972652016 @default.
- W2003497265 countsByYear W20034972652017 @default.
- W2003497265 countsByYear W20034972652018 @default.
- W2003497265 countsByYear W20034972652019 @default.
- W2003497265 countsByYear W20034972652020 @default.
- W2003497265 countsByYear W20034972652021 @default.
- W2003497265 countsByYear W20034972652022 @default.
- W2003497265 countsByYear W20034972652023 @default.
- W2003497265 crossrefType "proceedings-article" @default.
- W2003497265 hasAuthorship W2003497265A5005712172 @default.
- W2003497265 hasAuthorship W2003497265A5055679306 @default.
- W2003497265 hasBestOaLocation W20034972652 @default.
- W2003497265 hasConcept C106131492 @default.
- W2003497265 hasConcept C111472728 @default.
- W2003497265 hasConcept C124101348 @default.
- W2003497265 hasConcept C13280743 @default.
- W2003497265 hasConcept C136764020 @default.
- W2003497265 hasConcept C138885662 @default.
- W2003497265 hasConcept C151730666 @default.
- W2003497265 hasConcept C154945302 @default.
- W2003497265 hasConcept C177264268 @default.
- W2003497265 hasConcept C185798385 @default.
- W2003497265 hasConcept C189430467 @default.
- W2003497265 hasConcept C199360897 @default.
- W2003497265 hasConcept C205649164 @default.
- W2003497265 hasConcept C23123220 @default.
- W2003497265 hasConcept C2776321320 @default.
- W2003497265 hasConcept C2779343474 @default.
- W2003497265 hasConcept C2779530757 @default.
- W2003497265 hasConcept C31972630 @default.
- W2003497265 hasConcept C41008148 @default.
- W2003497265 hasConcept C62230096 @default.
- W2003497265 hasConcept C86803240 @default.
- W2003497265 hasConceptScore W2003497265C106131492 @default.
- W2003497265 hasConceptScore W2003497265C111472728 @default.
- W2003497265 hasConceptScore W2003497265C124101348 @default.
- W2003497265 hasConceptScore W2003497265C13280743 @default.
- W2003497265 hasConceptScore W2003497265C136764020 @default.
- W2003497265 hasConceptScore W2003497265C138885662 @default.
- W2003497265 hasConceptScore W2003497265C151730666 @default.
- W2003497265 hasConceptScore W2003497265C154945302 @default.
- W2003497265 hasConceptScore W2003497265C177264268 @default.
- W2003497265 hasConceptScore W2003497265C185798385 @default.
- W2003497265 hasConceptScore W2003497265C189430467 @default.
- W2003497265 hasConceptScore W2003497265C199360897 @default.
- W2003497265 hasConceptScore W2003497265C205649164 @default.
- W2003497265 hasConceptScore W2003497265C23123220 @default.
- W2003497265 hasConceptScore W2003497265C2776321320 @default.
- W2003497265 hasConceptScore W2003497265C2779343474 @default.
- W2003497265 hasConceptScore W2003497265C2779530757 @default.
- W2003497265 hasConceptScore W2003497265C31972630 @default.
- W2003497265 hasConceptScore W2003497265C41008148 @default.
- W2003497265 hasConceptScore W2003497265C62230096 @default.
- W2003497265 hasConceptScore W2003497265C86803240 @default.
- W2003497265 hasLocation W20034972651 @default.
- W2003497265 hasLocation W20034972652 @default.
- W2003497265 hasOpenAccess W2003497265 @default.
- W2003497265 hasPrimaryLocation W20034972651 @default.
- W2003497265 hasRelatedWork W172170795 @default.
- W2003497265 hasRelatedWork W1997653951 @default.
- W2003497265 hasRelatedWork W2103153231 @default.
- W2003497265 hasRelatedWork W2163901716 @default.
- W2003497265 hasRelatedWork W2187223183 @default.
- W2003497265 hasRelatedWork W2971142271 @default.
- W2003497265 hasRelatedWork W4298819607 @default.
- W2003497265 hasRelatedWork W4378377766 @default.
- W2003497265 hasRelatedWork W50774052 @default.
- W2003497265 hasRelatedWork W7602594 @default.
- W2003497265 isParatext "false" @default.
- W2003497265 isRetracted "false" @default.
- W2003497265 magId "2003497265" @default.
- W2003497265 workType "article" @default.